verify {verification}R Documentation

Verification function

Description

Based on the type of inputs, this function calculates a range of verification statistics and skill scores. Additionally, it creates a verify class object that can be used in further analysis or with other methods such as plot and summary.

Usage

    verify(obs, pred, baseline = NULL, 
           frcst.type = "prob", obs.type = "binary",  thresholds =
seq(0,1,0.1), show = TRUE, bins = TRUE, ... )
       

Arguments

obs The values with which the verifications are verified. May be a vector of length 4 if the forecast and predictions are binary data summarized in a contingency table. In this case, the value are entered in the order of c(n11, n01, n10, n00). If obs is a matrix, it is assumed to be a contingency table with observed values summarized in the columns and forecasted values summarized in the rows.
pred Prediction of event. The prediction may be in the form of the a point prediction or the probability of a forecast. Let pred = NULL if obs is a contingency table.
baseline In meteorology, climatology is the baseline that represents the no-skill forecast. In other fields this field would differ. This field is used to calculate certain skill scores. If left NULL, these statistics are calculated using sample climatology.
frcst.type Forecast type. Either "prob", "binary", "norm.dist", "cat" or "cont". Defaults to "prob". "norm.dist" is used when the forecast is in the form of a normal distribution. See crps for more details.
obs.type Observation type. Either "binary", "cat" or "cont". Defaults to "binary"
thresholds Thresholds to be considered for point forecasts of continuous events.
show Binary; if TRUE (the default), print warning message
bins Binary; if TRUE (default), the probabilistic forecasts are placed in bins defined by the sequence defined in threshold and assigned the midpoint value.
... Additional options.

Value

An object of the verify class. Depending on the type of data used, the following information may be returned. The following notation is used to describe which values are produced for which type of forecast/observations. (BB = binary/binary, PB = probablistic/binary, CC = continuous/continuous, CTCT = categorical/categorical)

BS Brier Score (PB)
BSS Brier Skill Score(PB)
SS Skill Score (BB)
hit.rate Hit rate, aka PODy, $h$ (PB, CTCT)
false.alarm.rate False alarm rate, PODn, $f$ (PB, CTCT)
TS Threat Score or Critical Success Index (CSI)(BB, CTCT)
ETS Equitable Threat Score (BB, CTCT)
BIAS Bias (BB, CTCT)
PC Percent correct or hit rate (BB, CTCT)
Cont.Table Contingency Table (BB)
HSS Heidke Skill Score(BB, CTCT)
KSS Kuniper Skill Score (BB)
PSS Pierce Skill Score (CTCT)
GS Gerrity Score (CTCT)
ME Mean error (CC)
MSE Mean-squared error (CC)
MAE Mean absolute error (CC)

Note

For the categorical forecast and verification, the Gerrity score only makes sense for forecast that have order, or are basically ordinal. It is assumed that the forecasts are listed in order. For example, if the rows of a contigency table were summarized as "medium, low, high", the Gerrity score will be incorrectly summarized.

Author(s)

Matt Pocernich <pocernic@rap.ucar.edu>

References

Wilks, D. S. (1995) Statistical Methods in the Atmospheric Sciences Chapter 7, San Diego: Academic Press.

WMO Joint WWRP/WGNE Working Group on Verification Website

http://www.bom.gov.au/bmrc/wefor/staff/eee/verif/verif_web_page.html

Examples

# binary/binary example
obs<- round(runif(100))
pred<- round(runif(100))

# binary/binary example
# Finley tornado data.

obs<- c(28, 72, 23, 2680)
A<- verify(obs, pred = NULL, frcst.type = "binary", obs.type = "binary")

summary(A)

# categorical/categorical example
# creates a simulated 5 category forecast and observation.
obs <- round(runif(100, 1,5) )
pred <- round(runif(100, 1,5) )

A<- verify(obs, pred, frcst.type = "cat", obs.type = "cat" )
summary(A)

#  probabilistic/ binary example

pred<- runif(100)
A<- verify(obs, pred, frcst.type = "prob", obs.type = "binary")
summary(A)

# continuous/ continuous example
obs<- rnorm(100)
pred<- rnorm(100)
baseline <- rnorm(100, sd = 0.5) 

A<- verify(obs, pred, baseline = baseline,  frcst.type = "cont", obs.type = "cont")
summary(A)

[Package verification version 1.29 Index]