Verifying the relationship between ensemble forecast spread and skill [presentation]
Hopson, T., & Weiss, J. (2006). Verifying the relationship between ensemble forecast spread and skill [presentation]. In AGU Fall Meeting 2006. American Geophysical Union: San Francisco, CA, US.
With the increased utilization of uncertainty information and ensemble approaches in weather and hydrologic forecasting, verification tools are needed to test the efficacy of these methods. In particular, we are interested in verifying the information content in the varying ensemble spread of a f... Show moreWith the increased utilization of uncertainty information and ensemble approaches in weather and hydrologic forecasting, verification tools are needed to test the efficacy of these methods. In particular, we are interested in verifying the information content in the varying ensemble spread of a forecast, as a representation of potentially varying forecast error; in other words, how do you determine if changing uncertainty bounds on a forecast truly capture changes in potential forecast error? Intuitively, one would expect the correlation between an ensemble forecast's spread and forecast error to provide one such measure. However, we point out the deficiencies of this spread-error correlation measure, showing how its value depends on factors other than forecast model performance: it cannot reach its maximum value of one; its asymptotic value depends on the specific definition of spread and error used; and its limits depend on the stability properties of the system being modeled. We argue, however, that by representing the correlation in a simple skill-score representation, these deficiencies can at least be partially mitigated. As well, we present two other alternative skill measures to validate the spread-error relationship. We argue that the factor that controls the theoretical upper limit of skill-spread correlation also provides a measure of the utility of generating ensemble forecasts with varying ensemble dispersion. Specifically, this governing factor in combination with an effective spread-error skill score provides an effective measure of whether the second moment of an ensemble forecast statistically outperforms its own ensemble mean forecast dressed with an error climatology. In particular, we argue the test of the utility of the second moment of an ensemble forecast is that it outperform a heteroscedastic error model. We show an example of this skill-spread analysis for Brahmaputra catchment-averaged ECMWF ensemble precipitation forecasts. Show less