Reliability diagrams for non-parametric density forecasts of continuous variables: Accounting for serial correlation

Reliability is seen as a primary requirement when verifying probabilistic forecasts, since a lack of reliability would introduce a systematic bias in subsequent decision-making. Reliability diagrams comprise popular and practical diagnostic tools for the reliability evaluation of density forecasts of continuous variables.

Such diagrams relate to the assessment of the unconditional calibration of probabilistic forecasts. A reason for their appeal is that deviations from perfect reliability can be visually assessed based on deviations from the diagonal. Deviations from the diagonal may, however, be caused by both sampling effects and serial correlation in the forecast-verification pairs.

We build on a recent proposal, consisting of associating reliability diagrams with consistency bars that would reflect the deviations from the diagonal that are potentially observable even if density forecasts are perfectly reliable. Our consistency bars, however, reflect potential deviations originating from the combined effects of limited counting statistics and serial correlation in the forecast-verification pairs.

They are generated based on an original surrogate consistency resampling method. Its ability to provide consistency bars with a significantly better coverage against the independent and identically distributed (i.i.d.) resampling alternative is shown from simulations. Finally, a practical example of the reliability assessment of non-parametric density forecasts of short-term wind-power generation is given.

Copyright © 2010 Royal Meteorological Society