Estimating reliability and resolution of probability forecasts through decomposition of the empirical score

Bröcker, Jochen
August 2012
Climate Dynamics;Aug2012, Vol. 39 Issue 3/4, p655
Academic Journal
Proper scoring rules provide a useful means to evaluate probabilistic forecasts. Independent from scoring rules, it has been argued that reliability and resolution are desirable forecast attributes. The mathematical expectation value of the score allows for a decomposition into reliability and resolution related terms, demonstrating a relationship between scoring rules and reliability/resolution. A similar decomposition holds for the empirical (i.e. sample average) score over an archive of forecast-observation pairs. This empirical decomposition though provides a too optimistic estimate of the potential score (i.e. the optimum score which could be obtained through recalibration), showing that a forecast assessment based solely on the empirical resolution and reliability terms will be misleading. The differences between the theoretical and empirical decomposition are investigated, and specific recommendations are given how to obtain better estimators of reliability and resolution in the case of the Brier and Ignorance scoring rule.


Related Articles

  • A general relativistic rotating evolutionary universe—Part II. Berman, Marcelo Samuel // Astrophysics & Space Science;May2008, Vol. 315 Issue 1-4, p367 

    As a sequel to Berman (Astrophys. Space Sci., ), we show that the rotation of the Universe can be dealt by generalised Gaussian metrics, defined in this paper. Robertson-Walker’s metric has been employed with proper-time, in its standard applications; the generalised Gaussian metric...

  • mt MASTERCLASS.  // Management Today;Feb2009, p20 

    The article provides an overview on the term forecasting. Forecasting was used by the Pagans to study the entrails of slaughtered beasts to see what the future held. The Celts had their druids as forecasters, while the Romans their soothsayers. However, columnist Sir Simon Jenkins said that no...

  • A New Beginning for the Future of Futuring in New Zealand. Curtis, Yvonne // Future Times;2013, Vol. 1/2, p6 

    The article reports on the New Zealand (NZ) Futures Trust Thinking Futures Workshop held on March 6, 2013 at Te Papa Tongarewa. How the work of the NZ Futures Trust could be continued open to the futures community was discussed. The need for a New Zealand Based futures thinking body is stressed....

  • Application of Hybrid Quantum Tabu Search with Support Vector Regression (SVR) for Load Forecasting. Cheng-Wen Lee; Bing-Yi Lin // Energies (19961073);Nov2016, Vol. 9 Issue 11, p873 

    Hybridizing chaotic evolutionary algorithms with support vector regression (SVR) to improve forecasting accuracy is a hot topic in electricity load forecasting. Trapping at local optima and premature convergence are critical shortcomings of the tabu search (TS) algorithm. This paper investigates...

  • A comment from Spyros Makridakis. Makridakis, Spyros // Interfaces;Jan/Feb86, Vol. 16 Issue 1, p103 

    Comments on issues concerning the field of forecasting. Examination of the most appropriate and most accurate way of forecasting; Discussion on large versus small econometric models used in forecasting theory; Information about various methods of forecasting.

  • Do probabilistic forecasts lead to better decisions? Ramos, M. H.; van Andel, S. J.; Pappenberger, F. // Hydrology & Earth System Sciences Discussions;2012, Vol. 9 Issue 12, p13569 

    The last decade has seen growing research in producing probabilistic hydro-meteorological forecasts and increasing their reliability. This followed the promise that, supplied with information about uncertainty, people would take better risk-based decisions. In recent years, therefore, research...

  • Life Cycle Costing Decision Analysis for Electric Multiple Units Based on Reliability. Zhang Kexin; Yao Jianwei; Zhao Zeping // Applied Mechanics & Materials;2014, Issue 644-650, p6274 

    According to the relationship between reliability and Life Cycle Costing (LCC), we studied the best optimized solution of system reliability and LCC when researching and manufacturing EMU subsystem, choosing the overall system scheme and the purchase scheme. Classified by the different stages of...

  • The Competitive Marketing Simulator--A New Management Tool. Kotler, Philip // California Management Review;Spring65, Vol. 7 Issue 3, p49 

    Of all the functional areas, marketing has been marked by the least analytical progress. This is not due to executive apathy, but to the difficult nature of marketing decisions. This new model should suggest better ways to conceptualize marketing. INSET: Illustration of the Interaction among...

  • Place Your Bets!  // Risk Management (00355593);Jan2006, Vol. 53 Issue 1, p8 

    This article discusses role of information aggregation in the creation of methodologies for making predictions about the future behavior of market trends ranging from the likelihood of terrorist attacks to employment rates to political outcomes. The increasing ability of the markets to predict...


Read the Article


Sorry, but this item is not currently available from your library.

Try another library?
Sign out of this library

Other Topics