Markus Walker, Marcel Reith-Braun, Uwe D. Hanebeck

2025 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI), College Station, TX, USA, 2025, pp. 1-8, Weaknesses of the ANEES and new calibration measures for multivariate predictions

Abstract:

Reliable quantification of uncertainty is crucial for trustworthy predictions in estimation theory and machine learning. However, existing credibility and calibration measures, such as the widely used averaged normalized estimation error squared (ANEES), often exhibit limitations when applied to biased model predictions. In this paper, we systematically review the ANEES and its alternatives, analyze their strengths and weaknesses, and highlight cases where standard measures fail to detect miscalibration. Building on recent advances in calibration measures, we propose two new measures: the generalized uncertainty calibration error and its normalized version. These measures unify and extend the concepts of estimator credibility and regression calibration to multivariate settings. Comprehensive experiments demonstrate the characteristics of credibility and calibration measures, including the proposed measures, and their applicability to regression models.