Validation and calibration of prediction models

• Grant number: 22/08579-7
• Support Opportunities: Scholarships in Brazil - Doctorate (Direct)
• Start date: October 01, 2022
• End date: September 30, 2027
• Field of knowledge: Physical Sciences and Mathematics - Probability and Statistics - Applied Probability and Statistics
• Principal Investigator: Rafael Izbicki
• Grantee: Luben Miguel Cruz Cabezas
• Host Institution: Centro de Ciências Exatas e de Tecnologia (CCET). Universidade Federal de São Carlos (UFSCAR). São Carlos , SP, Brazil

Abstract

Most supervised machine learning methods yielda point prediction for a target, $Y \in \sY$,based on features, $\X \in \sX$. Recently, a large body of work in statistics and machine learning has been devoted to develop methods that are able to quantify the uncertainty over point predictions. These go from prediction regions to conditional density estimators and quantile regression methods. In order for such methods to be useful, they need to be calibrated, that is, they should properly quantify uncertainty. Although several notions of calibration have been developed for different tasks, they only assess the overall validity of an algorithm. Hence, none of these definitions can properly measure local calibration, which can lead to fairness concerns in several practical problems. In this project, we propose an alternative calibration framework that is specially designed to assess and perform local calibration in both regression and classification, with applications in Dengue nowcasting and forecasting models. (AU)

Scientific publications

(References retrieved automatically from Web of Science and SciELO through information on FAPESP grants and their corresponding numbers as mentioned in the publications by the authors)

CABEZAS, LUBEN M. C.; OTTO, MATEUS P.; IZBICKI, RAFAEL; STERN, RAFAEL B.. Regression trees for fast and adaptive prediction intervals. INFORMATION SCIENCES, v. 686, p. 31-pg., 2024-08-27. (23/07068-1, 13/07699-0, 19/11321-9, 22/08579-7, 21/02178-8)