Validation of Approximate Likelihood and Emulator Models for Computationally Intensive Simulations

Author(s)

Dalmasso, Niccolo ; Lee, Ann B. ; Izbicki, Rafael ; Pospisil, Taylor ; Kim, Ilmun ; Lin, Chieh-An ; Chiappa, S ; Calandra, R

• Document type: Journal article
• Source: INTERNATIONAL CONFERENCE ON MACHINE LEARNING, VOL 162; v. 108, p. 12-pg., 2020-01-01.

Abstract

Complex phenomena in engineering and the sciences are often modeled with computationally intensive feed-forward simulations for which a tractable analytic likelihood does not exist. In these cases, it is sometimes necessary to estimate an approximate likelihood or fit a fast emulator model for efficient statistical inference; such surrogate models include Gaussian synthetic likelihoods and more recently neural density estimators such as autoregressive models and normalizing flows. To date, however, there is no consistent way of quantifying the quality of such a fit. Here we propose a statistical framework that can distinguish any arbitrary misspecified model from the target likelihood, and that in addition can identify with statistical confidence the regions of parameter as well as feature space where the fit is inadequate. At the heart of our approach is a two-sample test that quantifies the quality of the fit at fixed parameter values, and a global test that assesses goodness-of-fit across simulation parameters. While our general framework can incorporate any test statistic or distance metric, we specifically argue for a new two-sample test that can leverage any regression method to attain high power and provide diagnostics in complex data settings. Software for our approach is available on GitHub in Python and R (AU)

• FAPESP's process: 19/11321-9 - Neural networks in statistical inference problems
• Grantee: Rafael Izbicki
• Support Opportunities: Regular Research Grants
• FAPESP's process: 17/03363-8 - Interpretability and efficiency in hypothesis tests
• Grantee: Rafael Izbicki
• Support Opportunities: Regular Research Grants