A flexible approach to high-dimensional conditional density estimation

• Grant number: 14/25302-2
• Support Opportunities: Regular Research Grants
• Start date: March 01, 2015
• End date: February 29, 2016
• Field of knowledge: Physical Sciences and Mathematics - Probability and Statistics - Statistics
• Principal Investigator: Rafael Izbicki
• Grantee: Rafael Izbicki
• Host Institution: Centro de Ciências Exatas e de Tecnologia (CCET). Universidade Federal de São Carlos (UFSCAR). São Carlos , SP, Brazil
• Associated researchers: Afrânio Márcio Corrêa Vieira

Abstract

Several fields such as astronomy and economics have an increasing need for high-dimensional nonparametric conditional density estimators. As a case to point out, when estimating how far galaxies are from the Earth using photometric data, conditional densities can lead to substantially more precise estimates of cosmological parameters when compare to regressionestimates. Unfortunately, despite the need for efficient conditional density estimators, thestatistics community has been focusing mostly on creating good estimators of a regression function only. In this project we attempt to fill this gap by proposing a novel approach for estimating a conditional density. In our approach, we take advantage of well known regression estimators. In this way, due to the large literature on regression curve estimation, the nonparametric model we propose is extremely flexible and can have good performance for various problems. In this project we will compare our approach with traditional estimators both on artificial and real datasets. We will also derive rates of convergence, and compare them with minimax rates. Finally, we will implement our method in R, and make it available to the community. The real data we will use include results from the Sloan Digital Sky Survey on the aforementioned cosmological problem, as well as data on the estimation of broilers preslaughter mortality rates, where conditional density estimation plays a key role because of overdispersion. (AU)

Scientific publications

(14)

(References retrieved automatically from Web of Science and SciELO through information on FAPESP grants and their corresponding numbers as mentioned in the publications by the authors)

DA SILVA, GUSTAVO MIRANDA; ESTEVES, LUIS GUSTAVO; FOSSALUZA, VICTOR; IZBICKI, RAFAEL; WECHSLER, SERGIO. A Bayesian Decision-Theoretic Approach to Logically-Consistent Hypothesis Testing. Entropy, v. 17, n. 10, p. 6534-6559, OCT 2015. (14/25302-2, 09/03385-5)

ESTEVES, LUIS G.; IZBICKI, RAFAEL; STERN, JULIO M.; STERN, RAFAEL B.. The Logical Consistency of Simultaneous Agnostic Hypothesis Tests. Entropy, v. 18, n. 7, JUL 2016. (13/07375-0, 14/50279-4, 14/25302-2)

LEE, ANN B.; IZBICKI, RAFAEL. A spectral series approach to high-dimensional nonparametric regression. ELECTRONIC JOURNAL OF STATISTICS, v. 10, n. 1, p. 423-463, 2016. (14/25302-2)

DE ALMEIDA INACIO, MARCO HENRIQUE; IZBICKI, RAFAEL; SALASAR, LUIS ERNESTO. Comparing two populations using Bayesian Fourier series density estimation. COMMUNICATIONS IN STATISTICS-SIMULATION AND COMPUTATION, v. 49, n. 1, p. 261-282, JAN 2 2020. (14/25302-2, 17/03363-8)

ESTEVES, LUIS GUSTAVO; IZBICKI, RAFAEL; STERN, JULIO MICHAEL; STERN, RAFAEL BASSI. Pragmatic Hypotheses in the Evolution of Science. Entropy, v. 21, n. 9, SEP 2019. (14/50279-4, 19/11321-9, 13/07375-0, 17/03363-8, 14/25302-2)

ESTEVES, LUIS GUSTAVO; IZBICKI, RAFAEL; STERN, RAFAEL BASSI. Teaching Decision Theory Proof Strategies Using a Crowdsourcing Problem. AMERICAN STATISTICIAN, v. 71, n. 4, p. 336-343, 2017. (17/03363-8, 14/25302-2)

FOSSALUZA, VICTOR; IZBICKI, RAFAEL; DA SILVA, GUSTAVO MIRANDA; ESTEVES, LUIS GUSTAVO. Coherent Hypothesis Testing. AMERICAN STATISTICIAN, v. 71, n. 3, p. 242-248, 2017. (09/03385-5, 14/25302-2)

IZBICKI, RAFAEL; LEE, ANN B.; FREEMAN, PETER E.. PHOTO-z ESTIMATION: AN EXAMPLE OF NONPARAMETRIC CONDITIONAL DENSITY ESTIMATION UNDER SELECTION BIAS. Annals of Applied Statistics, v. 11, n. 2, p. 698-724, JUN 2017. (17/03363-8, 14/25302-2)

STERN, JULIO MICHAEL; IZBICKI, RAFAEL; ESTEVES, LUIS GUSTAVO; STERN, RAFAEL BASSI. Logically-consistent hypothesis testing and the hexagon of oppositions. LOGIC JOURNAL OF THE IGPL, v. 25, n. 5, p. 741-757, OCT 2017. (13/07375-0, 17/03363-8, 14/50279-4, 14/25302-2)

IZBICKI, RAFAEL; LEE, ANN B.. Converting high-dimensional regression to high-dimensional conditional density estimation. ELECTRONIC JOURNAL OF STATISTICS, v. 11, n. 2, p. 2800-2831, 2017. (17/03363-8, 14/25302-2)

IZSICKI, RAFAEL; LEE, ANN B.. Nonparametric Conditional Density Estimation in a High-Dimensional egression Setting. JOURNAL OF COMPUTATIONAL AND GRAPHICAL STATISTICS, v. 25, n. 4, p. 1297-1316, DEC 2016. (14/25302-2)

DINIZ, MARCIO ALVES; IZBICKI, RAFAEL; LOPES, DANILO; SALASAR, LUIS ERNESTO. Comparing probabilistic predictive models applied to football. Journal of the Operational Research Society, v. 70, n. 5, p. 770-782, MAY 4 2019. (14/25302-2, 17/03363-8)

P. IANISHI; R. IZBICKI. Classificação Morfológica de Galáxias em Conjuntos de Dados Desbalanceados. TEMA (São Carlos), v. 18, n. 1, p. 155-172, 2017-04-00. (14/25302-2)

FREEMAN, P. E.; IZBICKI, R.; LEE, A. B.. A unified framework for constructing, tuning and assessing photometric redshift density estimates in a selection bias setting. Monthly Notices of the Royal Astronomical Society, v. 468, n. 4, p. 4556-4565, JUL 2017. (14/25302-2)