Markov decision process and risk

• Grant number: 18/11236-9
• Support Opportunities: Regular Research Grants
• Start date: May 01, 2019
• End date: April 30, 2021
• Field of knowledge: Physical Sciences and Mathematics - Computer Science - Computing Methodologies and Techniques
• Principal Investigator: Karina Valdivia Delgado
• Grantee: Karina Valdivia Delgado
• Host Institution: Escola de Artes, Ciências e Humanidades (EACH). Universidade de São Paulo (USP). São Paulo , SP, Brazil

Abstract

Markov Decision Processes (MDPs) are widely used to solve sequential decision-making problems. The most commonly used performance criterion to find a solution in this type of problem is to minimize the expected total cost. However, this approach does not take into account the cost variability (i.e., fluctuations around the mean), which can significantly affect the policy performance. MDPs that deal with this type of problem are called risk-sensitive MDPs. Among the risk-sensitive MDPs, we have: (i) MDPs that use the expected exponential utility as the optimization criterion; (ii) MDPs whose goal is to maximize the probability that the cumulative cost is no greater than a given user-defined threshold, called MDPs with limited budget; (iii) MDPs whose criterion includes the CVaR metric, a robust measure of risk commonly used in the financial area, called CVaR MDPs; and (iv) MDPs whose criterion uses the expected total cost in conjunction with the CVaR criterion, called mean-CVAR MDPs. In this research project, we will work with MDPs with limited budget, CVar MDPs and mean-CVaR MDPs. The main objective is to propose accurate and approximate algorithms based on dynamic programming to solve these risk sensitive MDPs. (AU)

Scientific publications

(References retrieved automatically from Web of Science and SciELO through information on FAPESP grants and their corresponding numbers as mentioned in the publications by the authors)

PASTOR, HENRIQUE DIAS; BORGES, IGOR OLIVEIRA; FREIRE, VALDINEI; DELGADO, KARINA VALDIVIA; DE BARROS, LELIANE NUNES; MARTINEZVILLASENOR, L; HERRERAALCANTARA, O; PONCE, H; CASTROESPINOZA, FA. Risk-Sensitive Piecewise-Linear Policy Iteration for Stochastic Shortest Path Markov Decision Processes. ADVANCES IN SOFT COMPUTING, MICAI 2020, PT I, v. 12468, p. 13-pg., 2020-01-01. (18/11236-9)

PEREIRA NETO, EDUARDO LOPES; FREIRE, VALDINEI; DELGADO, KARINA VALDIVIA; MARTINEZVILLASENOR, L; HERRERAALCANTARA, O; PONCE, H; CASTROESPINOZA, FA. Risk Sensitive Markov Decision Process for Portfolio Management. ADVANCES IN SOFT COMPUTING, MICAI 2020, PT I, v. 12468, p. 13-pg., 2020-01-01. (18/11236-9)

CRISPINO, GABRIEL NUNES; FREIRE, VALDINEI; DELGADO, KARINA VALDIVIA. GUBS criterion: Arbitrary trade-offs between cost and probability-to-goal in stochastic planning based on Expected Utility Theory. ARTIFICIAL INTELLIGENCE, v. 316, p. 45-pg., 2023-03-01. (18/11236-9, 19/07665-4)