Transfer Learning in Reinforcement Learning Multi-Agent Systems

• Grant number: 15/16310-4
• Support Opportunities: Scholarships in Brazil - Doctorate
• Start date: December 01, 2015
• End date: September 30, 2019
• Field of knowledge: Physical Sciences and Mathematics - Computer Science
• Principal Investigator: Anna Helena Reali Costa
• Grantee: Felipe Leno da Silva
• Host Institution: Escola Politécnica (EP). Universidade de São Paulo (USP). São Paulo , SP, Brazil
• Associated scholarship(s): 18/00344-5 - Reusing previous task solutions in multiagent reinforcement learning, BE.EP.DR

Abstract

Reinforcement Learning is a powerful to train intelligent agents because the learning is performed completely autonomously. This learning is accomplished through repetitive interactions between agents and the environment through trial and error, until agents have enough information to properly actuate in order to solve a given task. However, an agent can take a long time to determine which actions are more appropriate for each situation. In order to work around this problem, researchers have started to utilize Transfer Learning solutions in which, after learning a task, the acquired knowledge is reused to accelerate the learning of a new similar task. If multiple agents are acting at the same time in an environment, a fault robust, scalable and highly parallel system can be obtained. However, in this case new problems arise, for example, the state space explosion and the difficulty of predicting consequences of joint actions. Researches proposed partial solutions for these problems, in which Transfer Learning has been proved to be beneficial also to multi-agent domains. Yet, the existing Transfer Learning methods must be improved to allow their application in complex domains. This research aims to propose methods, which deal some questions that have been only superficially answered by the state of art methods. Among these question are: How to properly abstract the knowledge acquired in the learning? How to represent this knowledge? How communicate between agents to transmit learned task knowledge? How to deal with partial observability?

Scientific publications

(17)

(References retrieved automatically from Web of Science and SciELO through information on FAPESP grants and their corresponding numbers as mentioned in the publications by the authors)

DA SILVA, FELIPE LENO; REALI COSTA, ANNA HELENA; AAAI. Accelerating Multiagent Reinforcement Learning through Transfer Learning. PROCEEDINGS OF THE TWENTY-NINTH AAAI CONFERENCE ON ARTIFICIAL INTELLIGENCE, v. N/A, p. 2-pg., 2017-01-01. (15/16310-4)

GLATT, RUBEN; DA SILVA, FELIPE LENO; REALI COSTA, ANNA HELENA; IEEE. Towards Knowledge Transfer in Deep Reinforcement Learning. PROCEEDINGS OF 2016 5TH BRAZILIAN CONFERENCE ON INTELLIGENT SYSTEMS (BRACIS 2016), v. N/A, p. 6-pg., 2016-01-01. (15/16310-4)

DA SILVA, FELIPE LENO; REALI COSTA, ANNA HELENA; ACM. Object-Oriented Curriculum Generation for Reinforcement Learning. PROCEEDINGS OF THE 17TH INTERNATIONAL CONFERENCE ON AUTONOMOUS AGENTS AND MULTIAGENT SYSTEMS (AAMAS' 18), v. N/A, p. 9-pg., 2018-01-01. (18/00344-5, 16/21047-3, 15/16310-4)

BONINI, RODRIGO CESAR; DA SILVA, FELIPE LENO; REALI COSTA, ANNA HELENA; AAAI. Learning Options in Multiobjective Reinforcement Learning. PROCEEDINGS OF THE TWENTY-NINTH AAAI CONFERENCE ON ARTIFICIAL INTELLIGENCE, v. N/A, p. 2-pg., 2017-01-01. (15/16310-4)

BONINI, RODRIGO CESAR; DA SILVA, FELIPE LENO; GLATT, RUBEN; SPINA, EDISON; REALI COSTA, ANNA HELENA; IEEE. A Framework to Discover and Reuse Object-Oriented Options in Reinforcement Learning. 2018 7TH BRAZILIAN CONFERENCE ON INTELLIGENT SYSTEMS (BRACIS), v. N/A, p. 6-pg., 2018-01-01. (16/21047-3, 15/16310-4, 18/00344-5)

DA SILVA, FELIPE LENO; REALI COSTA, ANNA HELENA. A Survey on Transfer Learning for Multiagent Reinforcement Learning Systems. JOURNAL OF ARTIFICIAL INTELLIGENCE RESEARCH, v. 64, p. 645-703, 2019. (18/00344-5, 16/21047-3, 15/16310-4)

DA SILVA, FELIPE LENO; TAYLOR, MATTHEW E.; REALI COSTA, ANNA HELENA; LANG, J. Autonomously Reusing Knowledge in Multiagent Reinforcement Learning. PROCEEDINGS OF THE TWENTY-SEVENTH INTERNATIONAL JOINT CONFERENCE ON ARTIFICIAL INTELLIGENCE, v. N/A, p. 7-pg., 2018-01-01. (16/21047-3, 15/16310-4, 18/00344-5)

DA SILVA, FELIPE LENO; GLATT, RUBEN; REALI COSTA, ANNA HELENA; ASSOC COMP MACHINERY. Simultaneously Learning and Advising in Multiagent Reinforcement Learning. 2022 25TH INTERNATIONAL CONFERENCE ON INFORMATION FUSION (FUSION 2022), v. N/A, p. 9-pg., 2017-01-01. (16/21047-3, 15/16310-4)

DA SILVA, FELIPE LENO; NISHIDA, CYNTIA E. H.; ROIJERS, DIEDERIK M.; COSTA, ANNA H. REALI. Coordination of Electric Vehicle Charging Through Multiagent Reinforcement Learning. IEEE TRANSACTIONS ON SMART GRID, v. 11, n. 3, p. 2347-2356, MAY 2020. (15/16310-4, 18/00344-5)

DA SILVA, FELIPE LENO; WARNELL, GARRETT; COSTA, ANNA HELENA REALI; STONE, PETER. Agents teaching agents: a survey on inter-agent transfer learning. AUTONOMOUS AGENTS AND MULTI-AGENT SYSTEMS, v. 34, n. 1, APR 2020. (15/16310-4, 18/00344-5)

DA SILVA, FELIPE LENO; GLATT, RUBEN; REALI COSTA, ANNA HELENA; AAAI. An Advising Framework for Multiagent Reinforcement Learning Systems. PROCEEDINGS OF THE TWENTY-NINTH AAAI CONFERENCE ON ARTIFICIAL INTELLIGENCE, v. N/A, p. 2-pg., 2017-01-01. (15/16310-4)

JACOMINI, RICARDO DE SOUZA; MARTINS, JR., DAVID CORREA; DA SILVA, FELIPE LENO; REALI COSTA, ANNA HELENA. GeNICE: A Novel Framework for Gene Network Inference by Clustering, Exhaustive Search, and Multivariate Analysis. JOURNAL OF COMPUTATIONAL BIOLOGY, v. 24, n. 8, p. 809-830, AUG 2017. (16/21047-3, 11/50761-2, 15/16310-4, 15/01587-0)

PERAFAN VILLOTA, JUAN CARLOS; DA SILVA, FELIPE LENO; JACOMINI, RICARDO DE SOUZA; REALI COSTA, ANNA HELENA. Pairwise registration in indoor environments using adaptive combination of 2D and 3D cues. Image and Vision Computing, v. 69, p. 113-124, JAN 2018. (16/21047-3, 15/16310-4)

DA SILVA, FELIPE LENO; GLATT, RUBEN; REALI COSTA, ANNA HELENA. MOO-MDP: An Object-Oriented Representation for Cooperative Multiagent Reinforcement Learning. IEEE TRANSACTIONS ON CYBERNETICS, v. 49, n. 2, p. 567-579, FEB 2019. (16/21047-3, 15/16310-4)

GLATT, RUBEN; DA SILVA, FELIPE LENO; DA COSTA BIANCHI, REINALDO AUGUSTO; REALI COSTA, ANNA HELENA. DECAF: Deep Case-based Policy Inference for knowledge transfer in Reinforcement Learning. EXPERT SYSTEMS WITH APPLICATIONS, v. 156, OCT 15 2020. (16/21047-3, 15/16310-4, 18/00344-5, 16/18792-9)

DA SILVA, FELIPE LENO; GLATT, RUBEN; REALI COSTA, ANNA HELENA; IEEE. Object-Oriented Reinforcement Learning in Cooperative Multiagent Domains. PROCEEDINGS OF 2016 5TH BRAZILIAN CONFERENCE ON INTELLIGENT SYSTEMS (BRACIS 2016), v. N/A, p. 6-pg., 2016-01-01. (15/16310-4)

DA SILVA, FELIPE LENO; ASSOC COMP MACHINERY. Integrating Agent Advice and Previous Task Solutions in Multiagent Reinforcement Learning. AAMAS '19: PROCEEDINGS OF THE 18TH INTERNATIONAL CONFERENCE ON AUTONOMOUS AGENTS AND MULTIAGENT SYSTEMS, v. N/A, p. 2-pg., 2019-01-01. (15/16310-4, 16/21047-3, 18/00344-5)

Academic Publications

(References retrieved automatically from State of São Paulo Research Institutions)

SILVA, Felipe Leno da. Methods and algorithms for knowledge reuse in multiagent reinforcement learning.. 2019. Doctoral Thesis - Universidade de São Paulo (USP). Escola Politécnica (EP/BC) São Paulo.