Multi-Agent Dynamic 3D Open-vocabulary SLAM and Scene Understanding

• Grant number: 25/02594-2
• Support Opportunities: Scholarships in Brazil - Doctorate
• Start date: April 01, 2025
• End date: March 31, 2028
• Field of knowledge: Engineering - Electrical Engineering
• Principal Investigator: Eric Rohmer
• Grantee: César Bastos da Silva
• Host Institution: Faculdade de Engenharia Elétrica e de Computação (FEEC). Universidade Estadual de Campinas (UNICAMP). Campinas , SP, Brazil
• Company: Universidade Estadual de Campinas (UNICAMP). Faculdade de Engenharia Elétrica e de Computação (FEEC)
• Associated research grant: 21/00199-8 - SMART NEtworks and ServiceS for 2030 (SMARTNESS), AP.PCPE

Abstract

This research proposes a Multi-Agent Dynamic 3D Open-Vocabulary SLAM framework thatleverages advanced vision-language models (VLMs) and large language models (LLMs) toenhance robotic scene understanding and task planning in dynamic, unstructuredenvironments. By integrating 2D open-vocabulary segmentation with depth and multi-sensordata, the system builds a hierarchical 3D scene graph for precise object clustering, roomsegmentation, and semantic interpretation. Key contributions include dynamic dataassociation to filter non-static objects, improving SLAM accuracy and efficiency, and a robustdata fusion method for enhanced mapping precision. This flexible approach aims to supportnatural language communication and reduce localization drift, making it suitable for diverseassistive and industrial robotics applications.