Modular Cognitive Architecture for Semantic, Distributed and Affective Coordination in IoRT

• Grant number: 25/11758-9
• Support Opportunities: Scholarships in Brazil - Doctorate
• Start date: August 01, 2025
• End date: July 31, 2029
• Field of knowledge: Engineering - Electrical Engineering - Industrial Electronics, Electronic Systems and Controls
• Principal Investigator: Eric Rohmer
• Grantee: Ervin Alain Bolivar Huayhua
• Host Institution: Faculdade de Engenharia Elétrica e de Computação (FEEC). Universidade Estadual de Campinas (UNICAMP). Campinas , SP, Brazil
• Company: Universidade Estadual de Campinas (UNICAMP). Faculdade de Engenharia Elétrica e de Computação (FEEC)
• Associated research grant: 21/00199-8 - SMART NEtworks and ServiceS for 2030 (SMARTNESS), AP.PCPE

Abstract

Coordinating heterogeneous robotic systems in dynamic, human-centered environments requires cognitive solutions capable of integrating multimodal perception, semantic reasoning, and affective adaptation. This research proposes an architecture that combines language and vision models (LLMs and VLMs) with egocentric perception, enabling robots to understand spatial commands, plan actions at multiple levels, and adjust their behavior according to the user's emotional state.While traditional symbolic models and LLM-based approaches have made progress in task planning, they still face limitations in real-world scenarios-particularly in linking language, visual perception, and motor control. Furthermore, few current systems incorporate emotional personalization as part of the interaction and decision-making loop.The proposal develops six interconnected modules, from semantic grounding of first-person visual inputs to distributed control across physical agents and IoT devices. It includes hierarchical planning strategies using ReAct and RAG, affective modeling, comparative evaluation, and edge deployment. The architecture will be validated in both simulated and real-world environments using robots such as Pepper and Kinova, with open-source protocols, quantitative benchmarks, and user studies. By integrating language, perception, emotion, and control into a single cognitive pipeline, the project aims to contribute to the next generation of intelligent, context-aware, and explainable robots designed for real-world operation. (AU)