Multimodal Fusion for Medical Diagnosis Support: Explainability and Robustness Analysis of Deep Models with Missing Data

• Grant number: 25/14443-9
• Support Opportunities: Scholarships in Brazil - Master
• Start date: December 01, 2025
• End date: August 31, 2027
• Field of knowledge: Physical Sciences and Mathematics - Computer Science - Computer Systems
• Principal Investigator: Agma Juci Machado Traina
• Grantee: Lucas Basolli Borsatto
• Host Institution: Instituto de Ciências Matemáticas e de Computação (ICMC). Universidade de São Paulo (USP). São Carlos , SP, Brazil
• Associated research grant: 24/13328-9 - Intelligent management of multimodal health data for decision-making in big data scenarios: IHealth-MD, AP.TEM

Abstract

In the era of Big Data, integrating multimodal data through deep learning techniques has become essential for advancing biomedical research. Despite significant technological progress in the medical field, diagnostic errors remain a major challenge. This project proposes the use of Multimodal Deep Learning (MDL) to integrate medical images, genomic data, and clinical records with the goal of reducing diagnostic errors and improving disease prediction accuracy. The study focuses on MDL strategies with attention mechanisms, exploring combinations of architectures to integrate multiple public biomedical data modalities while addressing missing data and enhancing model explainability. The methodology includes a systematic literature review, data curation, and preprocessing, with implementation in Python using PyTorch. Various MDL architectures will be explored, including Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Transformers, alongside advanced techniques for handling missing data and model interpretability using t-SNE. Model performance will be evaluated using confusion matrices and metrics such as accuracy, precision, recall, F1-score, and AUC. Expected contributions include selecting the most effective and generalizable MDL architecture for biomedical contexts, identifying key predictive variables per modality and their interactions to enhance explainability, and developing robust solutions for multimodal data integration under missing data scenarios. The outcomes of this research aim to significantly improve diagnostic accuracy and the effectiveness of clinical interventions, positively influencing public health. The proposed solutions will be applicable to complex biomedical domains such as oncology, neurology, and infectious diseases, positioning them as strategic tools for precision medicine. (AU)