Texture Feature Aggregation and Learning with Vision Transformers and its Applications on Biological and Medical Images

• Grant number: 24/00530-4
• Support Opportunities: Scholarships abroad - Research Internship - Post-doctor
• Start date: April 01, 2024
• End date: June 30, 2024
• Field of knowledge: Physical Sciences and Mathematics - Computer Science - Computing Methodologies and Techniques
• Principal Investigator: Odemir Martinez Bruno
• Grantee: Leonardo Felipe dos Santos Scabini
• Supervisor: Kevin Smith
• Host Institution: Instituto de Física de São Carlos (IFSC). Universidade de São Paulo (USP). São Carlos , SP, Brazil
• Institution abroad: KTH Royal Institute of Technology, Sweden
• Associated to the scholarship: 23/10442-2 - Deep learning for pattern recognition on multi-sensor and multidimensional data, BP.PD

Abstract

Sensors and biosensors play a crucial role in various fields, such as early cancer diagnosis, detection of viruses, contamination in food, water, etc. Intending to develop cost-effective and accurate detection and diagnosis strategies, researchers have explored the use of machine learning techniques and computer vision. Images obtained from these sources possess unique textural properties, and previous research indicates that texture analysis methods hold great promise for characterizing them. Meanwhile, deep learning and computer vision have made remarkable strides recently, with techniques such as Vision Transformers (ViTs) quickly emerging and delivering impressive results. However, ViT models have not yet been thoroughly analyzed for texture analysis. Therefore, this BEPE project proposes novel methods for feature aggregation and extraction from ViTs. Our proposal involves a randomized autoencoder model to extract multi-depth features from a pre-trained ViT, emphasizing texture characterization. The developed methods will also be analyzed in image data from different sources such as microscopy, sensors and biosensors, and other multidisciplinary applications from collaborations at the host university. The stay in a top-international center will also strengthen the collaboration network of our research group.