Study of Vision Transformers as texture feature extractors

• Grant number: 25/01057-3
• Support Opportunities: Scholarships in Brazil - Scientific Initiation
• Start date: April 01, 2025
• End date: December 31, 2025
• Field of knowledge: Physical Sciences and Mathematics - Computer Science - Computing Methodologies and Techniques
• Principal Investigator: Lucas Correia Ribas
• Grantee: Luan Bonizi Guerra
• Host Institution: Instituto de Biociências, Letras e Ciências Exatas (IBILCE). Universidade Estadual Paulista (UNESP). Campus de São José do Rio Preto. São José do Rio Preto , SP, Brazil

Abstract

Textures are essential attributes in pattern recognition in images and have practical applications in areas such as medicine, materials science, and botany. Although some textures present simple visual patterns, real textures, such as those found in nature, usually involve non-linear processes in their formation, which increases complexity and makes analysis difficult. In addition, many real-world problems do not have large amounts of data to train specialized and large-scale models. Currently, models based on Vision Transformers (ViTs) have achieved promising results in several computer vision tasks. However, they require high training costs and large amounts of data to achieve this performance. This project aims to study and implement pre-trained and random Vision Transformers for feature extraction in images, integrating these techniques with feature fusion methods, in order to improve texture analysis and classification. In Deep Convolutional Neural Networks (CNNs), it is widely known that the initial layers are better for texture extraction. On the other hand, little is known about the behavior of the different layers of ViTs in this type of task.Studying pre-trained ViTs as feature extractors has the advantage of not requiring additional training, and it is necessary to identify which layers and operations are most appropriate for texture analysis. On the other hand, studying random ViTs is an alternative to seek good performance in visual tasks without high computational costs or long training times, in addition to being important to better understand the impact of the architecture on feature extraction.Thus, it is expected that this project will contribute to a better understanding of the best strategies to employ ViTs in the extraction and analysis of textures in images, with a special focus on material images.