Transferability Estimation Scorers for Efficient Transfer Learning

• Grant number: 24/16685-7
• Support Opportunities: Scholarships in Brazil - Doctorate
• Start date: January 01, 2025
• End date: February 28, 2026
• Field of knowledge: Physical Sciences and Mathematics - Computer Science - Computer Systems
• Principal Investigator: Sandra Eliza Fontes de Avila
• Grantee: Levy Gurgel Chaves
• Host Institution: Instituto de Computação (IC). Universidade Estadual de Campinas (UNICAMP). Campinas , SP, Brazil
• Associated research grant: 23/12086-9 - Araceli: Artificial Intelligence in the Fight Against Child Sexual Abuse, AP.R

Abstract

In the evolving landscape of deep learning, selecting the best pre-trained models from a growing number of choices is a challenge. While fine-tuning each pre-trained model and comparing their final accuracy can provide precise rankings for a new task, this greedy approach is often prohibitive due to the substantial computing resources required. Transferability estimation literature proposes alleviating this scenario by relying on a proxy score correlating with the ranks of the fine-tuned performance. However, their recent proliferation, ironically, poses a challenge to their assessment. This doctoral project aims to develop robust benchmark guidelines for transferability scorers' literature, including standardized evaluation measures and pipelines, defining baselines, and a new scorer based on an ensemble of the existing solutions to create a well-founded estimator. By better understanding how different transferability scoring methods perform in various transfer learning scenarios, we can provide recommendations for practitioners. Our preliminary results cover an array of 13 scorers from literature across 11 datasets, comprising generalist, fine-grained, and medical imaging datasets. We expect our research to establish recommendations and guidelines for future research on transferability scorers. Our future research involves expanding the transferability estimation literature to multi-modal vision-language models. Specifically, we plan to design scorers to predict transfer performance in classification and image-text retrieval tasks using Parameter-Efficient Fine-Tuning (PEFT) methods applied to CLIP-based models.