| Texto completo | |
| Autor(es): |
Yang, Jing
;
Glockner, Max
;
Rocha, Anderson
;
Gurevych, Iryna
Número total de Autores: 4
|
| Tipo de documento: | Artigo Científico |
| Fonte: | TRANSACTIONS OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS; v. 13, p. 29-pg., 2025-04-03. |
| Resumo | |
Free-text explanations are expressive and easy to understand, but many datasets lack annotated explanation data, making it challenging to train models for explainable predictions. To address this, we investigate how to use existing explanation datasets for self-rationalization and evaluate models' out-of-distribution (OOD) performance. We fine-tune T5-Large and OLMo-7B models and assess the impact of fine-tuning data quality, the number of fine-tuning samples, and few-shot selection methods. The models are evaluated on 19 diverse OOD datasets across three tasks: natural language inference (NLI), fact-checking, and hallucination detection in abstractive summarization. For the generated explanation evaluation, we conduct a human study on 13 selected models and study its correlation with the Acceptability score (T5-11B) and three other LLM-based reference-free metrics. Human evaluation shows that the Acceptability score correlates most strongly with human judgments, demonstrating its effectiveness in evaluating free-text explanations. Our findings reveal: 1) few annotated examples effectively adapt models for OOD explanation generation; 2) compared to sample selection strategies, fine-tuning data source has a larger impact on OOD performance; and 3) models with higher label prediction accuracy tend to produce better explanations, as reflected by higher Acceptability scores. (AU) | |
| Processo FAPESP: | 19/04053-8 - Reconstrução de eventos a partir de dados visuais heterogêneos |
| Beneficiário: | Jing Yang |
| Modalidade de apoio: | Bolsas no Brasil - Doutorado |
| Processo FAPESP: | 23/12865-8 - Horus: técnicas de inteligência artificial para detecção e análise de realidades sintéticas |
| Beneficiário: | Anderson de Rezende Rocha |
| Modalidade de apoio: | Auxílio à Pesquisa - Temático |
| Processo FAPESP: | 22/05002-0 - Repensando a detecção de notícias falsas para o mundo real |
| Beneficiário: | Jing Yang |
| Modalidade de apoio: | Bolsas no Exterior - Estágio de Pesquisa - Doutorado |