Busca avançada
Ano de início
Entree


Assessing Logical Reasoning Capabilities of Encoder-Only Transformer Models

Texto completo
Autor(es):
Pirozelli, Paulo ; Jose, Marcos M. ; Filho, Paulo de Tarso P. ; Brandao, Anarosa A. F. ; Cozman, Fabio G.
Número total de Autores: 5
Tipo de documento: Artigo Científico
Fonte: NEURAL-SYMBOLIC LEARNING AND REASONING, PT I, NESY 2024; v. 14979, p. 18-pg., 2024-01-01.
Resumo

Transformer models have shown impressive abilities in natural language tasks such as text generation and question answering. Still, it is not clear whether these models can successfully conduct a rule-guided task such as logical reasoning. In this paper, we investigate the extent to which encoder-only transformer language models (LMs) can reason according to logical rules. We ask whether these LMs can deduce theorems in propositional calculus and first-order logic, if their relative success in these problems reflects general logical capabilities, and which layers contribute the most to the task. First, we show for several encoder-only LMs that they can be trained, to a reasonable degree, to determine logical validity on various datasets. Next, by cross-probing fine-tuned models on these datasets, we show that LMs have difficulty in transferring their putative logical reasoning ability, which suggests that they may have learned dataset-specific features instead of a general capability. Finally, we conduct a layerwise probing experiment, which shows that the hypothesis classification task is mostly solved through higher layers. (AU)

Processo FAPESP: 19/07665-4 - Centro de Inteligência Artificial
Beneficiário:Fabio Gagliardi Cozman
Modalidade de apoio: Auxílio à Pesquisa - Programa eScience e Data Science - Centros de Pesquisa em Engenharia