| Full text | |
| Author(s): |
Pirozelli, Paulo
;
Jose, Marcos M.
;
Filho, Paulo de Tarso P.
;
Brandao, Anarosa A. F.
;
Cozman, Fabio G.
Total Authors: 5
|
| Document type: | Journal article |
| Source: | NEURAL-SYMBOLIC LEARNING AND REASONING, PT I, NESY 2024; v. 14979, p. 18-pg., 2024-01-01. |
| Abstract | |
Transformer models have shown impressive abilities in natural language tasks such as text generation and question answering. Still, it is not clear whether these models can successfully conduct a rule-guided task such as logical reasoning. In this paper, we investigate the extent to which encoder-only transformer language models (LMs) can reason according to logical rules. We ask whether these LMs can deduce theorems in propositional calculus and first-order logic, if their relative success in these problems reflects general logical capabilities, and which layers contribute the most to the task. First, we show for several encoder-only LMs that they can be trained, to a reasonable degree, to determine logical validity on various datasets. Next, by cross-probing fine-tuned models on these datasets, we show that LMs have difficulty in transferring their putative logical reasoning ability, which suggests that they may have learned dataset-specific features instead of a general capability. Finally, we conduct a layerwise probing experiment, which shows that the hypothesis classification task is mostly solved through higher layers. (AU) | |
| FAPESP's process: | 19/07665-4 - Center for Artificial Intelligence |
| Grantee: | Fabio Gagliardi Cozman |
| Support Opportunities: | Research Grants - Research Program in eScience and Data Science - Research Centers in Engineering Program |