Robust Augmented Retrieval for Natural Language Inference over Transformer-based Models

• Grant number: 24/04890-5
• Support Opportunities: Regular Research Grants
• Start date: January 01, 2025
• End date: December 31, 2028
• Field of knowledge: Physical Sciences and Mathematics - Computer Science
• Principal Investigator: Daniel Carlos Guimarães Pedronette
• Grantee: Daniel Carlos Guimarães Pedronette
• Principal researcher abroad: Andre do Nascimento Freitas
• Institution abroad: Idiap Research Institute, Switzerland
• Host Institution: Instituto de Geociências e Ciências Exatas (IGCE). Universidade Estadual Paulista (UNESP). Campus de Rio Claro. Rio Claro , SP, Brazil
• Associated researchers: Ivan Rizzo Guilherme ; João Paulo Papa ; Jonas Felipe Pereira de Queiroz
• Associated scholarship(s): 25/03917-0 - Comparative Analysis of Contextual Ranking Methods for Textual Retrieval, BP.TT; 25/01118-2 - Contextual Ranking Models for Retrieval Augmented Natural Language Inference, BP.PD

Abstract

Large Language Models (LLMs) have emerged as universal models for language understanding and inference. In order to operate factually, these models need to be provided with a relevant associated factual context. Retrieval Augmented Generation (RAG) became the complementary architectural mechanism which enables LLMs to operate factually, in which a retrieval step delivers the associated facts relevant within a specific task. However, there is a disconnect between the performance of retrieval models, which are grounded on textual embeddings (comparatively lower semantic granularity and performance) and the capacity of LLMs to interpret and operate over these results (higher performance). As a result, LLMs are currently bound to the comparatively lower performance of RAGs. Moreover, textual retrieval tasks tends to be based on the notion of sentence/passage similarity, where it is assumed that the query is textually similar to the target text to be retrieved. In application scenarios that involves real-world evidence-based reasoning (e.g. in science, policy making), there are interpretation challenges which compound with the lower semantic granularity of RAG models, namely: (i) abstraction gap: abstraction and semantic gap between the query intent (as the information need is expressed by the end-user) and the way the evidence is expressed in the reference corpus; (ii) domain specificity: evidence-based reasoning is mostly relevant with specialised domains, requiring the associated background knowledge and specialisation; (iii) joint quantitative and qualitative reasoning: where a significant part of evidence-based reasoning requires both language-based and quantitative/numerical inference, over both textual and tabular data sources. These bottlenecks prevent the wide-application of LLMs to support evidence-based reasoning within high-impact application domains such as drug discovery, clinical trials design, policy making, fact checking, investigative intelligence, among other areas. The complexity of making sense from the growing number of available studies, introduces pragmatic barriers for the adoption of rigorous evidence-based reasoning within scientific and policy-making. The project aims to bridge this gap by developing novel evidence-based natural language inference (NLI) mechanisms targeting reasoning over large evidence-spaces. Abductive inference, or inference to the best explanation, allows for the formulation of a systematic reasoning process which dialogues with the set of available evidence, where the more controlled deductive or inductive inference processes cannot be applied. Given a certain hypothesis, question or claim, abductive reasoning will bridge the gap with available evidence, selecting the most likely from competing explanations, which can either corroborate or refute that hypothesis. (AU)