Development of Large Language Models for Applications in the Legal Domain

• Grant number: 23/10100-4
• Support Opportunities: Research Grants - Research in Public Policies
• Start date: May 01, 2024
• End date: April 30, 2028
• Field of knowledge: Physical Sciences and Mathematics - Computer Science - Computing Methodologies and Techniques
• Principal Investigator: Ricardo Marcondes Marcacini
• Grantee: Ricardo Marcondes Marcacini
• Host Institution: Instituto de Ciências Matemáticas e de Computação (ICMC). Universidade de São Paulo (USP). São Carlos , SP, Brazil
• Associated researchers: Diego Furtado Silva ; Diego Raphael Amancio ; Fábio Manoel França Lobato ; Nils Ever Murrugarra Llerena ; Osvaldo Novais de Oliveira Junior ; Silvio Levcovitz ; Solange Oliveira Rezende ; Thiago de Paulo Faleiros
• Associated scholarship(s): 24/18041-0 - Fine-Tuning LLMs with Instruction Fine-Tuning and Zero-Shot Learning Evaluation in the Legal Domain, BP.TT

Abstract

The project proposed under the scope of the Public Policy Research Program (PPPP FAPESP) aims to enhance the public management of the National Treasury Attorney's Office (PGFN) through the use of artificial intelligence (AI) in its legal activities, particularly employing large language models for process analysis. PGFN is responsible for representing the government in tax matters, conducting judicial and administrative collection of tax credits, and providing advisory services to the Ministry of Finance. One of the main current challenges is the manual classification of subjects present in the initial petitions of the processes, which is time-consuming and error-prone. Another challenge is measuring the success of processes in different judicial instances. Currently, PGFN prosecutors need help in systematically identifying the chances of success at each stage of the process, hindering strategic planning and proper resource allocation.Additionally, the project seeks to create an automated process identifier for new subjects that have not been listed. PGFN faces the challenge of agile monitoring and responding to strategies coordinated by large law firms that test new tax theses in different states in Brazil. The manual detection of these theses is challenging due to the volume of incoming processes. Creating an automated identifier would enable PGFN to proactively develop centralized and timely counter-argument models to effectively combat these theses from their inception, avoiding significant financial losses for the Union.To achieve these objectives, a partnership is established with the Institute of Mathematics and Computer Sciences (ICMC) at the University of São Paulo (USP) in São Carlos. The overall idea is to advance in developing large language models (LLM) that are open and permissive. Collaboration with academia allows leveraging technical-scientific knowledge to extend the open LLMs and customize the models to meet the specific needs of PGFN in the legal context, significantly enhancing the effectiveness and efficiency of the system, as well as enabling continuous collaboration and innovations in the legal domain.The project's expected results are divided into short, medium, and long term. In the short term, a comparative evaluation of pre-trained large language models regarding the classification of tax subjects, prediction of judicial process success, and the development of an automated petitioner for counter-arguments and appeals is envisaged. This will be facilitated by developing a prompt engineering module, allowing the pre-trained models to be quickly conditioned to perform specific tasks for PGFN with few labeled examples. In the medium term, the vocabulary adjustment of an LLM using historical textual corpora from PGFN will be developed to enhance the performance of process classification tasks and support success measurement and early identification of new tax theses.In the long term, the project aims to develop a custom LLM for PGFN by fine-tuning a pre-trained LLM, making it more robust for the legal domain. This LLM will be able to handle tasks through zero-shot prompt learning, enabling its use in new tasks without training data. To accomplish such fine-tuning of an LLM, the creation of a high-quality training database is planned, with technical area supervision from PGFN, which is also a relevant scientific outcome, allowing continuity and expansion of research in the legal field and assisting PGFN in its future activities. (AU)