Combining prompt-based language models and weak supervision for labeling named entity recognition on legal documents

Autor(es)

Oliveira, Vitor ; Nogueira, Gabriel ; Faleiros, Thiago ; Marcacini, Ricardo

• Tipo de documento: Artigo Científico
• Fonte: ARTIFICIAL INTELLIGENCE AND LAW; v. N/A, p. 21-pg., 2024-02-15.

Resumo

Named entity recognition (NER) is a very relevant task for text information retrieval in natural language processing (NLP) problems. Most recent state-of-the-art NER methods require humans to annotate and provide useful data for model training. However, using human power to identify, circumscribe and label entities manually can be very expensive in terms of time, money, and effort. This paper investigates the use of prompt-based language models (OpenAI's GPT-3) and weak supervision in the legal domain. We apply both strategies as alternative approaches to the traditional human-based annotation method, relying on computer power instead human effort for labeling, and subsequently compare model performance between computer and human-generated data. We also introduce combinations of all three mentioned methods (prompt-based, weak supervision, and human annotation), aiming to find ways to maintain high model efficiency and low annotation costs. We showed that, despite human labeling still maintaining better overall performance results, the alternative strategies and their combinations presented themselves as valid options, displaying positive results and similar model scores at lower costs. Final results demonstrate preservation of human-trained models scores averaging 74.0% for GPT-3, 95.6% for weak supervision, 90.7% for GPT + weak supervision combination, and 83.9% for GPT + 30% human-labeling combination. (AU)

• Processo FAPESP: 23/10100-4 - Desenvolvimento de Grandes Modelos de Língua para Aplicações no Domínio Jurídico
• Beneficiário: Ricardo Marcondes Marcacini
• Modalidade de apoio: Auxílio à Pesquisa - Pesquisa em Políticas Públicas