Benchmarks for Pirá 2.0, a Reading Comprehension Dataset about the Ocean, the Brazilian Coast, and Climate Change

Autor(es)

Pirozelli, Paulo ; Jose, Marcos M. ; Silveira, Igor ; Nakasato, Flavio ; Peres, Sarajane M. ; Brandao, Anarosa A. F. ; Costa, Anna H. R. ; Cozman, Fabio G.

• Tipo de documento: Artigo Científico
• Fonte: DATA INTELLIGENCE; v. 6, n. 1, p. 35-pg., 2024-02-01.

Resumo

Pira is a reading comprehension dataset focused on the ocean, the Brazilian coast, and climate change, built from a collection of scientific abstracts and reports on these topics. This dataset represents a versatile language resource, particularly useful for testing the ability of current machine learning models to acquire expert scientific knowledge. Despite its potential, a detailed set of baselines has not yet been developed for Pira. By creating these baselines, researchers can more easily utilize Pira as a resource for testing machine learning models across a wide range of question answering tasks. In this paper, we define six benchmarks over the Pira dataset, covering closed generative question answering, machine reading comprehension, information retrieval, open question answering, answer triggering, and multiple choice question answering. As part of this effort, we have also produced a curated version of the original dataset, where we fixed a number of grammar issues, repetitions, and other shortcomings. Furthermore, the dataset has been extended in several new directions, so as to face the aforementioned benchmarks: translation of supporting texts from English into Portuguese, classification labels for answerability, automatic paraphrases of questions and answers, and multiple choice candidates. The results described in this paper provide several points of reference for researchers interested in exploring the challenges provided by the Pira dataset. (AU)

• Processo FAPESP: 19/07665-4 - Centro de Inteligência Artificial
• Beneficiário: Fabio Gagliardi Cozman
• Modalidade de apoio: Auxílio à Pesquisa - Programa eScience e Data Science - Centros de Pesquisa em Engenharia
• Processo FAPESP: 19/26762-0 - Estruturas Lógicas em Argumentação
• Beneficiário: Paulo Pirozelli Almeida Silva
• Modalidade de apoio: Bolsas no Brasil - Pós-Doutorado