Benchmarks for Pirá 2.0, a Reading Comprehension Dataset about the Ocean, the Brazilian Coast, and Climate Change

Author(s)

Pirozelli, Paulo ; Jose, Marcos M. ; Silveira, Igor ; Nakasato, Flavio ; Peres, Sarajane M. ; Brandao, Anarosa A. F. ; Costa, Anna H. R. ; Cozman, Fabio G.

• Document type: Journal article
• Source: DATA INTELLIGENCE; v. 6, n. 1, p. 35-pg., 2024-02-01.

Abstract

Pira is a reading comprehension dataset focused on the ocean, the Brazilian coast, and climate change, built from a collection of scientific abstracts and reports on these topics. This dataset represents a versatile language resource, particularly useful for testing the ability of current machine learning models to acquire expert scientific knowledge. Despite its potential, a detailed set of baselines has not yet been developed for Pira. By creating these baselines, researchers can more easily utilize Pira as a resource for testing machine learning models across a wide range of question answering tasks. In this paper, we define six benchmarks over the Pira dataset, covering closed generative question answering, machine reading comprehension, information retrieval, open question answering, answer triggering, and multiple choice question answering. As part of this effort, we have also produced a curated version of the original dataset, where we fixed a number of grammar issues, repetitions, and other shortcomings. Furthermore, the dataset has been extended in several new directions, so as to face the aforementioned benchmarks: translation of supporting texts from English into Portuguese, classification labels for answerability, automatic paraphrases of questions and answers, and multiple choice candidates. The results described in this paper provide several points of reference for researchers interested in exploring the challenges provided by the Pira dataset. (AU)

• FAPESP's process: 19/07665-4 - Center for Artificial Intelligence
• Grantee: Fabio Gagliardi Cozman
• Support Opportunities: Research Grants - Research Program in eScience and Data Science - Research Centers in Engineering Program
• FAPESP's process: 19/26762-0 - Logical Structures in Argumentation
• Grantee: Paulo Pirozelli Almeida Silva
• Support Opportunities: Scholarships in Brazil - Post-Doctoral