Assessing Good, Bad and Ugly Arguments Generated by ChatGPT: a New Dataset, its Methodology and Associated Tasks

Author(s)

Nascimento Rocha, Victor Hugo ; Silveira, Igor Cataneo ; Pirozelli, Paulo ; Maua, Denis Deratani ; Cozman, Fabio Gagliardi

• Document type: Journal article
• Source: PROGRESS IN ARTIFICIAL INTELLIGENCE, EPIA 2023, PT I; v. 14115, p. 13-pg., 2023-01-01.

Abstract

The recent success of Large Language Models (LLMs) has sparked concerns about their potential to spread misinformation. As a result, there is a pressing need for tools to identify "fake arguments" generated by such models. To create these tools, examples of texts generated by LLMs are needed. This paper introduces a methodology to obtain good, bad and ugly arguments from argumentative essays produced by ChatGPT, OpenAI's LLM. We then describe a novel dataset containing a set of diverse arguments, ArGPT. We assess the effectiveness of our dataset and establish baselines for several argumentation-related tasks. Finally, we show that the artificially generated data relates well to human argumentation and thus is useful as a tool to train and test systems for the defined tasks. (AU)

• FAPESP's process: 19/07665-4 - Center for Artificial Intelligence
• Grantee: Fabio Gagliardi Cozman
• Support Opportunities: Research Grants - Research Program in eScience and Data Science - Research Centers in Engineering Program
• FAPESP's process: 22/02937-9 - Neural inductive logic programming
• Grantee: Denis Deratani Mauá
• Support Opportunities: Research Grants - Initial Project
• FAPESP's process: 19/26762-0 - Logical Structures in Argumentation
• Grantee: Paulo Pirozelli Almeida Silva
• Support Opportunities: Scholarships in Brazil - Post-Doctoral