Framework for Constructing Concept Inventories from Video Lectures: An Approach Grounded in Automatic Question Generation

• Grant number: 25/19399-8
• Support Opportunities: Regular Research Grants
• Start date: June 01, 2026
• End date: May 31, 2029
• Field of knowledge: Physical Sciences and Mathematics - Computer Science
• Principal Investigator: Rodolfo Jardim de Azevedo
• Grantee: Rodolfo Jardim de Azevedo
• Host Institution: Instituto de Computação (IC). Universidade Estadual de Campinas (UNICAMP). Campinas , SP, Brazil
• City of the host institution: Campinas
• Associated researchers: André Santanchè ; Jacques Wainer ; Julio Cesar dos Reis

Abstract

The increasing use of audiovisual resources in education, particularly video lectures, has transformed pedagogical practices and can contribute to more effective assessment of students' actual learning.In this context, assessment means interpreting evidence to guide pedagogical interventions-an essential step to prevent the persistence of misconceptions that may hinder learning. Specifically, it is desirable that an assessment includes items that allow clear identification of the specific misconception present in the student's knowledge. An organized set of such items constitutes a Concept Inventory (CI), typically arranged as multiple-choice questions in which incorrect answers reveal the main reasoning patterns behind the misconceptions. The development cycle of a CI is complex, as it requires detailed and time-consuming analysis of numerous open-ended responses to identify incorrect answers and their underlying lines of thought.Although validated CIs exist in fields such as Physics and Programming, they are not widespread due to the difficulty of their construction. This project aims to consolidate an integrated and reproducible framework for the automated creation of CIs from video lectures, structured around three main components: (1) Automatic Question Generation (AQG) with the support of Large Language Models (LLMs), (2) systematic analysis of assessment responses automated by software, and (3) construction of CIs with distractors based on real-world data, also supported by LLMs. As a key innovation, the project will use video lectures to generate parameterizable semi-open questions, which allow automatic grading while reducing the likelihood of correct answers by chance. The analysis of error patterns from these questions will be used to automatically generate distractors to form the multiple-choice items.In this project, we expect to (a) build validated question banks for multiple knowledge domains based on video lectures and, from these questions, (b) develop and validate two CIs in distinct subject areas. The knowledge generated will be consolidated into a methodological and software framework with potential applications across various fields. (AU)