What Don't You Understand? Identifying Student Misconceptions with LLMs

Overview

This study presents a two-stage approach for identifying and characterizing student misconceptions in online courses, combining quantitative performance analysis with large language model (LLM) assessment. We applied the methodology across 5 online biomedical science courses spanning 9 course periods and 3,802 medical student enrollments, drawing on 40–50 topic-focused quizzes per course.

The first stage identifies consistently challenging topics using quiz-level performance metrics, focusing on first-attempt performance across primarily multiple-choice questions to surface topics that are both central to course objectives and persistently difficult. The second stage uses LLMs to characterize the underlying misconceptions in those high-priority areas, drawing on three data sources together: quiz question content, student response patterns, and lecture transcripts.

This combined approach revealed actionable insights about student misconceptions that were not apparent from performance data alone. Subject matter experts rated the quality of the LLM-identified misconceptions as excellent, and faculty interviews indicated that the data-driven identification of challenging topics corroborated their own classroom observations. The methodology is broadly applicable to learning environments where quizzes are used, and points toward targeted and potentially personalized interventions, with clear pathways for measuring effectiveness through follow-up quiz performance.