Year
2023
Season
Fall
Paper Type
Master's Thesis
College
College of Computing, Engineering & Construction
Degree Name
Master of Science in Computer and Information Sciences (MS)
Department
Computing
NACO controlled Corporate Body
University of North Florida. School of Computing
First Advisor
Dr. Indika Kahanda
Second Advisor
Dr. Xudong Liu
Third Advisor
Dr. Zornitza Prodanoff
Department Chair
Dr. Asai Asaithambi
College Dean
Dr. William Klostermeyer
Abstract
As education technology continues to evolve, the domains of Automatic Short-Answer Grading (ASAG) and Automated Misconception Detection (AMD) stand at the forefront of innovative approaches to educational assessment. We explore the transformative potential of Large Language Models (LLMs) in revolutionizing these critical areas. Leveraging the remarkable capabilities of LLMs in semantic inference, contextual understanding, and transfer learning, we embark on a comprehensive journey to enhance both ASAG and AMD. On ASAG, we illuminate the efficacy of transfer learning by fine-tuning RoBERTa Large, a state-of-the-art LLM, on task-related corpora, e.g. the Multi-Genre Natural Language Inference (MNLI) corpus. The model's adaptability across unseen questions and domains on the minority class, coupled with its narrowed performance gap from unseen answers, highlights the profound impact of transfer learning on grading diverse student responses. In the emerging realm of AMD, we pioneer a dataset and methodology that inaugurates a new era in misconception detection. Framing the task as Recognizing Textual Entailment (RTE), our approach with RoBERTa Large MNLI captures nuanced misconceptions, unveiling the untapped potential of LLMs in unraveling the intricate landscape of automated misconception detection. The synergy between these endeavors presents a holistic view of the transformative role anticipated for LLMs in automated educational assessment. Our research, spanning adaptability in short-answer grading and groundbreaking advancements in misconception detection, establishes a foundation for a future where LLMs excel not only in understanding nuanced student responses but also in pinpointing and rectifying misconceptions with unparalleled precision. These insights contribute significantly to the dynamic field of educational technology, heralding a new era wherein the full potential of LLMs is utilized to shape the trajectory of educational assessment.
Suggested Citation
Kazi, Nazmul H., "Automated short-answer grading and misconception detection using large language models" (2023). UNF Graduate Theses and Dissertations. 1234.
https://digitalcommons.unf.edu/etd/1234