WIP: Detection of Student Misconceptions of Electrical Circuit Concepts in a Short Answer Question Using NLP

Document Type

Conference Proceeding

Publication Date

7-26-2021

Abstract

While the use of writing exercises in gateway STEM courses that focus on solving numeric problems is not widespread, there is evidence that students could benefit from the addition of such exercises [1]. Writing exercises may be effective in both uncovering student misconceptions that are not necessarily apparent with typical computation problems, and as tools to foster conceptual change and metacognitive skill. In this paper, pilot studies of the use of two Natural Language Processing (NLP) techniques to identify common misconceptions in the writing of students in a course on electric circuit analysis are described. Performance on the writing exercise in question has been shown to correlate with a student's performance in the course [2]. This is of particular interest as the writing exercise has been administered during the fifth class period, sufficiently early to direct additional resources to the success of students appearing to be at-risk for failing the course. Realizing an automated software solution to analyze the responses to this exercise would remove burden on instructor time and open the door to immediate and personalized feedback to the student. The first pilot study was run to determine how successful a simplistic rule-based approach would be in identifying the most common misconceptions found in a writing exercise requiring a student to speculate on the change in the power in the elements of a resistive circuit with a change to a single resistor value. An open-source NLP rule-based matching engine within spaCy [3] was used. The corpus consisted of one hundred and eighty-five unique responses to the question. Precision, recall, and F1-score [4] were used to assess the effectiveness of the rule-based NLP pipeline in comparison to that of a subject matter expert in identifying responses exemplifying seven misconceptions. Should this NLP pipeline be used in a system in which feedback is to be given to the student, a Directed Line of Reasoning (DLR) approach [5] would be beneficial in cases in which identification of a given misconception is in doubt. Considering this pilot study employed an extremely simplistic purely lexical-level rule-based classifier, the results are very promising and suggest the planned approach of developing a highly accurate, advanced rule-based classifier encompassing lexical/syntax/semantic driven rules is viable. As a compliment to the rule-based approach, this paper also describes a pilot study of the use of BERT (Bidirectional Encoder Representations from Transformers) [6], a machine learning approach that has shown tremendous promise in short-answer grading [7].

Publication Title

ASEE Annual Conference and Exposition, Conference Proceedings

E-ISSN

21535965

This document is currently not available here.

Share

COinS