Rethinking the Force Concept Inventory: Developing a Cognitive Diagnostic Assessment to Measure Misconceptions in Newton's Laws

TR Number
Date
2021-10-12
Journal Title
Journal ISSN
Volume Title
Publisher
Virginia Tech
Abstract

Student misconceptions in science are common and may be present even for students who are academically successful. Concept inventories, multiple-choice tests in which the distractors map onto common, previously identified misconceptions, are commonly used by researchers and educators to gauge the prevalence of student misconceptions in science. Distractor analysis of concept inventory responses could be used to create profiles of individual student misconceptions which could provide deeper insight into the phenomenon and provide useful information for instructional planning, but this is rarely done as the inventories are not designed to facilitate it. Researchers in educational measurement have suggested that diagnostic cognitive models (DCMs) could be used to diagnose misconceptions and to create such misconception profiles. DCMs are multidimensional, confirmatory latent class models which are designed to measure the mastery/presence of fine-grained skills/attributes. By replacing the skills/attributes in the model with common misconceptions, DCMs could be used to filter students into misconception profiles based on their responses to concept inventory-like questions. A few researchers have developed new DCMs that are specifically designed to do this and have retrofitted data from existing concept inventories to them. However, cognitive diagnostic assessments, which are likely to display better model fit with DCMs, have not been developed. This project developed a cognitive diagnostic assessment to measure knowledge and misconceptions about Newton's laws and fitted it with the deterministic input noisy-and-gate (DINA) model. Experienced physics instructors assessed content validity and Q-matrix alignment. A pilot test with 100 undergraduates was conducted to assess item quality within a classical test theory framework. The final version of the assessment was field tested with 349 undergraduates. Results showed that response data displayed acceptable fit to the DINA model at the item level, but more questionable fit at the overall model level; that responses to selected items were similar to those given to two items from the Force Concept Inventory; and that, although all students were likely to have misconceptions, those with lower knowledge scores were more likely to have misconceptions.

Description
Keywords
cognitive diagnostic modeling, physics education, DINA, psychometrics, misconceptions, concept inventory
Citation