The Role of Cognitive Interviews in Redesigning Legacy Conceptual Inventories

A project, the Physics Evidence-Centered Assessment Project (PE-CAP), is underway to use evidence-centered design to create a replacement for the Force Concept Inventory (FCI) and Force and Motion Conceptual Evaluation (FMCE) comprising subscales developed with input from the physics education community. We have begun to collect qualitative data on the first of these subscales, covering concepts in one-dimensional kinematics, through three-step test interviews in which we ask students to think aloud while solving problems, give retrospective probes intended to fill in apparent gaps in observed behavior and stated thought processes, and then elicit student opinions about the problems. In this talk, we'll share how the rich data collected in these interviews are being used to revise and validate items, as well as what we've learned about best practices in writing items and developing figures based on the sometimes surprising ways students approach these problems.