PE-CAP: Redesigning Legacy Conceptual Inventories: The Impact of Item Response Theory Parameter Dependencies in the Selection of a Scoring Model for Physics Conceptual Inventories

The use of scoring methods in multiple-choice instruments has been a particularly divisive topic, with many researchers preferring weighted sum scores or a latent variable score to a simple sum score. This study looked at the impact of different Item Response Theory (IRT) parameter dependencies in simulated data to better understand the conditions under which a weighted sum score outperforms the simple sum score, or vice versa. Utilizing the two-parameter logistic (2PL) IRT model, the study modeled different dependencies between the item difficulty and item discrimination, then varied the number of test items in order to investigate how both scoring methods performed with increasing test length. These simulated results were then compared with legacy conceptual instruments, the Force Concept Inventory and the Force and Motion Conceptual Evaluation, as well as a new instrument under development, the PE-CAP 1D Kinematics Instrument. These results showcased the impact that the 2PL-IRT parameter dependencies had on the choice of scoring method and should highlight the need to be considerate of the scoring methods employed when investigating these instruments for learning effects in future research.