A metric for measuring the constancy of item response curves

For common assessments, such as the Force Concept Inventory (FCI), we can construct item response curves (IRCs) for each item that show the percentage of students that select each answer choice as a function of overall score. Using IRCs we can test how well items of the concept inventory are functioning in terms of whether certain answer choices are effectively serving as distractors and how many answer choices are truly being selected. We can also rank the answer choices for each item from worst to best. This was done in prior work to construct transition matrices for the FCI were instructions can observe the transitions the students are making as a result of instruction—even when they select a wrong answer on both the pre-test and post-test. One question that came from that work was whether how the answer choices were ranked would depend on the population, i.e., whether the IRCs remained constant for different populations. This work outlines a method that allows for effectively comparing the IRCs between two populations. We apply the method to a set of 12,803 responses to the 25-item Science Literacy Concept Inventory (SLCI).