Student Performance on Conceptual Questions: Does Instruction Matter?

As part of the tutorial component of introductory calculus-based physics at the University of Washington, students take weekly pretests that consist of conceptual questions. Pretests are so named because they precede each tutorial, but they are frequently administered after lecture instruction. Many variables associated with class composition and prior instruction could, in principle, affect student performance. Nonetheless, the results are often found to be ``essentially the same'' in all classes. Selected questions for which we have accumulated thousands of responses, from dozens of classes representing different conditions with respect to the textbook in use, the amount of prior instruction, etc., serve as examples. A preliminary analysis suggests that the variation in performance across all classes is essentially random. No statistically significant difference is observed between results obtained before relevant instruction begins and after it has been completed. The results provide evidence that exposure to concepts in lecture and textbook is not sufficient to ensure an improvement in performance on questions that require qualitative reasoning.