Quantitative vs qualitative evaluation of automatic segmentation

Automatic segmentation of anatomic regions in medical images has the potential to improve treatment efficiency for image-guided interventions. While many algorithms for automatic segmentation have been developed, evaluation of their clinical usability is largely limited to quantitative metrics such as measures of region overlap (Dice coefficient) or surface distance (Hausdorff distance). Quantitative metrics only tell part of the story; an auto-segmented contour of a small organ, such as an optic nerve, may have a low Dice coefficient but still be clinically acceptable, while a large organ, such as a prostate, may have a high Dice coefficient but be clinically unacceptable. The goal of this work is to explore the use of a qualitative evaluation system for rating the clinical acceptability of auto-segmented contours, and establish the relation between quantitative and qualitative metrics. The qualitative system is designed with 5 levels ranging from “clinically acceptable” to “completely unacceptable” and evaluated by multiple expert observers for pelvic and abdominal structures. If correlation between quantitative and qualitative metrics are found, it would establish scientific basis for the use of quantitative metrics in the evaluation of medical image segmentation.