Toward a Scalable and Transparent Framework for Quantum Benchmarking

Quantum processors are approaching scales where assessing real capability requires rigorous, transparent, and reproducible benchmarking. This talk surveys progress in quantum benchmarking from randomized benchmarking, volumetric and workload-oriented tests, to emerging algorithmic and application-level benchmarks. I will identify several challenges that limit comparability and progress in benchmarking quantum computers: end-to-end transparency for results and provenance, methodologies that remain informative at large scale under sparse sampling, as well as software bottlenecks in defining, dispatching, and analyzing benchmarks in a uniform way across hardware platforms. I will then introduce metriq, an open-source framework that unifies benchmark definition, workload generation, multi-backend dispatch, automated result processing with uncertainty quantification, and publishing with rich metadata for reproducibility. Metriq captures device and run parameters, enforces standardized schemas, and supports community validation and data curation, which enables fair comparisons and longitudinal tracking of progress. Finally, I will discuss ongoing efforts in the transition from physical to logical benchmarking and show how metriq pipelines connect device-level characterization to logical performance on fault-tolerant primitives.