Full Stack Benchmarking of Utility Scale Quantum Computers

As quantum hardware and software improves towards advantage and beyond, well-designed full stack benchmarks are critical for gauging progress. Such benchmarks must be designed to proxy for value in applications of interest, inform on the impact of new capabilities, and track low-level system improvements. At the same time, benchmarks must be practical; a benchmarking suite must keep usage at a reasonable level so that it can be run frequently and without monopolizing the device. With that in mind, we are developing a benchmarking suite that probes quantum computers along the three cores axes -- functionality, quality, and timing. The suite answers three relevant questions along these axes: (1) Are hardware and software capabilities, such as dynamic circuits and advanced error mitigation methods, working as expected? (2) Is the fidelity of the benchmarks improving with newer generations of hardware and software? and (3) Are quantum circuit execution times improving? The notion of scale is explicit as the benchmarks are defined at different widths and depths as allowed by the state of the hardware. The suite consists of utility-scale algorithms and applications from chemistry, physics, and optimization, as well as widely accepted benchmarks such as GHZ state generation, whilst aiming to keep QPU usage at a reasonable level. As IBM's hardware and software ecosystem continuously evolves, we have included forward looking tests that will evaluate new capabilities along IBM's roadmap when they become available, and will provide us with clear comparisons across multi-generational device architectures.