Benchmarking Quantum Programs

What a benchmark actually is

A benchmark is a reproducible comparison under stated assumptions. For quantum programs, the assumptions are where the claim hides: the classical baseline, the circuit cost, oracle and data access, the noise model, the hardware target, the shot count, and the exact code version. Leave one out and the result cannot be rerun — it can only be admired.

The benchmark card answers six questions:

• What problem is being solved?
• What classical method is the baseline?
• What quantum circuit or workflow is used?
• What resources — qubits, depth, shots — does it require?
• How large is the output uncertainty?
• What evidence label does the result justify?

A card that answers all six can be challenged. A card that answers three can only be believed.

The two numbers every card needs

Shot estimates carry sampling error. For $S$ shots estimating a probability $p$, the standard error is approximately $\mathrm{SE}(\hat{p})\approx \sqrt{p(1-p)/S}$. This one formula kills a whole genre of overclaim: with 1,000 shots you cannot resolve a probability much finer than ±0.016, and with 100 shots the wobble is ±0.05. If the claimed advantage is smaller than the error bar, you do not have a result yet — you have a shot budget problem.

For application-level framing, score with explicit subtractions: $application\_score=evidence\_strength-baseline\_gap-integration\_risk-hardware\_risk$. The score is a discipline tool, not a verdict machine. It forces a memo to name its evidence and its risks instead of telling a one-sided story.

Worked example: two cards for the same run

Weak benchmark: "Quantum method achieved high success on a toy instance." Nothing in that sentence can be rerun or compared. Now the same run, written as a benchmark card:

• Task: four-item Grover search.
• Baseline: classical unstructured search under the same oracle model.
• Quantum resources: qubits, oracle calls, diffusion iterations, shot count.
• Evidence: ideal simulator, not hardware.
• Uncertainty: finite-shot sampling at the recorded $S$.
• Limitation: a toy instance does not prove application utility.
• Decision label: learning benchmark, not product benchmark.

The second card is not more impressive. It is more useful, because it can be compared, rerun, and challenged.

Where benchmarks go wrong

The standing trap is benchmark theater: impressive plots with no baseline, no metadata, and no rerun path. A quieter trap is the straw baseline — a tuned quantum workflow measured against an untuned or outdated classical method. Both produce charts that shrink under inspection.

Benchmarks should make claims smaller and stronger. A narrow result that reproduces is worth more than a broad one nobody can inspect.

The engineering view

A benchmark is a test harness plus a claim boundary. The harness needs input generation, runtime assumptions, output checks, result storage, and interpretation rules; the boundary states what the harness cannot show. The artifact should record:

• source code path, command, and environment
• random seeds and the backend or simulator used
• resource counts and the classical baseline
• expected output and how to interpret the result
• limitations, stated plainly

Make the evidence label explicit: teaching benchmark, simulator benchmark, noisy-simulator benchmark, hardware smoke test, hardware benchmark, application benchmark. Labels stop a small learning artifact from being promoted into a market claim, and the same circuit can climb the ladder as the evidence improves.

Record negative evidence too. If the classical baseline wins, write it down. If transpilation makes the circuit too deep, write it down. If shot cost dominates, write it down. The purpose of the card is decision quality, not marketing.

What this buys you in diligence

When you review a benchmark, ask four things: is the baseline current, does the task matter to a buyer, is the quantum circuit cost recorded, and does the result label match the evidence. A good benchmark can still justify a wait decision if hardware risk stays high — the card makes that visible instead of hiding it.

Prefer teams that publish benchmark failures clearly. A team that can explain why a method is not ready usually has more technical discipline than one that only shows curated successes.

Exercise

Application baseline scorecard. Select one algorithm or lab from this book and write its benchmark card.

• Record: task, code path, command, backend, shots, seed, circuit resources, baseline, output, and limitation.
• Check: confirm the classical baseline appears in the card before any quantum-advantage claim, and that every sampled output carries its shot standard error.
• Submit: the card plus a one-paragraph evidence label — learning-only, monitor, build, partner, wait, or avoid — defended from the recorded numbers.
• Repair: if your card opens with the quantum method or lacks rerun metadata, rework it after Chapter 41 (Noise Models and Noisy Simulation), which covers reproducible simulator output.

Check your understanding

Without notes: draft a benchmark card for one algorithm you know, including the classical baseline, the circuit cost, and the evidence label.

Oral defense: explain to a colleague why a 1,000-shot run cannot distinguish a success probability of 0.501 from 0.499 — and what you would change so that it could.

If you get stuck

If your benchmark output, shot assumptions, or baseline evidence cannot be reproduced, return to Chapter 41 (Noise Models and Noisy Simulation) and rebuild the rerun path first. Benchmarking skill is mostly reproducibility skill wearing a decision-maker's hat.