Benchmarking Quantum Programs

The comparison contract comes before the circuit#

Write the task, input ensemble, success condition, execution layer, compilation policy, excluded runs, and classical baseline before collecting results. If input sizes differ, if one system receives preprocessed answers, or if failed quantum jobs vanish from the denominator, the chart compares pipelines chosen to produce a conclusion rather than systems answering the same question.

Record SDK/backend versions, target identity, transpilation settings, shots, seeds, raw outputs, and calibration date [Qiskit documentation] [Cirq documentation]. Decide uncertainty and stopping rules in advance. A public plot without raw data and analysis code does not meet a rerunnable artifact standard [ACM artifact review guidance].

The unit of observation must be declared. A shot, circuit instance, random sequence, compilation seed, calibration session, and device day are different replication levels. Ten thousand shots from one compiled circuit characterize its measurement distribution; they do not provide ten thousand independent samples of compiler variability or day-to-day device behavior. Confidence intervals that treat nested observations as independent will be too narrow.

Predeclare missing-data policy. Queue timeout, calibration abort, compiler failure, and invalid output are benchmark outcomes unless the contract explicitly assigns them outside the evaluated service. Dropping them only from one system changes the denominator. Preserve a reason-coded exclusion table and publish the result both before and after any allowed exclusions.

Four axes that refuse to collapse#

Capability states which instances can run under declared width, connectivity, memory, and instruction constraints. Quality states success probability, distance, energy, fidelity, or another task metric with uncertainty. Time separates compile, queue, device, sampling, and total wall seconds. Resources report physical/logical qubits, shots, two-qubit operations, memory, energy, or cost under defined denominators.

A machine can be faster and less accurate, capable of a wider circuit and slower end to end, or lower-error at vastly greater physical overhead. Benchmark surveys therefore treat multiple non-equivalent properties and interpretation boundaries [benchmarking review]. A scalar score hides the trade the reader needs to see.

Denominators make the axes interpretable. Success per attempted instance differs from success per completed job. Seconds per shot differs from seconds per statistically accepted answer. Physical qubits differ from logical qubits, and both differ from qubit-seconds. Put the denominator in the column name and reject rows that omit it. Unit conversion cannot repair a different task or success criterion.

Present Pareto comparisons where appropriate. If system A is faster and system B is more accurate, neither dominates until the workload supplies a constraint or utility. A latency ceiling may select A; a minimum fidelity may select B. The benchmark should expose that decision boundary instead of choosing private weights and announcing a winner.

Fit the synthetic RB decay#

The teaching fixture uses single-qubit randomized-benchmarking survival $S(m)=Ap^m+B$, with fixed A=0.4 and B=0.5 in the synthetic generator. Under this convention, error per Clifford is (1−p)/2. The protocol’s decay interpretation is model-dependent, not an application runtime or advantage measure [randomized benchmarking] [benchmarking review].

Subtract B, take logarithms, and fit log(S−B) against sequence length m. For exact synthetic data planted with p=0.98, the slope is log(0.98). Publish lengths, survival values, fixed constants, fit implementation, residuals, and derived convention. A fitted number without those items is not portable.

The log transform changes the error model. Ordinary least squares in log space treats multiplicative residuals as homogeneous, while measured survival fractions have shot-dependent binomial variance in linear space. The exact teaching fixture avoids that complication; experimental analysis must state its likelihood or weighting. Points near B are especially sensitive because a small survival error becomes a large error in $\log(S-B)$, and observations at or below the fixed floor cannot be logged.

Sequence averaging is part of the protocol. For each length, sample multiple random Clifford sequences, execute multiple shots per sequence, and retain both levels. Variation across sequences can exceed finite-shot variation. Pooling all shots before analysis discards that heterogeneity and can make an unstable compilation ensemble look precise.

Under the chapter’s convention, p is a decay parameter and $(1-p)/2$ is an inferred error per Clifford for the stated single-qubit model. It is not automatically error per native gate. A Clifford may compile to different native sequences across targets or dates, so translating the estimate requires a declared decomposition model and additional assumptions.

Break the fit with leakage and drift#

A leakage-like floor shift bends residuals because the assumed B is wrong. Temporal drift makes early and late sequences follow different effective decays. Both datasets can return a numerical slope; neither earns the label “error per Clifford under the model.” The benchmark should fail its own validity gate.

Inspect signed residuals by length and acquisition order. Random scatter around zero supports, but does not prove, the model. Curvature, runs, or split-session offsets trigger a rejection code. Numerical convergence is not model validity.

A concrete floor-shift fixture can add an m-dependent offset that approaches 0.03. Fitting the fixed B=0.5 model forces the slope to absorb the curvature: short sequences pull one way and long sequences another. The residual signs then form a structured arc. The correct output is a failed model check plus the raw curve, not a more precise decimal for p.

For drift, interleave sequence lengths rather than acquiring all short sequences first and long ones last. Randomized acquisition order reduces confounding between m and time, while timestamps preserve the ability to diagnose drift. Split the record at the known intervention or use a time-aware model only if that analysis was specified. Averaging two regimes into one decay parameter describes neither regime.

Leakage may require an observable or outcome space beyond the assumed qubit survival model. If leaked population is coerced into a binary result, the residual test can miss the mechanism. Record invalid or out-of-computational-space outcomes when the platform exposes them, and state when the measurement cannot distinguish leakage from loss or readout error.

from math import log, exp
lengths = [1,2,4,8,16]
valid = [0.4*(0.98**m)+0.5 for m in lengths]
def fit_fixed_model(values):
    xs, ys = lengths, [log((s-.5)/.4) for s in values]
    slope = sum(x*y for x,y in zip(xs,ys))/sum(x*x for x in xs)
    p = exp(slope)
    residuals = [s-(.4*(p**m)+.5) for m,s in zip(xs,values)]
    return p, residuals
p, residuals = fit_fixed_model(valid)
assert abs(p-.98) < 1e-12 and max(map(abs,residuals)) < 1e-12
violated = valid[:]; violated[-1] += .03
_, bad_residuals = fit_fixed_model(violated)
assert max(map(abs,bad_residuals)) > 1e-3
print(f"PASS: 42 benchmark fitted_decay={p:.6f} valid_max_residual={max(map(abs,residuals)):.3g} mutant_max_residual={max(map(abs,bad_residuals)):.6f}")

The classical baseline gets its own tuning budget#

For program benchmarks, name the best relevant classical method and implementation, hardware, precision, initialization, preprocessing, tuning budget, and stopping rule. Give both paths the same task inputs and count all work. A quantum run compared with an untuned textbook baseline establishes only that the baseline was weak.

Keep device characterization separate from application comparison. RB may help explain a device layer; it cannot substitute for task quality, total time, or a classical baseline.

Count preprocessing symmetrically. If a quantum workflow receives a problem-specific embedding, compiled ansatz, or classically optimized parameters, their computation belongs in total time unless the comparison contract explicitly evaluates an amortized repeated-use setting. The baseline receives the same training data, precision target, and wall-clock or compute budget. Initialization and failed restarts remain in the record.

A credible baseline can be a portfolio rather than one algorithm. Choose among relevant exact, heuristic, and approximation methods using a tuning protocol fixed without access to held-out benchmark answers. Reporting only the weakest baseline invites a false advantage claim; reporting an unlimited classical search against a time-limited quantum run is equally uninformative.

A benchmark record that can expire#

Publish environment locks, code/data hashes, target and calibration date, circuit hashes before and after compilation, raw observations, exclusion log, analysis command, uncertainty method, baseline record, and review deadline. Dated hardware and SDK facts expire. The raw record should remain immutable while a later analysis version can be compared against it.

Include a machine-readable validity decision. Fields should name the assumed model, checks performed, thresholds chosen in advance, pass/fail outcome, and reason code. A downstream chart generator must refuse rejected rows by default while still allowing a diagnostic plot. That prevents a numerical result from escaping its failed model check during later aggregation.

Separate acquisition revision from analysis revision. Re-fitting old raw data with corrected code can be legitimate if both versions and changed conclusions are preserved. Replacing raw data after seeing a result is a new acquisition and needs a new identifier. The history is part of the evidence because seemingly minor cleaning choices can decide which system appears best.

Uncertainty should follow the comparison level. Binomial intervals describe shots within a circuit. Bootstrap or hierarchical estimates can capture variation across random sequences, compilation seeds, or sessions when those are the sampled units. A confidence interval over pooled shots cannot support a claim about day-to-day stability. Publish the grouping columns so a second analyst can reconstruct the chosen level.

Timing needs the same decomposition. Record classical preprocessing, circuit construction, compilation, queue, device execution, readout, network transfer, and analysis in seconds. For a service claim, total elapsed time includes queue and retries. For a device-control experiment, an explicitly scoped device duration may be the target. Both can be reported, but substituting one for the other after results arrive invalidates the comparison contract.

The intentionally failed fixture belongs in the primary artifact. Its fit routine returns a finite p, and the validity layer rejects the result because residual structure exceeds the predeclared threshold. CI must assert both facts. If later code suppresses the numerical output or lets it pass, the test fails. This is stronger than documenting, in prose, that models sometimes break.

A benchmark conclusion should be no broader than its input distribution. Success on selected small instances supports performance on that sampled set under the recorded stack. It does not establish asymptotic scaling, another workload family, or advantage over future classical methods. State the population to which the estimate generalizes, then identify the next experiment that would widen it.

Publication tables should carry a validity column beside every estimate. The rejected floor-shift and drift rows remain visible with their finite fitted slopes, but downstream ranking code excludes them. That design proves the validity gate controls interpretation instead of serving as prose that a chart can ignore.