Reproducible Quantum Labs

A lab is a claim attached to an artifact#

Write the claim first: “this package reproduces the ideal Bell vector and seeded sample, routes a distant CNOT on a line, evaluates two one-qubit channels, recovers a planted RB decay, and validates a teaching surface-code model.” Then write exclusions: no hardware calibration, no application advantage, no high-performance simulation, and optional SDK adapters only when their pinned environments are present.

Artifact review distinguishes availability, functionality, reusability, and reproducibility [ACM artifact review guidance]. A passing suite supports only the fixtures it runs. Scientific novelty, hardware accuracy, and external validity require other evidence.

Inventory the companion package#

The artifact includes a license and README; pyproject.toml; dependency-free modules for state-vector simulation, routing, noise, benchmarking, resource sensitivity, algorithms, and error correction; unit tests; and a machine-readable run record. The manifest records each file’s SHA-256, byte length, role, generation status, and test command.

Stable and optional dependencies must be separated. Official Qiskit and Cirq examples age with their APIs and need pinned environments plus scheduled review [Qiskit documentation] [Cirq documentation]. A missing optional environment is an explicit “not evaluated,” not a green check.

Inventory generated and hand-authored files separately. A generated CSV needs its producing command, input hashes, and generator revision; committing only the CSV proves availability, not regeneration. Conversely, rebuilding every fixture during installation can hide the exact evidence that accompanied publication. Keep immutable reference outputs and compare regenerated outputs against them under declared numeric or byte-level rules.

Reproduce the Bell record in a sealed environment#

Create a new virtual environment, install only declared dependencies, run from the recorded directory, and capture platform, Python, command, stdout, exit status, duration in seconds, and hashes. Strip machine-specific absolute paths before publication but preserve enough platform detail to diagnose differences. Recompute exact vector/probabilities and compare the seed-7 transcript byte for byte.

OpenQASM round-trip conformance would support serialization portability, not reconstruction of backend calibration or execution state [OpenQASM 3 specification]. Record that boundary.

“Sealed” means the audit starts without undeclared packages, local source paths, cached generated results, or inherited environment variables. It need not mean disconnected from a package index; network access and fetched artifacts must be logged. After installation, record the resolved dependency graph rather than assuming the input lock alone describes what was installed.

The Bell audit has two comparisons. Complex amplitudes and exact probabilities are checked numerically against the analytic fixture at 10⁻¹². The seed-7 CLI transcript is compared byte for byte only because the package, interpreter behavior, and generator are part of the frozen regression contract. A cross-generator rerun uses the support and distribution rule instead.

Reproducible randomness needs a distribution rule#

A seed provides a repeatable pseudorandom sequence for regression. It does not make observed frequencies mathematical constants. For unseeded or cross-generator replay, store shots, generator family, expected support, expected probabilities, and a predeclared interval or test. Never fail a correct experiment because it did not reproduce exactly 5,000 Bell counts.

Negative fixtures reveal the real boundary#

A useful dossier includes expected failures: non-power-of-two state dimension; non-normalized vector; non-unitary gate; invalid channel probability; even code distance where odd is required; RB data below its fixed floor; and routing to an absent site. Record exception type and a stable message fragment. If these cases succeed, the package is accepting claims outside its domain.

Expected failure is not an ignored test. The harness must reach the intended guard, match the declared exception class and reason, and fail if the call unexpectedly succeeds. Avoid matching an entire traceback, which is brittle across Python versions, but match enough of the stable message to distinguish “invalid probability” from an unrelated file or import error.

Include one tamper test: change a byte in a copied fixture and require the manifest check to fail before execution. Then restore it and rerun. This demonstrates that hashes are actually verified rather than merely printed. The test copy prevents the audit from mutating published evidence.

import hashlib, pathlib
here = pathlib.Path(__file__).resolve() if "__file__" in globals() else pathlib.Path.cwd()
candidates = [pathlib.Path.cwd() / "labs"]
if len(here.parents) > 3:
    candidates.append(here.parents[3] / "labs")
root = next((path for path in candidates if path.is_dir()), None)
assert root is not None, f"labs directory not found in: {candidates}"
required = ["LICENSE", "README.md", "pyproject.toml", "run-record.json",
            "src/quantum_end_to_end/simulator.py", "tests/test_simulator.py"]
def audit_package(package_root, names):
    if not names or len(names) != len(set(names)):
        raise ValueError("inventory must contain unique required paths")
    missing = [name for name in names
               if not (package_root / name).is_file() or (package_root / name).stat().st_size <= 0]
    if missing:
        raise ValueError("missing package paths: " + ",".join(missing))
    hashes = {name: hashlib.sha256((package_root / name).read_bytes()).hexdigest()
              for name in names}
    digest = hashlib.sha256("".join(name + hashes[name] for name in sorted(hashes)).encode()).hexdigest()
    return {"files": len(names), "hashes": hashes, "inventory_digest": digest}

full_result = audit_package(root, required)
metadata_result = audit_package(root, required[:4])
missing_rejected = False
try:
    audit_package(root, [*required, "tests/not-present.py"])
except ValueError:
    missing_rejected = True
assert full_result["files"] == 6 and all(len(value) == 64 for value in full_result["hashes"].values())
assert metadata_result["files"] == 4 and metadata_result["inventory_digest"] != full_result["inventory_digest"]
assert missing_rejected
print(f"PASS: 44 lab dossier files={full_result['files']} digest={full_result['inventory_digest'][:12]} metadata_files={metadata_result['files']} missing={missing_rejected}")

Artifact review and expiry#

Record review date, reviewer environment, commands, outcomes, discrepancies, and blocked components. A stable standard-library fixture may need infrequent review; an SDK adapter has a 180-day window or triggers on dependency release. Never overwrite a failed record. Append a new run and state which claim changed.

The most important red result is an honest one. A package with one recorded limitation is more reusable than a package whose CI silently skipped the dependency that mattered.

A completed local audit should identify its actor as the authoring process and its environment by platform and interpreter; it must not be labeled independent reproduction. The status vocabulary is finite: passed, failed, unavailable, blocked, or expired. Each non-passing status needs a reason and the command that would resolve it. Blank cells are invalid because they are indistinguishable from skipped work.

Archive stdout, stderr, and exit status for every command, including setup. A test suite that passes after an installer emitted a dependency conflict is not clean evidence. The dossier can summarize the run, but its claims link to raw logs and file hashes. Redact credentials and user-specific paths before publication without deleting diagnostic version and platform fields.

The book package audit also checks correspondence: every chapter command points to an existing module or test, every artifact identifier is unique, and every run-record path is relative to the declared root. A green unit suite cannot catch prose that names a removed command. Treat documentation examples as executable interfaces and run them during release validation.

Record failures without contaminating the working evidence. Expected-failure fixtures run in temporary inputs; drift experiments use a copied lock; tamper tests modify a copied artifact. The published manifest and reference records remain immutable. This makes the negative evidence repeatable and prevents an audit from changing the object it claims to inspect.

The handoff to an external evaluator is a one-command entry point plus the dossier, not an implicit shell history. It states expected duration, network needs, optional jobs, platform constraints, and where outputs appear. The evaluator should be able to delete the environment, start again, and obtain the same exact fixtures or the same statistical acceptance decisions without asking the author which cell to run.

A release is blocked when a stable command or manifest hash fails. An optional adapter may ship as unavailable only when that limitation is explicit in the claim boundary and dossier.