How to Benchmark Quantum Circuits on Simulators and Real Hardware

Overview

A quantum simulator and a quantum processor answer different benchmarking questions. A simulator can provide an ideal reference distribution, model selected noise, and make debugging inexpensive and repeatable. Real hardware exposes effects that a simulator may simplify or omit, including gate errors, readout errors, connectivity constraints, calibration drift, queue delays, and device-specific transpilation choices.

The goal is therefore not to declare one platform “better.” The goal is to separate four questions:

• Correctness: Does the circuit produce the expected output distribution?
• Robustness: How does performance change when noise, depth, or shot count changes?
• Efficiency: How much compilation, execution, and analysis time does the experiment require?
• Reproducibility: Can another developer repeat the run and understand why the results differ?

This distinction matters for quantum programming in Qiskit, Cirq, PennyLane, or another SDK. A circuit that looks identical in source code may be transformed into a different native-gate sequence on each target. Compare the executable circuit, not only the high-level code.

What to track

1. Define a small, representative test set

Start with circuits that isolate specific behaviors instead of beginning with a large application. A practical test set can include a single-qubit rotation, a two-qubit entangling circuit, a repeated-depth circuit, and a small algorithmic circuit relevant to your project. If your work involves a variational quantum eigensolver or another hybrid AI-quantum workflow, include one representative ansatz layer and record the cost-function evaluation separately from classical optimization.

For each circuit, record the number of qubits, logical depth, two-qubit gate count, measurement basis, expected result, and random seed where the SDK supports seeded simulation or compilation. Keep the source circuit under version control and assign each experiment a clear identifier.

2. Record ideal and noisy references

Run the circuit on an ideal quantum simulator when the circuit size permits. This establishes the expected probability distribution without hardware noise. If practical, add a noise-model simulation using documented assumptions. A noisy simulator is useful for testing whether a hardware result is directionally consistent with an error model, but it is not a substitute for measuring the device.

Do not compare only the most likely bitstring. Store the full measured counts or probability distribution. Two runs may have the same winning outcome while differing substantially in the rest of the distribution.

3. Control shot counts

Use the same shot count when comparing otherwise equivalent executions, or clearly label why counts differ. More shots reduce statistical uncertainty but do not remove systematic errors. A low-count result can make a stable circuit appear inconsistent, while a high-count result can make a small hardware bias easier to see.

For repeated runs, use a fixed baseline shot count and an optional higher-count confirmation run. Avoid changing shots, circuit depth, and backend at the same time unless the purpose is to study scaling.

4. Capture compilation and execution details

Record the transpiled or compiled circuit, including its physical-qubit mapping, native gate count, depth, inserted routing operations, and optimization level or equivalent compiler setting. These fields often explain why a hardware run is worse than an ideal simulation. A two-qubit interaction that requires routing can become several operations on a device with limited connectivity.

Track wall-clock submission time, queue time when available, device execution time, and local post-processing time as separate values. Queue time describes access to a shared service; it is not a measure of circuit quality. Provider terminology and available runtime fields can change, so preserve the raw job metadata alongside your normalized columns.

5. Choose accuracy metrics deliberately

Use a metric that matches the experiment. For a known target distribution, consider total variation distance, a distribution-level similarity measure, or another clearly defined distance. For a state-preparation experiment, fidelity may be appropriate when the required state information or tomography method is available. For a computational task, track success probability or approximation error against a classical reference where one exists.

Also report readout error separately when possible. A low success rate can result from incorrect gates, incorrect measurement, insufficient shots, or an unsuitable compilation. A single “fidelity” number should not hide these distinct sources.

Simulator versus hardware comparison

Experiment log template

Use a spreadsheet, database, or machine-readable file with one row per job. At minimum, include:

experiment_id,
circuit_version,
source_commit,
qubits,
logical_depth,
transpiled_depth,
two_qubit_gate_count,
backend_or_simulator,
noise_model,
shots,
seed,
compiler_settings,
physical_qubit_mapping,
queue_time,
execution_time,
counts_or_result_path,
reference_result,
accuracy_metric,
metric_value,
run_timestamp,
job_id,
notes

Store raw counts and compiled circuit files rather than relying only on summary metrics. This makes later analysis possible when you revise the distance measure or discover a compilation difference.

Cadence and checkpoints

Benchmarking is most useful as a recurring process, not a one-time screenshot. Establish a baseline when a circuit is first implemented, then repeat it whenever a meaningful variable changes.

• At initial implementation: Run the ideal simulator, a controlled noisy simulation if available, and at least one hardware comparison when access and project requirements permit.
• After circuit changes: Re-run the baseline before evaluating whether an optimization helped. A shorter source circuit can still produce a longer hardware circuit after transpilation.
• After SDK or compiler changes: Record the software versions and compare compiled depth, gate composition, and result quality.
• Monthly or quarterly: Repeat a small smoke-test suite on the backends your team uses. This helps identify changes in availability, timing, or observed performance without requiring a full application run.
• Before a release or paper: Freeze circuit versions, settings, backend identifiers, and analysis code. Run repeated jobs rather than relying on a single hardware result.

A fair repeated-run design keeps the circuit, shot count, measurement method, and analysis constant. If you change one of these, create a new experiment group instead of mixing results. For hardware, schedule multiple runs when possible and report the spread, not only the best outcome.

How to interpret changes

Begin with the compiled circuit. If hardware accuracy declines while transpiled depth or two-qubit gate count increases, the most immediate explanation may be compilation overhead rather than a change in the algorithm. Compare the physical mapping and native operations before attributing the result to the processor itself.

Next, separate statistical variation from systematic change. Shot noise usually appears as variation in measured frequencies that becomes smaller with more shots. A stable, directional bias across repeated runs suggests a different issue, such as readout or gate error. Do not label a change meaningful until it exceeds the variability seen in repeated baseline runs.

Queue time deserves separate treatment. A longer wait can affect project throughput and cost, but it does not directly indicate lower circuit fidelity. Likewise, simulator runtime depends on the simulation method, machine resources, circuit size, and configuration; it should not be presented as a direct proxy for quantum hardware speed.

When a result changes, use a simple diagnostic order:

1. Confirm that the source circuit and measurement basis are unchanged.
2. Compare SDK, compiler, simulator, and backend identifiers.
3. Inspect transpiled depth, native gate counts, and qubit mapping.
4. Check shot count, random seeds, and noise-model settings.
5. Separate queue, execution, and post-processing times.
6. Repeat the run and compare distributions, not just one output.

If the experiment becomes difficult to explain, use the debugging workflow in Quantum Circuit Debugging: A Practical Checklist for Common Qiskit, Cirq, and PennyLane Errors. For reproducible notebooks, see Quantum Computing with Jupyter Notebooks: Best Practices for Reproducible Experiments.

When to revisit

Revisit this benchmark whenever a provider changes backend specifications, access limits, simulator features, pricing terminology, compiler behavior, or job metadata. These details are operational and can change independently of the quantum algorithm. Update your experiment log schema when a platform introduces a new timing field or renames an execution metric, but preserve the original raw response for auditability.

Review the benchmark monthly or quarterly if the circuit supports an active project. A shorter review may be appropriate before a deployment, hardware comparison, publication, or major optimization. At each review, confirm that the test circuits still represent the workload, the accuracy metric still answers the project question, and the selected simulator remains a valid reference.

Use this final checklist before trusting a comparison:

• Is the circuit version identified and reproducible?
• Are simulator and hardware runs using comparable shots and measurements?
• Have ideal, noisy, and hardware results been kept distinct?
• Are transpiled depth, native gates, and qubit mapping recorded?
• Are queue and execution times reported separately?
• Were multiple hardware runs used where variability matters?
• Is the accuracy metric defined and supported by raw result data?
• Are software versions, backend identifiers, and timestamps stored?

Once the baseline is stable, connect it to the rest of your development process. The guide to building a hybrid quantum-classical workflow in Python can help place benchmark jobs inside a larger pipeline, while the Quantum API and SDK Release Tracker provides a useful prompt for reviewing toolchain changes. A benchmark that preserves context is far more valuable than a single impressive result: it tells you what changed, why it changed, and whether the change matters for the workload you actually intend to run.