I measured X. Published is Y. Why?

Contents

1. What this engine answers
2. Inputs and defaults
3. Measured-value normalisation
4. The reference rule
5. The verdict rule and constraint status
6. The ranking rule
7. Cause catalogue
8. Configuration hand-off
9. Submit prefill
10. Evidence classes and confidence
11. What invalidates a result
12. Limitations
13. Method changelog

1. What this engine answers

"I measured 38 fps for YOLO11n INT8 on an Orin Nano Super; the docs say up to 67 fps — is my board broken, my config wrong, or is the published number optimistic?" Benchmark Reality Check does not run a benchmark itself: it takes the number you already measured, looks up what the Benchmark Explorer database (or its class B → C → D estimator fallback) says to expect for the exact same hardware × model family × variant × precision, and reports whether your number is within, below, or above that range — then ranks the reasons a gap of that size is typically caused by, so you check the likely one first instead of re-flashing at random.

2. Inputs and defaults

3. Measured-value normalisation

The engine compares like with like: your measurement is converted into whichever metric the reference uses (fps, latency or tokens/s), not the other way round.

If the value used was derived rather than given directly, the result carries an assumption stating so and flags measured.converted. The reference row decides which metric is compared: LLM/VLM rows compare on tokens_per_s; everything else compares on fps unless the only published metric is latency_ms.

4. The reference rule

The reference is the Benchmark Explorer's expected range for the exact hardware × model family × variant × precision: the min / median / max of every matching row (both evidence classes), the count, and how many of those rows are class C (measured and published). The characteristic runtime, resolution, batch and power mode shown alongside the range come from the single best-evidenced row in that set (class C over D).

When no row exists for the requested precision, the reference falls back to the same estimator hierarchy the Inference Estimator and Benchmark Explorer use — it tries a class-B (EdgeAIStack measurement), then class-C (measured benchmark), then class-D (derived) source in that order, and reports which tier answered plus the whole tier chain it walked. Reference platforms outside the hardware catalog (e.g. NVIDIA A100, T4, Raspberry Pi 5) have no estimator hierarchy; an unmatched precision on those reports no reference (NO_REFERENCE) rather than a guess.

Other precisions. When the requested precision has zero matching rows, the result also lists every other precision that does have rows for the same hardware × family × variant, with its median and best evidence class — so a request for FP8 on a board that only has published INT8 numbers still shows what is actually known.

5. The verdict rule and constraint status

The band is built from the range edges, not the median, with a ±15% tolerance:

Direction-aware for latency. BELOW_RANGE / ABOVE_RANGE describe measured performance relative to the band, not the numeric position of the value. For fps and tokens/s, higher is better, so a measured value under lower is BELOW_RANGE (worse) and over upper is ABOVE_RANGE (better) — direction matches magnitude. For latency, lower is better, so the comparison flips: a latency under lower (faster than the band's low end) is ABOVE_RANGE, and a latency over upper (slower than the band's high end) is BELOW_RANGE. The gap_ratio itself is always the raw measured ÷ median — for latency a ratio above 1 means slower (worse), even though the verdict word is "below".

Constraint status, shown alongside the verdict:

ABOVE_RANGE always resolves to PASS — outperforming the published range is not a constraint violation (though §7's "reference is an estimate" cause still explains it when the reference is class D).

6. The ranking rule

Causes are ranked only for BELOW_RANGE and ABOVE_RANGE verdicts (direction = "below" or "above"); WITHIN_RANGE and NO_REFERENCE rank nothing. For every cause in the library whose direction matches (or is both) and whose applies() guard is true for the request:

Causes are sorted by likelihood descending. weight() is a per-cause function of the request and, where relevant, the Configuration Checker's checks (§8) — for example, "power mode below maximum" weights near zero when you already reported the maximum preset, and weights higher the further your reported preset's wattage is below the maximum. Each ranked cause carries its explanation, its verify command(s) with expected output and source, its remediation, and its sources. The engine also builds an ordered, de-duplicated verify plan across the top causes (up to 8 steps) so you can work down it and stop at the first command that settles the gap.

7. Cause catalogue

Every cause the engine can rank, from edgeaistack-api/src/v2/data/reality_check_causes.js. Evidence class A = a vendor document backs the mechanism (NVIDIA developer guide, TensorRT developer guide); D = the cause itself is about the reference being an EdgeAIStack estimate; E = an engineering heuristic — plausible and widely reported, but not tied to a specific vendor document.

Below range (your number is slower / worse than the band)

Above range (your number is faster / better than the band)

Publisher documents behind the class-A causes

Causes flagged E without a cited document (host-side bottleneck, newer-software, lighter-variant-or-precision) are engineering heuristics stated as such — widely reported directionally, not backed by a specific vendor claim about the magnitude.

8. Configuration hand-off

When both jetpack and (optionally) power_mode are given for Jetson hardware, the engine runs the Jetson Configuration Checker internally and folds its non-PASS checks into the cause weights:

If the Configuration Checker cannot run (non-Jetson hardware, or an internal error), the result carries a warning (Configuration check skipped: …) and the causes that read its checks fall back to their non-configuration weight.

9. Submit prefill

Every result carries a submit_prefill object — hardware, model_family, variant, precision, runtime (yours, or the reference's if you didn't state one), scope, resolution (yours, or the reference's), batch, fps, latency_ms, tokens_per_s, jetpack and power_mode — pre-filling the measurement protocol submission form so a WITHIN_RANGE or ABOVE_RANGE result (or any NO_REFERENCE result) can become a class-C row with one click, without retyping the workload.

10. Evidence classes and confidence

This grades the reference, not the ranking: a HIGH-confidence verdict still ranks causes with the same likelihood × fit rule as a LOW one — confidence tells you how much to trust the number you were compared against, not how much to trust which cause is listed first.

11. What invalidates a result

• A dataset update. The reference is read live from the Benchmark Explorer database; a newer research pass or measurement submission that adds or replaces a row for this hardware × family × variant × precision changes the range and can flip the verdict.
• A class-D reference later measured. Submitting a measurement for a configuration that currently only has a class-D estimate replaces it with a class-C row — an ABOVE_RANGE result explained by "the reference is an estimate" (§7) is exactly the signal that this is about to happen.
• Self-reported fields that don't match reality. clocks_locked, warmup, thermal_state and super_flash are taken as given; the ranking is only as good as what you reported. The verify commands exist precisely to check them independently.
• A JetPack / power-mode configuration that changes. Re-flashing, changing the power mode or moving JetPack versions changes what the Configuration Checker hand-off (§8) reports, which changes the weights on several causes.
• A coverage gap filled by future research — a NO_REFERENCE result today may gain a real reference once a measurement lands for that exact configuration.

12. Limitations

• Causes are ranked heuristics, not measurements. likelihood combines a prior (how often this cause explains a gap of this shape, in general) with a fit score (how well its typical effect band matches your specific gap) — it is not a probability estimated from your board's actual telemetry. The verify command, not the ranking, is what actually settles a cause.
• No thermal model. The thermal-throttling cause reasons from your reported thermal_state, not from a measured temperature curve; it cannot tell you how close to throttling you are, only that enclosed/passive cooling raises the prior.
• End-to-end pipelines need a different engine. This engine's reference rows are inference-only; if you measured a full camera pipeline (decode, pre/post-processing, tracking, encode), use Camera Stream Capacity to budget the pipeline, not this engine to explain the gap.
• Several causes carry a fixed weight regardless of context (host-side bottleneck 0.6, newer-software 0.5, lighter-variant-or-precision 0.5, DLA-vs-GPU 0.25) because no further signal distinguishes cases — they rank on fit to the gap size alone.
• Inherits every limitation of the Benchmark Explorer reference (§13 there): most rows are class D, publisher measurement conditions are not always fully stated, and the cross-hardware comparison is not repeated here.

13. Method changelog

Method changes bump the method version. The underlying reference data (the benchmark database, the JetPack registry, the power-mode registry) versions independently — see the Benchmark Explorer and Jetson Configuration Checker methodology pages for their dataset versions.