What is, and is not, in the benchmark database.

Contents

1. What the database is, and is not
2. Row schema
3. Sources merged
4. Dedupe rule
5. Evidence classes
6. Expected-range and fallback rules
7. Comparison rule
8. Confidence rule
9. Publishers and documents
10. Dataset statistics
11. Coverage gaps
12. What invalidates a result
13. Limitations
14. Method changelog

1. What the database is, and is not

The database is 567 rows, one per hardware × model × runtime × precision × input × power mode, each carrying a throughput or latency metric with a source. It is inference-only unless a row's scope says end_to_end: publisher numbers are almost always the model forward pass at batch 1, measured with the hardware's clocks locked to maximum (jetson_clocks on Jetson) — not a full camera pipeline (decode, pre/post-processing, tracking, encode). A full pipeline runs slower; use Camera Stream Capacity for the end-to-end budget.

It does not run any benchmark itself. It reads numbers the hardware or model vendor already published, or numbers EdgeAIStack derived from those with the scaling stated, and reports them with their class and source. Where nothing has been measured for an exact configuration, it consults the same benchmark hierarchy the Inference Estimator engine uses (class B → C → D) and reports the fallback with its tier chain — never a silent guess.

2. Row schema

3. Sources merged

The database is built (edgeaistack-api/scripts/build-benchmarks-v2.js) from three inputs:

• Catalog benchmark library (437 rows before dedupe) — EdgeAIStack's existing per-platform benchmark table, sourced from vendor model pages (Ultralytics, Hailo Model Zoo, Google Coral) and NVIDIA-AI-IOT project READMEs.
• Research corpus (118 rows) — the deep-research pipeline's benchmark corpus (tools/stream-calculator/benchmark-corpus-research.json), aggregated from published community and vendor measurements across platforms not otherwise covered.
• LLM/VLM tokens/s (12 rows) — tokens-per-second measurements from the Jetson AI Lab benchmark pass, layered onto the Model Memory Fit registry's Jetson measurements.

Rows without any usable metric (no fps, latency or tokens/s) are dropped at build time.

4. Dedupe rule

Rows are grouped by hardware × model (family + variant) × runtime × precision × input (resolution + batch). Within a group, the power_mode is folded in: a row recorded at the default power mode (the publisher did not state one) merges into the same slot as an explicit power mode for the same configuration, rather than creating a second row — so a class-D estimate made before a power mode was known does not survive alongside the same configuration measured at, say, maxn_super. Rows for genuinely different explicit power modes on the same hardware × model × precision keep separate rows.

Inside one group, the best-evidenced row is kept: higher evidence class wins (C over D), and within the same class, higher stated confidence wins. Everything else in the group is recorded on the kept row's duplicates[] (origin, class, metric value, source doc id) rather than discarded outright — 96 rows were folded this way across the current build.

5. Evidence classes

Only two classes appear in this database today; A and B are reserved.

A (vendor specification) does not apply to a performance number by definition — a spec sheet states a chip's peak TOPS, not a model's measured fps. B (EdgeAIStack measurement) is reserved for rows measured by EdgeAIStack itself under the measurement protocol and reviewed against it; none exist yet. Submit one and it lands here.

6. Expected-range and fallback rules

For a given hardware × model, the expected range per precision is the min / median / max of every matching row (both classes), plus the count and how many of those are class C. This is not a statistical confidence interval — it is the spread of what has actually been reported, which can be wide when sources disagree on measurement conditions (JetPack version, power mode, warm-up).

When a precision has no matching row, the engine calls the same estimator hierarchy the Inference Estimator engine uses: it tries a class-B (EdgeAIStack measurement), then class-C (measured benchmark), then class-D (derived) source in that order, reports which tier answered, and lists the whole tier chain it walked — so a fallback value is never presented as if it were a measured row. Reference platforms outside the hardware catalog (e.g. NVIDIA A100, T4, V100, Raspberry Pi 5) have no estimator hierarchy to fall back to; an unmatched precision on those platforms reports no value rather than a guess.

7. Comparison rule

The cross-hardware comparison takes the best row per hardware for the same model and precision (highest fps or tokens/s; lowest latency if that is the only metric published), across every hardware that has at least one matching row, and ranks them relative to the fastest. It is a comparison of best-published numbers, not a guarantee every platform was measured under the same conditions — check each row's power mode, input size and runtime before treating the ranking as a like-for-like result.

8. Confidence rule

The rule is deliberately coarse — it is a signal to look at the rows, not a statistical measure. A single class-C row against a wall of class-D estimates for other precisions still reports HIGH or MEDIUM overall depending on the row mix for the exact query; narrow the query (hardware + model + precision) to see the mix that produced the label.

9. Publishers and documents

10. Dataset statistics

Full breakdown, always current: benchmarks-v2.json (stats object).

11. Coverage gaps

From the research corpus's own gap analysis (tools/stream-calculator/benchmark-corpus-research.json):

• "jetson_agx_thor - only YOLO26 benchmarks available publicly as of 2026-03; no YOLOv8/YOLO11 data"
• "jetson_orin_nano_4gb - no public benchmarks; expect ~55% of Nano 8GB throughput"
• "jetson_agx_orin_32gb - no distinct benchmarks; interpolate between Orin NX and AGX Orin 64GB"
• "whisper models on AGX Orin / Orin NX - only Nano measured by NVIDIA-AI-IOT/whisper_trt"
• "whisper_small on any edge platform - no published measurements found"
• "coral TPU - YOLOv8 incompatible (anchor-free head not fully Edge TPU compilable)"
• "hailo_8 - yolov8l/x too large for on-chip SRAM without model partitioning"
• "NanoSAM on Orin NX - only Nano and AGX Orin measured"
• "raspberry_pi_5 INT8 CPU benchmarks - no INT8 TFLite delegate benchmarks found for detection"
• "intel_nuc - OpenVINO iGPU benchmarks are community-only; no official Intel NUC AI benchmark suite"

These are exactly the configurations where the expected-range table (§6) falls back to the estimator hierarchy rather than a matched row — a submitted measurement for any of them replaces a fallback with a class-C row.

12. What invalidates a result

• A dataset update. Every result URL carries mv (method version) and dv (dataset version); a newer research pass or measurement submission bumps dv, and a result page built against an older one says so.
• A different power mode, JetPack/SDK version or input size than the row's. Rows are keyed to an exact configuration; a measurement at a different power mode or clock lock can differ meaningfully, especially near thermal limits.
• A class-D row later measured. Submitting a measurement for a configuration that currently only has a class-D estimate replaces it with a class-C row and moves the derived estimate to that row's duplicates[].
• A coverage gap filled by future research (§11) — the fallback chain reported today may shorten once a real measurement lands.

13. Limitations

• 404 of 567 rows (71%) are class D — EdgeAIStack-derived, not independently measured. Treat any figure without a class-C row nearby as a planning estimate.
• No class-A or class-B rows exist yet; class B is reserved for EdgeAIStack's own measurements once the measurement protocol produces some.
• Publisher measurement conditions (exact JetPack/SDK version, thermal state, warm-up length) are not always fully stated on the source page; where a row's runtime is inferred rather than stated, it is flagged runtime_inferred.
• The cross-hardware comparison (§7) does not normalise for different power modes, input sizes or measurement dates across hardware — read each row before treating a ranking as apples-to-apples.
• See §11 for the specific hardware × model combinations known to be unmeasured.

14. Method changelog

Method changes bump the method version; dataset updates (new research passes, new measurement submissions) bump the dataset version. Both appear in the citation block under every result and in the provenance object of every API response.