How to measure a benchmark EdgeAIStack will publish under your name.

Contents

1. 1. Record the environment
2. 2. Fix the model artefact
3. 3. Run the benchmark command
4. 4. Warm-up and sampling
5. 5. What to submit
6. Example request
7. Review rule
8. Attribution and licensing

1. Record the environment

Before running anything, capture the board's exact state — a number without its environment cannot be reproduced or trusted.

For non-Jetson hardware, capture the equivalent: driver/SDK version (hailortcli fw-control identify for Hailo, rknn_server -v or the RKNN Toolkit2 version for RK3588, the Coral Edge TPU runtime version, the OpenVINO / CUDA driver version for x86), and confirm the board is at idle thermal state before starting. Note whether clocks are locked to maximum — jetson_clocks (no --show) does this on Jetson and is how NVIDIA's own published numbers are captured; state clearly if you did not lock clocks.

2. Fix the model artefact

Pin every variable that changes throughput before you start:

• The exact weights file (e.g. yolo11n.pt → exported .engine, or a specific Hugging Face commit/quant for an LLM).
• Precision (fp32, fp16, bf16, fp8, int8, int4) and, for INT8, whether calibration was done and on what dataset.
• Input size (e.g. 640x640) and batch size — the protocol asks for batch 1 unless you are specifically measuring batched throughput (state it in notes if so).
• Engine build flags: TensorRT trtexec builder flags, ONNX opset, any layer fusion or plugin options used to compile the artefact.

Two runs of "the same model" at different precisions or input sizes are different rows — submit each separately.

3. Run the benchmark command

Use the tool's own benchmark harness rather than a hand-rolled timing loop — it handles warm-up, batching and reporting consistently with how publishers measure. Exact command shapes:

Ultralytics (YOLO family)

TensorRT (any TensorRT-compiled model)

Hailo

llama.cpp (LLM / GGUF)

MLC-LLM

If your runtime is not listed here (ONNX Runtime, OpenVINO's benchmark_app, RKNN Toolkit2's built-in benchmark, a Coral pycoral timing script), use its own official benchmark utility and say which command you ran — the command field of the submission is required precisely so a reviewer can judge any tool.

4. Warm-up and sampling

• Warm up at least 50 iterations before recording anything — the first few inferences on most runtimes are slower (CUDA context setup, kernel autotuning, page faults).
• Sample at least 200 iterations after warm-up. Report the mean fps and the p50 latency at minimum; p99 latency is welcome if your tool reports it.
• Watch tegrastats (or the platform equivalent) during the run for thermal throttling — a clock drop mid-run invalidates the number as a "sustained" figure. Note in notes if you observed throttling and kept the number anyway.
• Run the full benchmark at least twice and confirm the two runs agree within a few percent before submitting; note the run count in samples.

5. What to submit

POST to /api/v1/measurements/submit. Every field, its meaning and its bounds (validated by edgeaistack-api/src/measurement_submission.js):

Example request

Review rule

A submission does not become a published row automatically. A human checks the command against the tool named (does it match the shapes in step 3?), checks the environment fields for internal consistency (does the power mode exist for that hardware? is the JetPack/SDK version real?), and — where an evidence_url is given — checks it. Only after that does the submission become a class-C row in the benchmark database with your attribution shown on the row, the same way a vendor benchmark page is cited. Anything estimated, extrapolated, or that fails review stays out of the database entirely — it is never silently downgraded to a class-D row.

Attribution and licensing

Your attribution (name or handle) is published on the row exactly as submitted; contact is never published. By submitting, you confirm the measurement is your own work and license it to EdgeAIStack under the same CC BY 4.0 terms as the rest of the benchmark database, so it can be redistributed (with attribution) in the dataset download and API responses.