Model Match — methodology

Contents

1. What it decides, and what it does not
2. The candidate universe
3. The canonical model map
4. What each candidate is evaluated on
5. Reference accuracy and quantised accuracy
6. The vision memory model
7. The filter, and how UNKNOWN is treated
8. The four terms and the weight table
9. Winner, roles and ties
10. What the registries actually contain
11. What invalidates a result
12. Method changelog

1. What it decides, and what it does not

Model Match answers one question: on the platform you already have, under this load and these constraints, which model should you run — and why that one and not the next. It returns a ranked list of every model with a benchmark row for that silicon, the four term scores and the weight table that produced each total, the checks behind the winner, every excluded model with the constraint that excluded it, and the accuracy of each candidate with the metric and the evaluation set named.

It does not recommend a platform. When the question is which platform, that is Hardware Match. It does not compute a throughput, a latency or a memory figure of its own either: it owns exactly two things, the candidate universe and the ranking. Everything inside a candidate comes from an engine or a registry that already owns it, and re-running that engine directly with the same inputs reproduces the same numbers — which is what makes the sentence "the winner fits" checkable rather than asserted.

It also does not model the rest of the pipeline. The throughput figures are inference-only; decode, preprocessing, tracking and encoding are Camera Stream Capacity's question.

2. The candidate universe

A model is a candidate on a platform if and only if the benchmark database holds at least one row for it on that platform. The candidate is the triple (model, variant, precision), which is exactly how the database records a measurement, and its universe size is a fact about coverage rather than a design choice: 90 triples on the Jetson T5000, 58 on the Orin Nano Super, 48 on the Orin NX 16GB, 11 on the Orin NX 8GB — and zero on the Jetson T4000, which returns an error naming the platform and the gap instead of a list.

The alternative rule — admit any model and model the ones nobody measured — was rejected. A modelled throughput is labelled as modelled by Compute Fit and can never confirm a fit, so every added row would be a candidate that cannot win on the strength of its own evidence; and a comparison whose deciding number was invented is worse than a shorter list. A model with no row here is reported in the gaps, never as a silently modelled entry.

Filters narrow the universe before evaluation, never after ranking: a task, a precision, a list of families to keep or to drop. When the task is one of the three language tasks — text generation, vision + language, speech recognition — the universe is instead the class A model-memory registry of that type at the quantisation asked for, because a language-model question on an edge module is a memory question first and the registry, not the benchmark rows, is what carries sourced memory facts for it.

Above max_candidates (default 40, cap 90) the universe is truncated by benchmark-row count, before any scoring, and the truncation is reported. Truncating after scoring would let the cap silently decide the answer.

3. The canonical model map

The benchmark database records the same checkpoint under more than one id, because it was assembled from sources that spelled models differently. YOLOv8n-seg has its throughput rows under yolov8_seg|n and its 0.367 mAP under yolov8_seg|n-seg. SSD MobileNet v1 appears under three family spellings. EfficientNet-EdgeTPU appears under three more. Without a map that joins them, one candidate has speed and no accuracy while a phantom second has accuracy and no speed, and neither can be compared to anything.

The map lives in the model-facts registry and is published in the dataset. Every entry names the non-canonical spelling, the canonical model and the reason the two are the same checkpoint; no entry points at another entry, and no canonical model is itself the source of an alias. After the map is applied, the merged candidate carries every row from every member spelling, and the result reports how many of its candidates are joins.

The map is Model Match's alone. The benchmark database is not rewritten by it, so no other engine's answers move because of it.

4. What each candidate is evaluated on

Every constraint is stamped with the engine that asserted it and the check it answers, so a reader can always say which engine is responsible for any number on the page. Nothing is recomputed here, and there is no constraint type that only this engine produces.

5. Reference accuracy and quantised accuracy

These are two different measurements and this engine never merges them.

The reference accuracy is the figure the model's own publisher puts on its model card, with the metric and the evaluation set named. For the detector families that is mAP50-95 on COCO val2017. Naming the metric is not decoration: it is what resolves the contradiction this engine was built on. YOLO11n has 0.395 published on the Ultralytics model page and 0.5085 published in the same publisher's Jetson guide. Eleven points apart is far more than quantisation explains, so they cannot be the same metric on the same set — and the guide does not say which set it used. The model card figure is the reference; the guide figure is recorded as a per-platform observation whose dataset is unstated, flagged as conflicting, and never presented as the model's accuracy.

The quantised accuracy is what was observed at the precision asked for. It is resolved in a fixed order: the benchmark row for this exact platform and precision if one exists; otherwise a per-platform figure the benchmark database dropped in its de-duplication; otherwise the publisher's own quantisation table, which also carries the drop in points, the quantisation method and the calibration set. Where the published drop at the requested precision exceeds two mAP points, the result names it in a warning — the ranking uses the reference figure, so that loss is not priced into the score and the reader has to decide whether they can afford it.

Evidence class. A vendor model-zoo or model-card accuracy table is an external measured benchmark: class C. Class B is reserved for EdgeAIStack's own measurements and class A for a vendor specification, so no accuracy figure on this page claims either. That is checked, not merely intended: a reference accuracy claiming class A or B fails a rule.

6. The vision memory model

Vision memory is built here rather than taken from the legacy memory estimator, because that estimator assigns a documented total peak figure to its weight term and then multiplies it again — a six-fold over-count on the one row backed by a real measurement (logged as engine bug #18, owned by the package that rewrites its consumers). Model Match never reads that table.

Instead the budget is:

The third row is the honest limitation: it is a five-point curve applied to architectures it was not measured on, which is an engineering heuristic and is labelled as one. That is why the size term is never evidence for a vision candidate, and why the memory constraint on such a candidate reports class E rather than borrowing the class of the split it was interpolated from. A candidate with no parameter count at all reports memory UNKNOWN rather than a guess.

Verdict: FITS below 85 % of module memory, TIGHT to 100 %, DOES NOT FIT at or above it — the same thresholds Model Memory Fit uses.

7. The filter, and how UNKNOWN is treated

A candidate is excluded if and only if a compute check FAILs or is UNSUPPORTED, or its memory does not fit. That is the whole filter. There is no accuracy floor, no size ceiling, no throughput margin requirement and no preference anywhere in it: a preference changes the order of the ranked list and never its membership, which is a property a rule checks on every combination by re-running the same question at all four preferences.

Every excluded candidate is returned with the exact constraint that excluded it — the check, its status, what was required, what was available and the first assumption behind it — so an absence from the list is always explained.

UNKNOWN never excludes anything and never passes anything free. A batch size nobody measured, a precision with no row, a model with no parameter count: each comes back UNKNOWN, caps the candidate's verdict at NEEDS VALIDATION, and counts as class E in the evidence term. A candidate whose throughput and memory are both UNKNOWN is thin and sorts after every candidate with coverage, so it can only reach the top when nothing else has coverage either — which is what makes "the winner is the top of the list" safe.

8. The four terms and the weight table

Score is a plain weighted sum, Σ wᵢ·sᵢ, over four terms in [0,1]. No multipliers, no overlays, no per-preference code path. Each term is published beside the score, so the arithmetic can be recomputed from the numbers on the page — a ranking that cannot be checked from what it prints is not falsifiable.

The weight table below is class E — engineering judgement, not measurement. It is the only weight table in the engine: there is no overlay, no multiplier and no preference-specific branch anywhere else, and the applied row travels on every result so it can be read next to the scores it produced. Each row sums to 1.0.

The evidence weight never drops below 0.20 in any row. That is deliberate: a preference is a statement about what you want, not permission to prefer an unsourced number over a sourced one.

9. Winner, roles and ties

The winner is ranked[0], full stop. There is no override, no re-pick and no input by which a caller can choose differently; a caller who dislikes the winner changes the preference or the constraints, not the result. The result's verdict is the winner's verdict, and the winner is never NO FIT.

Roles are labels on the ranked list, not a second selection pass. most_accurate, fastest and smallest mark the true maximum or minimum of the feasible set and may land on the winner itself. stretch marks a candidate that is at least a tenth more accurate than the winner but whose verdict is FIT WITH RISKS or NEEDS VALIDATION — the model worth measuring before you settle. A candidate may carry several roles or none.

Ties break deterministically and in this order: coverage (thin candidates last), score, higher evidence, higher reference accuracy, lower total memory, then the stable candidate id. Two identical calls return byte-identical results.

10. What the registries actually contain

• Accuracy covers roughly a third of the candidate universe. The model-facts registry carries a reference accuracy for the detector families whose publishers print one; classification, pose, segmentation and the transformer tail mostly have none. Those models compete on margin, evidence and size and score 0 on accuracy.
• Licence is not on record for any model. A licence filter therefore has nothing to filter on, and the field is null everywhere rather than guessed.
• Speech-recognition word error rate and language-model quality scores do not exist here at all. A language candidate is ranked on evidence and size.
• Weight size is a parameter count times a byte width for every vision candidate — a model, not a published artefact size, and labelled class D on every row.
• The TensorRT memory split is published for five detector variants. Every other vision candidate's activation figure is that curve interpolated, class E.
• Per-platform accuracy figures often do not name their evaluation set. Those are recorded with dataset: unstated and flagged when they conflict with the model card by more than five points.

11. What invalidates a result

• Your own evaluation set. Every accuracy here is COCO or ImageNet. Relative ordering on a public benchmark is a weak predictor of relative ordering on your scenes, your classes and your lighting; if accuracy is the deciding term, evaluate the top two on your own data.
• Quantisation you have not done yourself. The published INT8 drops assume the publisher's calibration set and method. A different calibration set, or post-training quantisation where the publisher used quantisation-aware training, can move the figure by several points.
• The vision memory estimate. Outside the five variants with a published split it is a class E interpolation, so a TIGHT verdict there is a hypothesis, not a measurement.
• Everything Compute Fit's own caveats cover. Software versions, thermal state, input resolution, batch, the multi-stream efficiency curve and the fact that every benchmark row is inference-only.
• The weight table. It is judgement. A deployment with a hard accuracy floor or a hard memory ceiling should treat those as constraints on the question — narrow the precision, exclude the families that cannot meet them — rather than hoping a weighted sum expresses them.
• Coverage. A model that nobody has benchmarked on your silicon is absent, not rejected. Absence from this list is not evidence against a model.

12. Method changelog

Method changes bump the method version; the composite dataset version follows the newest of the registries this engine reads, and every result also carries the per-engine versions in sub_engines.