Joe Bowser
2956c527b6
Cover the real Link Preview production model (SmolLM2-360M Q8)
with the same structural assertions the mochitest sibling in
browser_ml_native.js applies to TinyStories. Runs under the
perftest harness because hooks_local_hub.py can serve the GGUF
from MOZ_FETCHES_DIR/onnx-models/; the mochitest sandbox blocks
external hosts and the GGUF is too large to vendor.
Unlike TinyStories, SmolLM2's greedy output is byte-identical on
every architecture in our CI fleet (Linux/Windows x86_64, macOS
aarch64). Its top-1 logit margins are wide enough to absorb
FP-rounding differences from any of the SIMD widths or math
libraries in play (Apple Accelerate included), so the pinned
EXPECTED_TEXT + EXPECTED_HASH don't dispatch on
AppConstants.platform.
Each task emits a heartbeat info() line around engine init so
the harness's "no output for 370s" outer timeout doesn't fire
during the slow GGUF mmap.
Wiring:
- ml-llama-smollm2-smoke task in
taskcluster/kinds/perftest/{linux,windows11-24h2,windows11-24h2-ref}.yml,
each fetching only smollm2-360-instruct-gguf.
- perftest.toml entry with disabled = ... so the test runs only
via mach perftest.
- python/mozperftest/perfdocs/config.yml entry + regenerated
testing/perfdocs/generated/mozperftest.rst for the perfdocs lint.
macOS Intel 10.15 is intentionally not covered: native llama.cpp
was unstable there for TinyStories (see the runtime skip in
test_ml_smoke_test_llama_golden_text), macOS 10.15 is EOL, and
the releng-hardware pool has multi-hour queue depth. Revisit
once a macOS aarch64 perftest worker pool exists.
Not covered here:
- Android perftest task — Link Preview is desktop-only today.
- Semantic similarity / KL divergence — LlamaRunner doesn't
expose logits from JS; that warrants its own bug.
Mach Try (All Green): https://treeherder.mozilla.org/jobs?repo=try&revision=4e54266f9c7d9a4db5b3a058bab3dc763d421255
Differential Revision: https://phabricator.services.mozilla.com/D308152
mozperftest
mozperftest can be used to run performance tests against browsers. See the docs directory.