Joe Bowser
2956c527b6
Cover the real Link Preview production model (SmolLM2-360M Q8)
with the same structural assertions the mochitest sibling in
browser_ml_native.js applies to TinyStories. Runs under the
perftest harness because hooks_local_hub.py can serve the GGUF
from MOZ_FETCHES_DIR/onnx-models/; the mochitest sandbox blocks
external hosts and the GGUF is too large to vendor.
Unlike TinyStories, SmolLM2's greedy output is byte-identical on
every architecture in our CI fleet (Linux/Windows x86_64, macOS
aarch64). Its top-1 logit margins are wide enough to absorb
FP-rounding differences from any of the SIMD widths or math
libraries in play (Apple Accelerate included), so the pinned
EXPECTED_TEXT + EXPECTED_HASH don't dispatch on
AppConstants.platform.
Each task emits a heartbeat info() line around engine init so
the harness's "no output for 370s" outer timeout doesn't fire
during the slow GGUF mmap.
Wiring:
- ml-llama-smollm2-smoke task in
taskcluster/kinds/perftest/{linux,windows11-24h2,windows11-24h2-ref}.yml,
each fetching only smollm2-360-instruct-gguf.
- perftest.toml entry with disabled = ... so the test runs only
via mach perftest.
- python/mozperftest/perfdocs/config.yml entry + regenerated
testing/perfdocs/generated/mozperftest.rst for the perfdocs lint.
macOS Intel 10.15 is intentionally not covered: native llama.cpp
was unstable there for TinyStories (see the runtime skip in
test_ml_smoke_test_llama_golden_text), macOS 10.15 is EOL, and
the releng-hardware pool has multi-hour queue depth. Revisit
once a macOS aarch64 perftest worker pool exists.
Not covered here:
- Android perftest task — Link Preview is desktop-only today.
- Semantic similarity / KL divergence — LlamaRunner doesn't
expose logits from JS; that warrants its own bug.
Mach Try (All Green): https://treeherder.mozilla.org/jobs?repo=try&revision=4e54266f9c7d9a4db5b3a058bab3dc763d421255
Differential Revision: https://phabricator.services.mozilla.com/D308152
110 lines
3.9 KiB
YAML
110 lines
3.9 KiB
YAML
# This Source Code Form is subject to the terms of the Mozilla Public
|
|
# License, v. 2.0. If a copy of the MPL was not distributed with this
|
|
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
|
---
|
|
name: mozperftest
|
|
manifest: None
|
|
static-only: False
|
|
suites:
|
|
netwerk/test/perf:
|
|
description: "Performance tests from the 'network/test/perf' folder."
|
|
tests:
|
|
youtube-scroll: ""
|
|
facebook-scroll: ""
|
|
cloudflare: ""
|
|
controlled: ""
|
|
g-search: ""
|
|
g-image: ""
|
|
lq-fetch: ""
|
|
youtube-noscroll: ""
|
|
netwerk/test/unit:
|
|
description: "Performance tests from the 'netwerk/test/unit' folder."
|
|
tests:
|
|
DNS Benchmark: ""
|
|
http3 raw: ""
|
|
TRR Benchmark: ""
|
|
testing/performance:
|
|
description: "Performance tests from the 'testing/performance' folder."
|
|
tests:
|
|
Politico Link: ""
|
|
BBC Link: ""
|
|
JSConf (cold): ""
|
|
Facebook: ""
|
|
YouTube Link: ""
|
|
pageload: ""
|
|
JSConf (warm): ""
|
|
perfstats: ""
|
|
webpagetest-firefox: ""
|
|
webpagetest-chrome: ""
|
|
|
|
browser/base/content/test:
|
|
description: "Performance tests from the 'browser/base/content/test' folder."
|
|
tests:
|
|
Dom-size: ""
|
|
|
|
browser/components/translations/tests/browser:
|
|
description: "Performance tests for Translations models on Firefox Desktop"
|
|
tests:
|
|
"Full-Page Translations Base Model": ""
|
|
"Full-Page Translations BaseMemory Model": ""
|
|
"Full-Page Translations Tiny Model": ""
|
|
|
|
dom/serviceworkers/test/performance:
|
|
description: "Performance tests running through Mochitest for Service Workers"
|
|
tests:
|
|
"Service Worker Caching": ""
|
|
"Service Worker Fetch": ""
|
|
"Service Worker Registration": ""
|
|
"Service Worker Update": ""
|
|
|
|
toolkit/components/ml/tests/browser:
|
|
description: "Performance tests running through Mochitest for ML Models"
|
|
tests:
|
|
"browser_ml_engine_perf.js": ""
|
|
"browser_ml_engine_multi_perf.js": ""
|
|
"ML Semantic History Search": ""
|
|
"browser_ml_suggest_feature_perf.js": ""
|
|
"browser_ml_suggest_inference.js": ""
|
|
"browser_ml_autofill_perf.js": ""
|
|
"browser_ml_summarizer_perf.js": ""
|
|
"ML Smart Tab Clustering": ""
|
|
"browser_ml_smart_tab_perf.js": ""
|
|
"browser_ml_llama_smollm2_smoke.js": ""
|
|
"browser_ml_llama_summarizer_perf.js": ""
|
|
"ML Security Orchestrator Performance Tests": ""
|
|
"ML Speech T5 TTS": ""
|
|
|
|
intl/benchmarks/test/xpcshell:
|
|
description: "Performance tests running through XPCShell for Intl code"
|
|
tests:
|
|
"IntlDateTimeFormat": ""
|
|
"IntlLocale": ""
|
|
"IntlNumberFormat": ""
|
|
"IntlPluralRules": ""
|
|
|
|
dom/webgpu/tests/mochitest:
|
|
description: "Performance tests from the 'dom/webgpu/tests/mochitest' folder."
|
|
tests:
|
|
"Queue Write": ""
|
|
|
|
toolkit/components/url-classifier/tests/performance:
|
|
description: "Performance tests for the URL Classifier"
|
|
tests:
|
|
"UrlClassifierExceptionListLookup": ""
|
|
|
|
accessible/tests/browser/performance:
|
|
description: "Performance tests for the accessibility engine"
|
|
tests:
|
|
"browser_addManyIds.js": ""
|
|
"browser_addManyNodes.js": ""
|
|
"browser_bounds.js": ""
|
|
"browser_displayContentsBounds.js": ""
|
|
"browser_reflowPseudoelements.js": ""
|
|
"browser_removeManySpellingErrors.js": ""
|
|
"browser_implicitRoleARIAExpanded.js": ""
|
|
|
|
toolkit/components/places/tests/browser/performance:
|
|
description: "Performance tests for Toolkit: Places"
|
|
tests:
|
|
"Calculate Frecency Speed": ""
|