sousa-gecko/testing/perfdocs/generated/adding-tests.md

Adding Performance Tests

:depth: 3

This guide helps you choose the right framework for a new performance test and shows how to write one. For general information about performance testing at Mozilla (handling regressions, running tests, comparisons), see Performance Testing in a Nutshell.

Choosing a Framework

.. list-table::
   :header-rows: 1
   :widths: 18 32 8 12 30

   * - Framework
     - Use when
     - Android
     - Cross-browser
     - Run with
   * - `MozPerftest <mozperftest.html>`_ (xpcshell)
     - Testing low-level platform code from JS (networking, intl, storage).
       No browser chrome needed.
     - No
     - No
     - ``./mach perftest path/to/test.js``
   * - `MozPerftest <mozperftest.html>`_ (mochitest)
     - Testing browser-level operations that need a content page or chrome
       privileges (DOM, accessibility, service workers, ...).
     - No
     - No
     - ``./mach perftest path/to/test.html``
   * - `MozPerftest <mozperftest.html>`_ (browsertime)
     - Full page-load or navigation scenarios that need a real browser
       session driven via Selenium/WebDriver.
     - Yes
     - Yes (Chrome, Safari)
     - ``./mach perftest path/to/perftest_example.js``
   * - `MozPerftest <mozperftest.html>`_ (custom script)
     - Shell-script-based tests (e.g. startup timing on Android).
     - Yes
     - Yes, script-dependent
     - ``./mach perftest path/to/script.sh``
   * - `GTest <https://firefox-source-docs.mozilla.org/gtest/index.html>`_
     - Micro-benchmarking C++ or Rust code at the platform level
       (parsing, layout, networking internals). Very low overhead.
     - No
     - No
     - ``./mach gtest 'SuiteName.TestName'``
   * - `Raptor <raptor.html>`__
     - Industry-standard benchmarks (Speedometer, JetStream, MotionMark)
       or cross-browser page-load comparisons. See `Getting Help`_ first.
     - Yes
     - Yes (Chrome, Safari)
     - ``./mach raptor``
   * - `Talos <talos.html>`__
     - Legacy framework. **Do not add new tests here** unless there is a
       specific limitation that prevents use of MozPerftest.
     - No
     - No
     - ``./mach talos-test``
   * - `AWSY <https://firefox-source-docs.mozilla.org/testing/perfdocs/awsy.html#awsy>`__
     - Memory-usage tracking (Are We Slim Yet) across builds.
     - No
     - No
     - ``./mach awsy-test``

For most new tests, MozPerftest is the recommended choice. Pick the flavor (xpcshell, mochitest, browsertime) that matches the level at which your code operates. For the full reference of perfMetadata fields, options, and supported flavors, see the MozPerftest writing guide.

MozPerftest: XPCShell Example

XPCShell tests are the simplest flavor: a plain xpcshell test with a perfMetadata variable and info("perfMetrics", ...) calls to report results.

In-tree examples:

• intl/benchmarks/test/xpcshell/
• netwerk/test/unit/test_trr_bench.js (bug 2009372)

Test file (intl/benchmarks/test/xpcshell/perftest_dateTimeFormat.js):

"use strict";

var perfMetadata = {
  owner: "Intl team",
  name: "Intl.DateTimeFormat",
  description: "Test the speed of Intl.DateTimeFormat",
  options: {
    default: {
      perfherder: true,
      perfherder_metrics: [
        { name: "DateTimeFormat constructor", unit: "ms" },
        { name: "DateTimeFormat.format", unit: "ms" },
      ],
    },
  },
  tags: ["intl"],
};

add_task(function test_dateTimeFormat() {
  let start = Cu.now();
  for (let i = 0; i < 100000; i++) {
    new Intl.DateTimeFormat("en-US");
  }
  let constructorTime = Cu.now() - start;

  let fmt = new Intl.DateTimeFormat("en-US");
  let date = new Date();
  start = Cu.now();
  for (let i = 0; i < 100000; i++) {
    fmt.format(date);
  }
  let formatTime = Cu.now() - start;

  info(
    "perfMetrics",
    JSON.stringify({
      "DateTimeFormat constructor": constructorTime,
      "DateTimeFormat.format": formatTime,
    })
  );
});

Manifest (perftest.toml):

[DEFAULT]

["perftest_dateTimeFormat.js"]
disabled = "Disabled as we want to run this only as perftest, not regular CI"

The disabled field keeps the test out of the normal unit-test chunks; it will still run when invoked via ./mach perftest.

Registration (moz.build):

PERFTESTS_MANIFESTS += ["test/xpcshell/perftest.toml"]

Run locally:

./mach perftest intl/benchmarks/test/xpcshell/perftest_dateTimeFormat.js

MozPerftest: Mochitest Example

Mochitest-flavored perftests are standard mochitests with perfMetadata and info("perfMetrics", ...) calls. They use the regular mochitest manifest variable (e.g. BROWSER_CHROME_MANIFESTS) instead of PERFTESTS_MANIFESTS, and set disabled in the manifest to prevent the test from running in regular CI.

In-tree examples:

• mochitest-plain: dom/serviceworkers/test/performance/ (bug 1299271)
• browser-chrome: accessible/tests/browser/performance/ (bug 1963174)
• browser-chrome: toolkit/components/ml/tests/browser/

Test file (dom/serviceworkers/test/performance/test_caching.html):

<!DOCTYPE HTML>
<html>
<head>
  <title>Service worker caching perftest</title>
</head>
<script src="/tests/SimpleTest/SimpleTest.js"></script>
<script src="perfutils.js"></script>
<script>
  "use strict";

  var journal = {
    "No cache": [],
    "Cached": [],
  };

  const ITERATIONS = 10;

  var perfMetadata = {
    owner: "DOM LWS",
    name: "Service Worker Caching",
    description: "Test service worker caching.",
    options: {
      default: {
        perfherder: true,
        perfherder_metrics: [
          { name: "No cache", unit: "ms", shouldAlert: true },
          { name: "Cached", unit: "ms", shouldAlert: true },
        ],
        verbose: true,
        manifest: "perftest.toml",
        manifest_flavor: "plain",
      },
    },
  };

  add_task(async () => {
    // ... run iterations, collect timing into journal ...
  });

  add_task(() => {
    // Compute medians and report
    let metrics = {};
    for (const name in journal) {
      let sorted = [...journal[name]].sort((a, b) => a - b);
      metrics[name] = sorted[Math.floor(sorted.length / 2)];
    }
    info("perfMetrics", JSON.stringify(metrics));
  });
</script>
<body></body>
</html>

Manifest (perftest.toml):

[DEFAULT]

["test_caching.html"]
disabled = "Disabled as we want to run this only as perftest, not regular CI"

Registration (moz.build) -- use the standard mochitest manifest variable for the flavor you're using, not PERFTESTS_MANIFESTS:

MOCHITEST_MANIFESTS += ["test/performance/perftest.toml"]
# or, for browser-chrome:
# BROWSER_CHROME_MANIFESTS += ["test/browser/performance/perftest.toml"]

Key differences from xpcshell perftests:

• perfMetadata must include manifest and manifest_flavor fields ("plain" for mochitest-plain, "browser-chrome" for browser chrome tests).
• extra_args can specify additional mochitest arguments (e.g. ["headless"]).
• The manifest is registered with the standard mochitest variable, not PERFTESTS_MANIFESTS. The disabled field on each test entry is what keeps it out of the normal mochitest CI chunks; it will still run when invoked via ./mach perftest.

MozPerftest: Browsertime Example

Browsertime tests drive a full browser session via Selenium. Use them when you need real navigation, user interaction, or page-load measurements.

In-tree examples (under testing/performance/):

• perftest_pageload.js -- minimal navigation example.
• perftest_facebook.js -- form interaction with addText/click.

Test file (testing/performance/perftest_pageload.js):

async function setUp(context) {
  context.log.info("setUp example!");
}

async function test(context, commands) {
  let url = context.options.browsertime.url;
  await commands.navigate("https://www.mozilla.org/en-US/");
  await commands.wait.byTime(100);
  await commands.navigate("about:blank");
  await commands.wait.byTime(50);
  return commands.measure.start(url);
}

async function tearDown(context) {
  context.log.info("tearDown example!");
}

module.exports = {
  setUp,
  tearDown,
  test,
  owner: "Performance Team",
  name: "pageload",
  description: "Measures time to load mozilla page",
};

Manifest (testing/performance/perftest.toml) just lists the test files:

[DEFAULT]

["perftest_pageload.js"]

By convention, browsertime perftest files are prefixed with perftest_.

For full documentation on the Browsertime scripting API, see the sitespeed.io scripting docs.

MozPerftest: Custom Script Example

Custom-script tests are shell scripts (or programs invoked from one) that write a single perfMetrics: ... line to stdout. They are useful for platform-level measurements that don't fit into xpcshell or browsertime -- for example, Android startup timing where the test needs to drive adb.

In-tree examples:

• testing/performance/mobile-startup/ -- Android cold-startup tests (cvns.sh, cmff.sh, ...).
• testing/performance/android-resource/ -- Android resource-usage tests.

Test file (illustrative, based on perftest_custom.sh):

# Name: custom-script-test
# Owner: Perftest Team
# Description: Runs a sample custom script test.
# Options: {"default": {"perfherder": true, "perfherder_metrics": [{ "name": "Registration", "unit": "ms" }]}}

echo Running...

# ${BROWSER_BINARY} is the package name on Android, or the path to the
# browser binary on desktop. MozPerftest verifies that it exists.
echo Binary: ${BROWSER_BINARY}

# The single perfMetrics line is what MozPerftest parses for results.
# Curly braces in the JSON must be doubled.
echo 'perfMetrics: [{{"name": "metric1", "shouldAlert": false, "lowerIsBetter": false, "unit": "speed", "values": [1, 2, 3, 4]}}]'

Run locally:

./mach perftest path/to/your-script.sh

GTest: C++ Micro-benchmarks

GTest is the simplest way to add a low-level C++ or Rust performance test. The MOZ_GTEST_BENCH macro wraps a test function so that GTest times its execution and reports the result to Perfherder under the platform_microbench framework.

In debug/ASAN builds the test runs once without timing (as a correctness check). In optimized builds it runs 5 times by default (configurable via MOZ_GTEST_NUM_ITERATIONS) and reports the median duration.

In-tree examples:

• layout/style/test/gtest/StyloParsingBench.cpp
• netwerk/test/gtest/TestStandardURL.cpp (currently disabled by default)

Example (from StyloParsingBench.cpp):

#include "gtest/MozGTestBench.h"

static void ServoParsingBench() {
  // ... setup code ...
  for (int i = 0; i < PARSING_REPETITIONS; i++) {
    RefPtr<StyleStylesheetContents> stylesheet =
        Servo_StyleSheet_FromUTF8Bytes(/* ... */).Consume();
  }
}

MOZ_GTEST_BENCH(Stylo, Servo_StyleSheet_FromUTF8Bytes_Bench,
                [] { ServoParsingBench(); });

There is also MOZ_GTEST_BENCH_F for fixture-based tests (wraps TEST_F instead of TEST).

When the test runs in CI, it outputs:

PERFHERDER_DATA: {"framework": {"name": "platform_microbench"},
  "suites": [{"name": "Stylo", "subtests":
    [{"name": "Servo_StyleSheet_FromUTF8Bytes_Bench",
      "value": 252674, "lowerIsBetter": true}]}]}

:::{note} Micro-benchmark regressions are not treated as strictly as other performance test regressions. A regression in a micro-benchmark may not correspond to a user-visible regression, and large changes are expected when the benchmarked code is intentionally modified. :::

See the GTest documentation for more details.

Raptor

Raptor is the framework used for industry-standard benchmarks (Speedometer, JetStream, MotionMark) and cross-browser page-load comparisons.

In-tree examples (under testing/raptor/browsertime/):

• browsertime_pageload.js -- canonical page-load test runner.
• speculative-connect.js -- privileged-call + custom metric.

Test file (excerpt of speculative-connect.js, bug 1818798):

const { logTest } = require("./utils/profiling");

module.exports = logTest(
  "speculative connect pageload",
  async function (context, commands) {
    const url = "https://en.wikipedia.org/wiki/Barack_Obama";

    await commands.navigate("about:blank");
    await commands.wait.byTime(1000);

    // Privileged JS to trigger a speculative connection.
    const script = `
      var URI = Services.io.newURI("${url}");
      var principal = Services.scriptSecurityManager
        .createContentPrincipal(URI, {});
      Services.io.speculativeConnect(URI, principal, callbacks, false);
    `;
    commands.js.runPrivileged(script);
    await commands.wait.byTime(1000);

    // Measure the pageload.
    await commands.measure.start();
    await commands.navigate(url);
    await commands.measure.stop();

    // Report a custom metric alongside the pageload measurements.
    const connect_time = await commands.js.run(
      `return (window.performance.timing.connectEnd -
               window.performance.timing.navigationStart);`
    );
    await commands.measure.addObject({
      custom_data: { connect_time },
    });

    return true;
  }
);

The wiring (manifest entry, taskcluster kind, alert thresholds, etc.) is larger than for MozPerftest tests -- see the contributing guide for adding new Raptor tests, then Raptor documentation for the full reference.

See [Getting Help] before adding a new Raptor test.

Talos

Talos is the legacy framework. Do not add new tests here unless there is a specific limitation that prevents using MozPerftest -- see [Getting Help] first.

A Talos test is a Python class that subclasses TsBase (or another base) and is registered via @register_test() in testing/talos/talos/test.py. The class fields configure the runner (URL, cycles, filters, units, etc.).

Example (excerpt of ts_paint from testing/talos/talos/test.py):

@register_test()
class ts_paint(TsBase):
    """
    Launches tspaint_test.html with the current timestamp in the url,
    waits for [MozAfterPaint and onLoad] to fire, then records the end
    time and calculates the time to startup.
    """

    cycles = 20
    timeout = 150
    gecko_profile_startup = True
    url = "startup_test/tspaint_test.html"
    filters = filter.ignore_first.prepare(1) + filter.median.prepare()
    tpmozafterpaint = True
    unit = "ms"

The page (startup_test/tspaint_test.html) lives under testing/talos/talos/; manifests and config wiring are also in that tree. See the Talos documentation for the full reference.

AWSY

AWSY (Are We Slim Yet) tracks memory usage across builds.

An AWSY test is a Python class that subclasses AwsyTestCase. It declares the perfherder suites, the memory checkpoints to capture from about:memory, and the URLs to load. AWSY then drives the browser through those steps and reports the configured metrics.

Example (excerpt of testing/awsy/awsy/test_base_memory_usage.py):

from awsy.awsy_test_case import AwsyTestCase

CHECKPOINTS = [
    {
        "name": "After tabs open [+30s, forced GC]",
        "path": "memory-report-TabsOpenForceGC-4.json.gz",
        "name_filter": ["web ", "Web Content"],
        "median": True,
    },
]

PERF_SUITES = [
    {"name": "Base Content Resident Unique Memory", "node": "resident-unique"},
    {"name": "Base Content Heap Unclassified", "node": "explicit/heap-unclassified"},
    {"name": "Base Content JS", "node": "js-main-runtime/", "alertThreshold": 0.25},
    {"name": "Base Content Explicit", "node": "explicit/"},
]


class TestMemoryUsage(AwsyTestCase):
    """Loads about:memory and reports content-process memory usage."""

    def urls(self):
        return self._urls

    def perf_suites(self):
        return PERF_SUITES

    def perf_checkpoints(self):
        return CHECKPOINTS

Run locally:

./mach awsy-test --base

See the AWSY documentation for the full set of fields and configuration options.

Running Tests in CI

To run your test in CI you need to add a task definition under the appropriate taskcluster kinds/ directory:

• MozPerftest: taskcluster/kinds/perftest/
• Raptor and browsertime: taskcluster/kinds/browsertime/

This is what determines which platforms and configurations your test runs on. If you're not sure where your task belongs, see the [Getting Help] section below -- the perf team can help.

For xpcshell and mochitest MozPerftest tests, you may also need to update the _TRY_MAPPING variable in mozperftest/utils.py so CI can locate your test file.

Once your test is registered, ./mach try perf will include it in try pushes -- but only if the test matches an existing category. If you can't find your test in the category picker, run with --full to fall back to the full ./mach try fuzzy interface, or add a new category for your test. To reproduce a specific alert:

./mach try perf --alert <ALERT-NUMBER>

Note that --alert searches across all tasks regardless of category. For more details, see Mach Try Perf.

Getting Help

Reach out to the Performance Testing and Tooling team in the #perftest channel on Matrix or #perf-help on Slack.