sousa-gecko/testing/perfdocs/generated/adding-tests.md

# Adding Performance Tests

```{contents}
:depth: 3
```

This guide helps you choose the right framework for a new performance test
and shows how to write one. For general information about performance testing
at Mozilla (handling regressions, running tests, comparisons), see
[Performance Testing in a Nutshell](perftest-in-a-nutshell.md).

## Choosing a Framework

```{eval-rst}
.. list-table::
   :header-rows: 1
   :widths: 18 32 8 12 30

   * - Framework
     - Use when
     - Android
     - Cross-browser
     - Run with
   * - `MozPerftest <mozperftest.html>`_ (xpcshell)
     - Testing low-level platform code from JS (networking, intl, storage).
       No browser chrome needed.
     - No
     - No
     - ``./mach perftest path/to/test.js``
   * - `MozPerftest <mozperftest.html>`_ (mochitest)
     - Testing browser-level operations that need a content page or chrome
       privileges (DOM, accessibility, service workers, ...).
     - No
     - No
     - ``./mach perftest path/to/test.html``
   * - `MozPerftest <mozperftest.html>`_ (browsertime)
     - Full page-load or navigation scenarios that need a real browser
       session driven via Selenium/WebDriver.
     - Yes
     - Yes (Chrome, Safari)
     - ``./mach perftest path/to/perftest_example.js``
   * - `MozPerftest <mozperftest.html>`_ (custom script)
     - Shell-script-based tests (e.g. startup timing on Android).
     - Yes
     - Yes, script-dependent
     - ``./mach perftest path/to/script.sh``
   * - `GTest <https://firefox-source-docs.mozilla.org/gtest/index.html>`_
     - Micro-benchmarking C++ or Rust code at the platform level
       (parsing, layout, networking internals). Very low overhead.
     - No
     - No
     - ``./mach gtest 'SuiteName.TestName'``
   * - `Raptor <raptor.html>`__
     - Industry-standard benchmarks (Speedometer, JetStream, MotionMark)
       or cross-browser page-load comparisons. See `Getting Help`_ first.
     - Yes
     - Yes (Chrome, Safari)
     - ``./mach raptor``
   * - `Talos <talos.html>`__
     - Legacy framework. **Do not add new tests here** unless there is a
       specific limitation that prevents use of MozPerftest.
     - No
     - No
     - ``./mach talos-test``
   * - `AWSY <https://firefox-source-docs.mozilla.org/testing/perfdocs/awsy.html#awsy>`__
     - Memory-usage tracking (Are We Slim Yet) across builds.
     - No
     - No
     - ``./mach awsy-test``
```

For most new tests, **MozPerftest** is the recommended choice.
Pick the flavor (xpcshell, mochitest, browsertime) that matches
the level at which your code operates. For the full reference of
`perfMetadata` fields, `options`, and supported flavors, see the
MozPerftest [writing guide](writing.md).

## MozPerftest: XPCShell Example

XPCShell tests are the simplest flavor: a plain xpcshell test with a
`perfMetadata` variable and `info("perfMetrics", ...)` calls to report
results.

In-tree examples:

- [intl/benchmarks/test/xpcshell/](https://searchfox.org/mozilla-central/source/intl/benchmarks/test/xpcshell)
- [netwerk/test/unit/test_trr_bench.js](https://searchfox.org/mozilla-central/source/netwerk/test/unit/test_trr_bench.js)
  ([bug 2009372](https://bugzilla.mozilla.org/show_bug.cgi?id=2009372))

**Test file** ([intl/benchmarks/test/xpcshell/perftest_dateTimeFormat.js](https://searchfox.org/mozilla-central/source/intl/benchmarks/test/xpcshell/perftest_dateTimeFormat.js)):

```javascript
"use strict";

var perfMetadata = {
  owner: "Intl team",
  name: "Intl.DateTimeFormat",
  description: "Test the speed of Intl.DateTimeFormat",
  options: {
    default: {
      perfherder: true,
      perfherder_metrics: [
        { name: "DateTimeFormat constructor", unit: "ms" },
        { name: "DateTimeFormat.format", unit: "ms" },
      ],
    },
  },
  tags: ["intl"],
};

add_task(function test_dateTimeFormat() {
  let start = Cu.now();
  for (let i = 0; i < 100000; i++) {
    new Intl.DateTimeFormat("en-US");
  }
  let constructorTime = Cu.now() - start;

  let fmt = new Intl.DateTimeFormat("en-US");
  let date = new Date();
  start = Cu.now();
  for (let i = 0; i < 100000; i++) {
    fmt.format(date);
  }
  let formatTime = Cu.now() - start;

  info(
    "perfMetrics",
    JSON.stringify({
      "DateTimeFormat constructor": constructorTime,
      "DateTimeFormat.format": formatTime,
    })
  );
});
```

**Manifest** (`perftest.toml`):

```toml
[DEFAULT]

["perftest_dateTimeFormat.js"]
disabled = "Disabled as we want to run this only as perftest, not regular CI"
```

The `disabled` field keeps the test out of the normal unit-test chunks; it
will still run when invoked via `./mach perftest`.

**Registration** (`moz.build`):

```python
PERFTESTS_MANIFESTS += ["test/xpcshell/perftest.toml"]
```

Run locally:

```
./mach perftest intl/benchmarks/test/xpcshell/perftest_dateTimeFormat.js
```

## MozPerftest: Mochitest Example

Mochitest-flavored perftests are standard mochitests with `perfMetadata` and
`info("perfMetrics", ...)` calls. They use the regular mochitest manifest
variable (e.g. `BROWSER_CHROME_MANIFESTS`) instead of `PERFTESTS_MANIFESTS`,
and set `disabled` in the manifest to prevent the test from running in regular CI.

In-tree examples:

- **mochitest-plain**: [dom/serviceworkers/test/performance/](https://searchfox.org/mozilla-central/source/dom/serviceworkers/test/performance)
  ([bug 1299271](https://bugzilla.mozilla.org/show_bug.cgi?id=1299271))
- **browser-chrome**: [accessible/tests/browser/performance/](https://searchfox.org/mozilla-central/source/accessible/tests/browser/performance)
  ([bug 1963174](https://bugzilla.mozilla.org/show_bug.cgi?id=1963174))
- **browser-chrome**: [toolkit/components/ml/tests/browser/](https://searchfox.org/mozilla-central/source/toolkit/components/ml/tests/browser)

**Test file** ([dom/serviceworkers/test/performance/test_caching.html](https://searchfox.org/mozilla-central/source/dom/serviceworkers/test/performance/test_caching.html)):

```html
<!DOCTYPE HTML>
<html>
<head>
  <title>Service worker caching perftest</title>
</head>
<script src="/tests/SimpleTest/SimpleTest.js"></script>
<script src="perfutils.js"></script>
<script>
  "use strict";

  var journal = {
    "No cache": [],
    "Cached": [],
  };

  const ITERATIONS = 10;

  var perfMetadata = {
    owner: "DOM LWS",
    name: "Service Worker Caching",
    description: "Test service worker caching.",
    options: {
      default: {
        perfherder: true,
        perfherder_metrics: [
          { name: "No cache", unit: "ms", shouldAlert: true },
          { name: "Cached", unit: "ms", shouldAlert: true },
        ],
        verbose: true,
        manifest: "perftest.toml",
        manifest_flavor: "plain",
      },
    },
  };

  add_task(async () => {
    // ... run iterations, collect timing into journal ...
  });

  add_task(() => {
    // Compute medians and report
    let metrics = {};
    for (const name in journal) {
      let sorted = [...journal[name]].sort((a, b) => a - b);
      metrics[name] = sorted[Math.floor(sorted.length / 2)];
    }
    info("perfMetrics", JSON.stringify(metrics));
  });
</script>
<body></body>
</html>
```

**Manifest** (`perftest.toml`):

```toml
[DEFAULT]

["test_caching.html"]
disabled = "Disabled as we want to run this only as perftest, not regular CI"
```

**Registration** (`moz.build`) -- use the standard mochitest manifest variable
for the flavor you're using, not `PERFTESTS_MANIFESTS`:

```python
MOCHITEST_MANIFESTS += ["test/performance/perftest.toml"]
# or, for browser-chrome:
# BROWSER_CHROME_MANIFESTS += ["test/browser/performance/perftest.toml"]
```

Key differences from xpcshell perftests:

- `perfMetadata` must include `manifest` and `manifest_flavor` fields
  (`"plain"` for mochitest-plain, `"browser-chrome"` for browser chrome tests).
- `extra_args` can specify additional mochitest arguments (e.g. `["headless"]`).
- The manifest is registered with the standard mochitest variable, not
  `PERFTESTS_MANIFESTS`. The `disabled` field on each test entry is what
  keeps it out of the normal mochitest CI chunks; it will still run when
  invoked via `./mach perftest`.

## MozPerftest: Browsertime Example

Browsertime tests drive a full browser session via Selenium. Use them when
you need real navigation, user interaction, or page-load measurements.

In-tree examples (under [testing/performance/](https://searchfox.org/mozilla-central/source/testing/performance)):

- `perftest_pageload.js` -- minimal navigation example.
- `perftest_facebook.js` -- form interaction with `addText`/`click`.

**Test file** ([testing/performance/perftest_pageload.js](https://searchfox.org/mozilla-central/source/testing/performance/perftest_pageload.js)):

```javascript
async function setUp(context) {
  context.log.info("setUp example!");
}

async function test(context, commands) {
  let url = context.options.browsertime.url;
  await commands.navigate("https://www.mozilla.org/en-US/");
  await commands.wait.byTime(100);
  await commands.navigate("about:blank");
  await commands.wait.byTime(50);
  return commands.measure.start(url);
}

async function tearDown(context) {
  context.log.info("tearDown example!");
}

module.exports = {
  setUp,
  tearDown,
  test,
  owner: "Performance Team",
  name: "pageload",
  description: "Measures time to load mozilla page",
};
```

**Manifest** ([testing/performance/perftest.toml](https://searchfox.org/mozilla-central/source/testing/performance/perftest.toml))
just lists the test files:

```toml
[DEFAULT]

["perftest_pageload.js"]
```

By convention, browsertime perftest files are prefixed with `perftest_`.

For full documentation on the Browsertime scripting API, see the
[sitespeed.io scripting docs](https://www.sitespeed.io/documentation/sitespeed.io/scripting/).

## MozPerftest: Custom Script Example

Custom-script tests are shell scripts (or programs invoked from one) that
write a single `perfMetrics: ...` line to stdout. They are useful for
platform-level measurements that don't fit into xpcshell or browsertime --
for example, Android startup timing where the test needs to drive `adb`.

In-tree examples:

- [testing/performance/mobile-startup/](https://searchfox.org/mozilla-central/source/testing/performance/mobile-startup)
  -- Android cold-startup tests (`cvns.sh`, `cmff.sh`, ...).
- [testing/performance/android-resource/](https://searchfox.org/mozilla-central/source/testing/performance/android-resource)
  -- Android resource-usage tests.

**Test file** (illustrative, based on
[perftest_custom.sh](https://searchfox.org/mozilla-central/source/python/mozperftest/mozperftest/tests/data/samples/perftest_custom.sh)):

```bash
# Name: custom-script-test
# Owner: Perftest Team
# Description: Runs a sample custom script test.
# Options: {"default": {"perfherder": true, "perfherder_metrics": [{ "name": "Registration", "unit": "ms" }]}}

echo Running...

# ${BROWSER_BINARY} is the package name on Android, or the path to the
# browser binary on desktop. MozPerftest verifies that it exists.
echo Binary: ${BROWSER_BINARY}

# The single perfMetrics line is what MozPerftest parses for results.
# Curly braces in the JSON must be doubled.
echo 'perfMetrics: [{{"name": "metric1", "shouldAlert": false, "lowerIsBetter": false, "unit": "speed", "values": [1, 2, 3, 4]}}]'
```

Run locally:

```
./mach perftest path/to/your-script.sh
```

## GTest: C++ Micro-benchmarks

GTest is the simplest way to add a low-level C++ or Rust performance test.
The `MOZ_GTEST_BENCH` macro wraps a test function so that GTest times its
execution and reports the result to Perfherder under the
`platform_microbench` framework.

In debug/ASAN builds the test runs once without timing (as a correctness check).
In optimized builds it runs 5 times by default (configurable via
`MOZ_GTEST_NUM_ITERATIONS`) and reports the median duration.

In-tree examples:

- [layout/style/test/gtest/StyloParsingBench.cpp](https://searchfox.org/mozilla-central/source/layout/style/test/gtest/StyloParsingBench.cpp)
- [netwerk/test/gtest/TestStandardURL.cpp](https://searchfox.org/mozilla-central/source/netwerk/test/gtest/TestStandardURL.cpp)
  (currently disabled by default)

**Example** (from [StyloParsingBench.cpp](https://searchfox.org/mozilla-central/source/layout/style/test/gtest/StyloParsingBench.cpp)):

```cpp
#include "gtest/MozGTestBench.h"

static void ServoParsingBench() {
  // ... setup code ...
  for (int i = 0; i < PARSING_REPETITIONS; i++) {
    RefPtr<StyleStylesheetContents> stylesheet =
        Servo_StyleSheet_FromUTF8Bytes(/* ... */).Consume();
  }
}

MOZ_GTEST_BENCH(Stylo, Servo_StyleSheet_FromUTF8Bytes_Bench,
                [] { ServoParsingBench(); });
```

There is also `MOZ_GTEST_BENCH_F` for fixture-based tests (wraps `TEST_F`
instead of `TEST`).

When the test runs in CI, it outputs:

```
PERFHERDER_DATA: {"framework": {"name": "platform_microbench"},
  "suites": [{"name": "Stylo", "subtests":
    [{"name": "Servo_StyleSheet_FromUTF8Bytes_Bench",
      "value": 252674, "lowerIsBetter": true}]}]}
```

:::{note}
Micro-benchmark regressions are not treated as strictly as other
performance test regressions. A regression in a micro-benchmark may not
correspond to a user-visible regression, and large changes are expected
when the benchmarked code is intentionally modified.
:::

See the [GTest documentation](https://firefox-source-docs.mozilla.org/gtest/index.html#mozgtestbench)
for more details.

## Raptor

Raptor is the framework used for industry-standard benchmarks (Speedometer,
JetStream, MotionMark) and cross-browser page-load comparisons.

In-tree examples (under
[testing/raptor/browsertime/](https://searchfox.org/mozilla-central/source/testing/raptor/browsertime)):

- `browsertime_pageload.js` -- canonical page-load test runner.
- `speculative-connect.js` -- privileged-call + custom metric.

**Test file** (excerpt of
[speculative-connect.js](https://searchfox.org/mozilla-central/source/testing/raptor/browsertime/speculative-connect.js),
[bug 1818798](https://bugzilla.mozilla.org/show_bug.cgi?id=1818798)):

```javascript
const { logTest } = require("./utils/profiling");

module.exports = logTest(
  "speculative connect pageload",
  async function (context, commands) {
    const url = "https://en.wikipedia.org/wiki/Barack_Obama";

    await commands.navigate("about:blank");
    await commands.wait.byTime(1000);

    // Privileged JS to trigger a speculative connection.
    const script = `
      var URI = Services.io.newURI("${url}");
      var principal = Services.scriptSecurityManager
        .createContentPrincipal(URI, {});
      Services.io.speculativeConnect(URI, principal, callbacks, false);
    `;
    commands.js.runPrivileged(script);
    await commands.wait.byTime(1000);

    // Measure the pageload.
    await commands.measure.start();
    await commands.navigate(url);
    await commands.measure.stop();

    // Report a custom metric alongside the pageload measurements.
    const connect_time = await commands.js.run(
      `return (window.performance.timing.connectEnd -
               window.performance.timing.navigationStart);`
    );
    await commands.measure.addObject({
      custom_data: { connect_time },
    });

    return true;
  }
);
```

The wiring (manifest entry, taskcluster kind, alert thresholds, etc.) is
larger than for MozPerftest tests -- see the
[contributing guide](contributing.md) for adding new Raptor tests, then
[Raptor documentation](raptor.md) for the full reference.

See [Getting Help] before adding a new Raptor test.

## Talos

Talos is the legacy framework. **Do not add new tests here** unless there is
a specific limitation that prevents using MozPerftest -- see [Getting Help]
first.

A Talos test is a Python class that subclasses `TsBase` (or another base)
and is registered via `@register_test()` in
[testing/talos/talos/test.py](https://searchfox.org/mozilla-central/source/testing/talos/talos/test.py).
The class fields configure the runner (URL, cycles, filters, units, etc.).

**Example** (excerpt of `ts_paint` from
[testing/talos/talos/test.py](https://searchfox.org/mozilla-central/source/testing/talos/talos/test.py)):

```python
@register_test()
class ts_paint(TsBase):
    """
    Launches tspaint_test.html with the current timestamp in the url,
    waits for [MozAfterPaint and onLoad] to fire, then records the end
    time and calculates the time to startup.
    """

    cycles = 20
    timeout = 150
    gecko_profile_startup = True
    url = "startup_test/tspaint_test.html"
    filters = filter.ignore_first.prepare(1) + filter.median.prepare()
    tpmozafterpaint = True
    unit = "ms"
```

The page (`startup_test/tspaint_test.html`) lives under
[testing/talos/talos/](https://searchfox.org/mozilla-central/source/testing/talos/talos/startup_test);
manifests and config wiring are also in that tree. See the
[Talos documentation](talos.md) for the full reference.

## AWSY

AWSY (Are We Slim Yet) tracks memory usage across builds.

An AWSY test is a Python class that subclasses `AwsyTestCase`. It declares
the perfherder suites, the memory checkpoints to capture from
`about:memory`, and the URLs to load. AWSY then drives the browser through
those steps and reports the configured metrics.

**Example** (excerpt of
[testing/awsy/awsy/test_base_memory_usage.py](https://searchfox.org/mozilla-central/source/testing/awsy/awsy/test_base_memory_usage.py)):

```python
from awsy.awsy_test_case import AwsyTestCase

CHECKPOINTS = [
    {
        "name": "After tabs open [+30s, forced GC]",
        "path": "memory-report-TabsOpenForceGC-4.json.gz",
        "name_filter": ["web ", "Web Content"],
        "median": True,
    },
]

PERF_SUITES = [
    {"name": "Base Content Resident Unique Memory", "node": "resident-unique"},
    {"name": "Base Content Heap Unclassified", "node": "explicit/heap-unclassified"},
    {"name": "Base Content JS", "node": "js-main-runtime/", "alertThreshold": 0.25},
    {"name": "Base Content Explicit", "node": "explicit/"},
]


class TestMemoryUsage(AwsyTestCase):
    """Loads about:memory and reports content-process memory usage."""

    def urls(self):
        return self._urls

    def perf_suites(self):
        return PERF_SUITES

    def perf_checkpoints(self):
        return CHECKPOINTS
```

Run locally:

```
./mach awsy-test --base
```

See the
[AWSY documentation](https://firefox-source-docs.mozilla.org/testing/perfdocs/awsy.html#awsy)
for the full set of fields and configuration options.

## Running Tests in CI

To run your test in CI you need to add a task definition under the
appropriate taskcluster `kinds/` directory:

- MozPerftest: [taskcluster/kinds/perftest/](https://searchfox.org/mozilla-central/source/taskcluster/kinds/perftest)
- Raptor and browsertime: [taskcluster/kinds/browsertime/](https://searchfox.org/mozilla-central/source/taskcluster/kinds/browsertime)

This is what determines which platforms and configurations your test runs on.
If you're not sure where your task belongs, see the [Getting Help] section
below -- the perf team can help.

For xpcshell and mochitest MozPerftest tests, you may also need to update the
`_TRY_MAPPING` variable in
[mozperftest/utils.py](https://searchfox.org/mozilla-central/search?q=_TRY_MAPPING&path=mozperftest)
so CI can locate your test file.

Once your test is registered, `./mach try perf` will include it in try
pushes -- but only if the test matches an existing category. If you can't
find your test in the category picker, run with `--full` to fall back to
the full `./mach try fuzzy` interface, or add a new category for your test.
To reproduce a specific alert:

```
./mach try perf --alert <ALERT-NUMBER>
```

Note that `--alert` searches across all tasks regardless of category. For
more details, see [Mach Try Perf](mach-try-perf.md).

## Getting Help

Reach out to the Performance Testing and Tooling team in the
[#perftest channel on Matrix](https://matrix.to/#/#perftest:mozilla.org)
or #perf-help on Slack.