111 lines
3.8 KiB
Markdown
111 lines
3.8 KiB
Markdown
# Tests
|
||
|
||
A high-level test strategy for Firefox Telemetry is defined in the
|
||
[Test Strategy document](https://docs.google.com/document/d/1Mi6va3gE4HSv5LjXNREvMa2V4q-LKIFDTwA2o4yeo_c/edit).
|
||
|
||
Firefox Telemetry is a complicated and old component.
|
||
So too are the organization and expanse of its tests.
|
||
Let’s break them down by harness.
|
||
|
||
Unless otherwise mentioned the tests live in subdirectories of
|
||
`toolkit/components/telemetry/tests`.
|
||
|
||
## Mochitest
|
||
|
||
```{eval-rst}
|
||
|
||
:Location: ``t/c/t/t/browser/``
|
||
:Language: Javascript
|
||
(`mochitest <https://firefox-source-docs.mozilla.org/testing/mochitest-plain>`__)
|
||
```
|
||
|
||
This test harness runs nearly the entire Firefox and gives access to multiple tabs and browser chrome APIs.
|
||
It requires window focus to complete correctly,
|
||
so it isn’t recommended to add new tests here.
|
||
The tests that are here maybe would be more at home as telemetry-tests-client tests as they tend to be integration tests.
|
||
|
||
## Google Test
|
||
|
||
```{eval-rst}
|
||
|
||
:Location: ``t/c/t/t/gtest/``
|
||
:Language: C++
|
||
(`googletest <https://github.com/google/googletest>`_)
|
||
```
|
||
|
||
This test harness runs a specially-built gtest shell around libxul which allows you to write unit tests against public C++ APIs.
|
||
It should be used to test the C++ API and core of Firefox Telemetry.
|
||
This is for tests like
|
||
“Do we correctly accumulate to bucket 0 if I pass -1 to `Telemetry::Accumulate`?”
|
||
|
||
## Integration Tests (telemetry-tests-client)
|
||
|
||
```{eval-rst}
|
||
|
||
:Location: ``t/c/t/t/marionette/tests/client``
|
||
:Language: Python
|
||
(`unittest <https://docs.python.org/3/library/unittest.html>`__)
|
||
```
|
||
|
||
At time of writing there are two “editions” of integration tests.
|
||
Prefer writing new tests in telemetry-tests-client
|
||
(the unittest-based one in `t/c/t/t/marionette/tests/client`)
|
||
while we evaluate CI support for telemetry-integration-tests.
|
||
|
||
More info: {doc}`./integration_tests/index`
|
||
|
||
## Definitions Files Tests
|
||
|
||
```{eval-rst}
|
||
|
||
:Location: ``t/c/t/t/python``
|
||
:Language: Python
|
||
(`unittest <https://docs.python.org/3/library/unittest.html>`__)
|
||
```
|
||
|
||
This harness pulls in the parsers and scripts used to turn JSON and YAML probe definitions into code.
|
||
It should be used to test the build scripts and formats of the definitions files
|
||
Histograms.json, Scalars.yaml, and Events.yaml.
|
||
This is for tests like
|
||
“Does the build fail if someone forgot to put in a bug number for a new Histogram?”.
|
||
|
||
## xpcshell
|
||
|
||
```{eval-rst}
|
||
|
||
:Location: ``t/c/t/t/unit``
|
||
:Language: Javascript
|
||
(`xpcshell <https://firefox-source-docs.mozilla.org/testing/xpcshell>`__)
|
||
```
|
||
|
||
This test harness uses a stripped-down shell of the Firefox browser to run privileged Javascript.
|
||
It should be used to write unit tests for the Javascript API and app-level logic of Firefox Telemetry.
|
||
This is for tests like
|
||
“Do we correctly accumulate to bucket 0 if I pass -1 to `Telemetry.getHistogramById(...).add`?”
|
||
and
|
||
“Do we reschedule pings that want to be sent near local midnight?”.
|
||
|
||
Since these tests are easy to write and quick to run we have in the past bent this harness in a few interesting shapes
|
||
(see PingServer)
|
||
to have it support integration tests as well.
|
||
New integration tests should use telemetry-tests-client instead.
|
||
|
||
## Instrumentation Tests
|
||
|
||
```{eval-rst}
|
||
|
||
:Location: Various
|
||
:Language: Usually Javascript
|
||
(`xpcshell <https://firefox-source-docs.mozilla.org/testing/xpcshell>`__ or
|
||
`mochitest <https://firefox-source-docs.mozilla.org/testing/mochitest-plain>`__)
|
||
```
|
||
|
||
In addition to the tests of Firefox Telemetry,
|
||
other code owners have written tests that ensure that their code records appropriate values to Telemetry.
|
||
They should use the
|
||
`toolkit/components/telemetry/tests/unit/TelemetryTestUtils.sys.mjs`
|
||
module to make their lives easier.
|
||
This can be used for tests like
|
||
“If five bookmarks are read from the database,
|
||
does the bookmark count Histogram have a value of 5 in it?”.
|