Valerie Young
fe1161da6d
Automatic update from web-platform-tests Create new test type `aamtest` for accessibility API testing This PR adds a new test type to test accessibility APIs exposed by browsers, as defined by the [ARIA](https://w3c.github.io/aria/), [Core-AAM](https://w3c.github.io/core-aam/) and [HTML-AAM](https://w3c.github.io/html-aam/) specifications. The now merged RFC can be found [here](https://github.com/web-platform-tests/rfcs/pull/204). This is a replacement for: https://github.com/web-platform-tests/wpt/pull/53733 Instead of extending testharness, I added a new test type (`aamtest`) that is similar to `wdspec` tests and uses a lot of the same infrastructure. This idea came from foolip in [this comment on the RFC](https://github.com/web-platform-tests/rfcs/pull/204#issuecomment-3655616580), and I think it looks good! Here is a PR that adds documentation for this test type: https://github.com/web-platform-tests/wpt/pull/58632 -- wpt-commits: 2827ad177da748e8785c6bb3cabc03c8ce6f589b wpt-pr: 57696
83 lines
3.6 KiB
Markdown
83 lines
3.6 KiB
Markdown
# Test Suite Design
|
|
|
|
The vast majority of the test suite is formed of HTML pages, which can
|
|
be loaded in a browser and either programmatically provide a result or
|
|
provide a set of steps to run the test and obtain the result.
|
|
|
|
The tests are, in general, short, cross-platform, and self-contained,
|
|
and should be easy to run in any browser.
|
|
|
|
|
|
## Test Layout
|
|
|
|
Most of the repository's top-level directories hold tests for specific web
|
|
standards. For [W3C specs](https://www.w3.org/standards/), these directories
|
|
are typically named after the shortname of the spec (i.e. the name used for
|
|
snapshot publications under `/TR/`); for [WHATWG
|
|
specs](https://spec.whatwg.org/), they are typically named after the subdomain
|
|
of the spec (i.e. trimming `.spec.whatwg.org` from the URL); for other specs,
|
|
something deemed sensible is used. The `css/` directory contains test suites
|
|
for [the CSS Working Group
|
|
specifications](https://www.w3.org/Style/CSS/current-work).
|
|
|
|
Within the specification-specific directory there are two common ways
|
|
of laying out tests: the first is a flat structure which is sometimes
|
|
adopted for very short specifications; the alternative is a nested
|
|
structure with each subdirectory corresponding to the id of a heading
|
|
in the specification. The latter provides some implicit metadata about
|
|
the part of a specification being tested according to its location in
|
|
the filesystem, and is preferred for larger specifications.
|
|
|
|
For example, tests in HTML for ["The History
|
|
interface"](https://html.spec.whatwg.org/multipage/history.html#the-history-interface)
|
|
are located in `html/browsers/history/the-history-interface/`.
|
|
|
|
Many directories also include files named `META.yml` and/or `WEB_FEATURES.yml`.
|
|
These files contain test metadata; please refer to [Out-of-band
|
|
Metadata][metadata] for more information about their purpose and format.
|
|
|
|
Various resources that tests depend on are in `common`, `images`, `fonts`,
|
|
`media`, and `resources`.
|
|
|
|
## Test Types
|
|
|
|
Tests in this project use a few different approaches to verify expected
|
|
behavior. The tests can be classified based on the way they express
|
|
expectations:
|
|
|
|
* Rendering tests ensure that the browser graphically displays pages as
|
|
expected. There are a few different ways this is done:
|
|
|
|
* [Reftests][] render two (or more) web pages and combine them with equality
|
|
assertions about their rendering (e.g., `A.html` and `B.html` must render
|
|
identically), run either by the user switching between tabs/windows and
|
|
trying to observe differences or through [automated
|
|
scripts][running-from-local-system].
|
|
|
|
* [Visual tests][visual] display a page where the result is determined either
|
|
by a human looking at it or by comparing it with a saved screenshot for
|
|
that user agent on that platform.
|
|
|
|
* [testharness.js][] tests verify that JavaScript interfaces behave as
|
|
expected. They get their name from the JavaScript harness that's used to
|
|
execute them.
|
|
|
|
* [wdspec][] tests are written in Python and test [the WebDriver browser
|
|
automation protocol](https://w3c.github.io/webdriver/)
|
|
|
|
* [aamtest][] tests are written in Python and test accessibility API mappings,
|
|
such as [Core Accessibility API Mappings](https://w3c.github.io/core-aam/)
|
|
or [HTML Accessibility API Mappings](https://w3c.github.io/html-aam/).
|
|
|
|
* [Manual tests][manual] rely on a human to run them and determine their
|
|
result.
|
|
|
|
[reftests]: writing-tests/reftests
|
|
[testharness.js]: writing-tests/testharness
|
|
[visual]: writing-tests/visual
|
|
[manual]: writing-tests/manual
|
|
[metadata]: writing-tests/out-of-band-metadata
|
|
[running-from-local-system]: running-tests/from-local-system
|
|
[wdspec]: writing-tests/wdspec
|
|
[aamtest]: writing-tests/aamtest
|