Bug 2063085 - Part 2: Add the Raptor power test definition for media playback. r=perftest-reviewers,frontend-codestyle-reviewers,sparky
Nothing in Raptor measures what watching a video costs in power. Add a suite that does, as two tracks that each force the decode path they are named for. Forcing it is what lets a subtest name state what was measured. The software track is sound by construction, since disabling hardware decode is a UserDisable no fallback can undo, and it is Firefox-only because a manifest's preferences are not applied to other browsers. The hardware track is weaker: the preference only lifts the gfxInfo blocklist, and H.264 falls back to software silently, so that path is assumed rather than asserted and the manifest says so. The workload is not added here. It lives in the shared benchmark repository, which Raptor clones at a named revision, so the same page can be driven by other harnesses. The two are versioned separately, so the script feature-detects the entry points it needs and fails immediately when they are absent, rather than waiting for a state that will never arrive. Energy over the chip package is the only series that alerts, at five percent. The presented frame rate guards it, since playback that stops keeping up decodes less and would otherwise read as an improvement. The reported value is a median of the measurement windows rather than a mean, so a cycle disturbed by something other than playback does not move it. Five percent because it is roughly four times the run-to-run scatter, measured over ten runs on ten machines and once Perfherder's averaging over a dozen revisions is accounted for. Worth revisiting in either direction once there is trunk history. The generated documentation is included here rather than split out, because the perfdocs check verifies each commit independently and a manifest without its generated entry fails it. Differential Revision: https://phabricator.services.mozilla.com/D321062
This commit is contained in:
Alastor Wu
committed by
alwu@mozilla.com
alwu@mozilla.com
parent
c9dc9388da
commit
a6a4840e87
@@ -449,6 +449,7 @@ export default [
|
||||
"testing/raptor/browsertime/indexeddb_open.js",
|
||||
"testing/raptor/browsertime/indexeddb_write.js",
|
||||
"testing/raptor/browsertime/jetstream3.js",
|
||||
"testing/raptor/browsertime/media_playback.js",
|
||||
"testing/raptor/browsertime/media_seek.js",
|
||||
"testing/raptor/browsertime/motionmark-1-3.js",
|
||||
"testing/raptor/browsertime/network_bench.js",
|
||||
|
||||
@@ -190,6 +190,16 @@ Similar to SpeedIndex, except that it uses the structural similarity index measu
|
||||
* **Mobile**: {ref}`allrecipes <allrecipes-m>`, {ref}`amazon <amazon-m>`, {ref}`amazon-search <amazon-search-m>`, {ref}`bild-de <bild-de-m>`, {ref}`bing <bing-m>`, {ref}`bing-search-restaurants <bing-search-restaurants-m>`, {ref}`booking <booking-m>`, {ref}`cnn <cnn-m>`, {ref}`cnn-ampstories <cnn-ampstories-m>`, {ref}`dailymail <dailymail-m>`, {ref}`ebay-kleinanzeigen <ebay-kleinanzeigen-m>`, {ref}`ebay-kleinanzeigen-search <ebay-kleinanzeigen-search-m>`, {ref}`espn <espn-m>`, {ref}`facebook <facebook-m>`, {ref}`facebook-cristiano <facebook-cristiano-m>`, {ref}`google <google-m>`, {ref}`google-maps <google-maps-m>`, {ref}`google-search-restaurants <google-search-restaurants-m>`, {ref}`imdb <imdb-m>`, {ref}`instagram <instagram-m>`, {ref}`microsoft-support <microsoft-support-m>`, {ref}`reddit <reddit-m>`, {ref}`sina <sina-m>`, {ref}`stackoverflow <stackoverflow-m>`, {ref}`wikipedia <wikipedia-m>`, {ref}`youtube <youtube-m>`, {ref}`youtube-watch <youtube-watch-m>`
|
||||
|
||||
|
||||
### Power Usage
|
||||
|
||||
Energy in microwatt-hours consumed over the measured interval, read from the platform's own energy counters. The package channel covers the whole chip and is the steadiest of the three; the core and GPU planes are narrower and noisier. Sampled system-wide by a separate process rather than from inside the browser, which is what makes the measurement independent of which browser is under test.
|
||||
|
||||
|
||||
* **Aliases**: powerUsage_cpu_cores, powerUsage_cpu_package, powerUsage_gpu
|
||||
* **Tests using it**:
|
||||
* **Custom**: {ref}`mp-1080p30-av1-hw <mp-1080p30-av1-hw-c>`, {ref}`mp-1080p30-av1-sw <mp-1080p30-av1-sw-c>`, {ref}`mp-1080p30-h264-hw <mp-1080p30-h264-hw-c>`, {ref}`mp-1080p30-h264-sw <mp-1080p30-h264-sw-c>`, {ref}`mp-1080p30-vp9-hw <mp-1080p30-vp9-hw-c>`, {ref}`mp-1080p30-vp9-sw <mp-1080p30-vp9-sw-c>`, {ref}`mp-2160p30-av1-hw <mp-2160p30-av1-hw-c>`, {ref}`mp-2160p30-h264-hw <mp-2160p30-h264-hw-c>`, {ref}`mp-2160p30-vp9-hw <mp-2160p30-vp9-hw-c>`
|
||||
|
||||
|
||||
### Seek Latency
|
||||
|
||||
Time (in ms) from issuing a buffered currentTime seek to the seeked event, measured with the video decoder warm (kept playing) and cold (released while paused and recreated per seek).
|
||||
|
||||
@@ -19844,6 +19844,455 @@ Browsertime tests that use a custom pageload test script. These use the pageload
|
||||
|
||||
::::
|
||||
|
||||
(mp-1080p30-av1-hw-c)=
|
||||
|
||||
::::{dropdown} mp-1080p30-av1-hw
|
||||
:class-container: anchor-id-mp-1080p30-av1-hw-c
|
||||
|
||||
* **Command to Run Locally**
|
||||
|
||||
```
|
||||
./mach raptor -t mp-1080p30-av1-hw
|
||||
```
|
||||
|
||||
**Owner**: Media Team
|
||||
|
||||
**Description**: Measures the power used by sustained 1080p30 AV1 playback with hardware decode forced, alongside the frame rate that shows whether playback kept up. This is the cross-browser track.
|
||||
|
||||
* **alert on**: [powerUsage_cpu_package](raptor-metrics.md#power-usage)
|
||||
* **alert threshold**: 5.0
|
||||
* **apps**: firefox, chrome
|
||||
* **benchmark webserver**: true
|
||||
* **browser cycles**: 1
|
||||
* **custom data**: true
|
||||
* **expected**: pass
|
||||
* **gecko profile features**: stackwalk,processcpu,power
|
||||
* **gecko profile interval**: 1
|
||||
* **gecko profile threads**: GeckoMain,MediaDecoderStateMachine,MediaPDecoder
|
||||
* **link searchfox**: {searchfox}`testing/raptor/raptor/tests/custom/browsertime-media-playback.toml#48`
|
||||
* **lower is better**: true
|
||||
* **measure**: powerUsage_cpu_package, powerUsage_cpu_cores, powerUsage_gpu, presentedFps
|
||||
* **page cycles**: 15
|
||||
* **page timeout**: 60000
|
||||
* **preferences**: media.autoplay.default=0 media.autoplay.blocking_policy=0 media.block-autoplay-until-in-foreground=false media.suspend-background-video.enabled=false media.hardware-video-decoding.force-enabled=true
|
||||
* **repository**: https://github.com/mozilla/perf-automation
|
||||
* **repository path**: benchmarks/media-playback
|
||||
* **repository revision**: 7046a4bd167a082989802856aaa86b77e84e939a
|
||||
* **sparse checkout**: true
|
||||
* **subtest lower is better**: false
|
||||
* **subtest unit**: uWh
|
||||
* **support class**: media_playback.py
|
||||
* **test script**: media_playback.js
|
||||
* **test url**: <http://\<host\>:\<port\>/index.html?clip=av1_1080p30&driver=single&raptor=1>
|
||||
* **type**: pageload
|
||||
* **unit**: uWh
|
||||
* **use live sites**: false
|
||||
::::
|
||||
|
||||
(mp-1080p30-av1-sw-c)=
|
||||
|
||||
::::{dropdown} mp-1080p30-av1-sw
|
||||
:class-container: anchor-id-mp-1080p30-av1-sw-c
|
||||
|
||||
* **Command to Run Locally**
|
||||
|
||||
```
|
||||
./mach raptor -t mp-1080p30-av1-sw
|
||||
```
|
||||
|
||||
**Owner**: Media Team
|
||||
|
||||
**Description**: Measures the power used by sustained 1080p30 AV1 playback with software decode forced, alongside the frame rate that shows whether playback kept up. Firefox only, since the forcing preference is not applied to other browsers.
|
||||
|
||||
* **alert on**: [powerUsage_cpu_package](raptor-metrics.md#power-usage)
|
||||
* **alert threshold**: 5.0
|
||||
* **apps**: firefox
|
||||
* **benchmark webserver**: true
|
||||
* **browser cycles**: 1
|
||||
* **custom data**: true
|
||||
* **expected**: pass
|
||||
* **gecko profile features**: stackwalk,processcpu,power
|
||||
* **gecko profile interval**: 1
|
||||
* **gecko profile threads**: GeckoMain,MediaDecoderStateMachine,MediaPDecoder
|
||||
* **link searchfox**: {searchfox}`testing/raptor/raptor/tests/custom/browsertime-media-playback-sw.toml#39`
|
||||
* **lower is better**: true
|
||||
* **measure**: powerUsage_cpu_package, powerUsage_cpu_cores, powerUsage_gpu, presentedFps
|
||||
* **page cycles**: 15
|
||||
* **page timeout**: 60000
|
||||
* **preferences**: media.autoplay.default=0 media.autoplay.blocking_policy=0 media.block-autoplay-until-in-foreground=false media.suspend-background-video.enabled=false media.hardware-video-decoding.enabled=false
|
||||
* **repository**: https://github.com/mozilla/perf-automation
|
||||
* **repository path**: benchmarks/media-playback
|
||||
* **repository revision**: 7046a4bd167a082989802856aaa86b77e84e939a
|
||||
* **sparse checkout**: true
|
||||
* **subtest lower is better**: false
|
||||
* **subtest unit**: uWh
|
||||
* **support class**: media_playback.py
|
||||
* **test script**: media_playback.js
|
||||
* **test url**: <http://\<host\>:\<port\>/index.html?clip=av1_1080p30&driver=single&raptor=1>
|
||||
* **type**: pageload
|
||||
* **unit**: uWh
|
||||
* **use live sites**: false
|
||||
::::
|
||||
|
||||
(mp-1080p30-h264-hw-c)=
|
||||
|
||||
::::{dropdown} mp-1080p30-h264-hw
|
||||
:class-container: anchor-id-mp-1080p30-h264-hw-c
|
||||
|
||||
* **Command to Run Locally**
|
||||
|
||||
```
|
||||
./mach raptor -t mp-1080p30-h264-hw
|
||||
```
|
||||
|
||||
**Owner**: Media Team
|
||||
|
||||
**Description**: Measures the power used by sustained 1080p30 H264 playback with hardware decode forced, alongside the frame rate that shows whether playback kept up. This is the cross-browser track.
|
||||
|
||||
* **alert on**: [powerUsage_cpu_package](raptor-metrics.md#power-usage)
|
||||
* **alert threshold**: 5.0
|
||||
* **apps**: firefox, chrome
|
||||
* **benchmark webserver**: true
|
||||
* **browser cycles**: 1
|
||||
* **custom data**: true
|
||||
* **expected**: pass
|
||||
* **gecko profile features**: stackwalk,processcpu,power
|
||||
* **gecko profile interval**: 1
|
||||
* **gecko profile threads**: GeckoMain,MediaDecoderStateMachine,MediaPDecoder
|
||||
* **link searchfox**: {searchfox}`testing/raptor/raptor/tests/custom/browsertime-media-playback.toml#52`
|
||||
* **lower is better**: true
|
||||
* **measure**: powerUsage_cpu_package, powerUsage_cpu_cores, powerUsage_gpu, presentedFps
|
||||
* **page cycles**: 15
|
||||
* **page timeout**: 60000
|
||||
* **preferences**: media.autoplay.default=0 media.autoplay.blocking_policy=0 media.block-autoplay-until-in-foreground=false media.suspend-background-video.enabled=false media.hardware-video-decoding.force-enabled=true
|
||||
* **repository**: https://github.com/mozilla/perf-automation
|
||||
* **repository path**: benchmarks/media-playback
|
||||
* **repository revision**: 7046a4bd167a082989802856aaa86b77e84e939a
|
||||
* **sparse checkout**: true
|
||||
* **subtest lower is better**: false
|
||||
* **subtest unit**: uWh
|
||||
* **support class**: media_playback.py
|
||||
* **test script**: media_playback.js
|
||||
* **test url**: <http://\<host\>:\<port\>/index.html?clip=h264_1080p30&driver=single&raptor=1>
|
||||
* **type**: pageload
|
||||
* **unit**: uWh
|
||||
* **use live sites**: false
|
||||
::::
|
||||
|
||||
(mp-1080p30-h264-sw-c)=
|
||||
|
||||
::::{dropdown} mp-1080p30-h264-sw
|
||||
:class-container: anchor-id-mp-1080p30-h264-sw-c
|
||||
|
||||
* **Command to Run Locally**
|
||||
|
||||
```
|
||||
./mach raptor -t mp-1080p30-h264-sw
|
||||
```
|
||||
|
||||
**Owner**: Media Team
|
||||
|
||||
**Description**: Measures the power used by sustained 1080p30 H264 playback with software decode forced, alongside the frame rate that shows whether playback kept up. Firefox only, since the forcing preference is not applied to other browsers.
|
||||
|
||||
* **alert on**: [powerUsage_cpu_package](raptor-metrics.md#power-usage)
|
||||
* **alert threshold**: 5.0
|
||||
* **apps**: firefox
|
||||
* **benchmark webserver**: true
|
||||
* **browser cycles**: 1
|
||||
* **custom data**: true
|
||||
* **expected**: pass
|
||||
* **gecko profile features**: stackwalk,processcpu,power
|
||||
* **gecko profile interval**: 1
|
||||
* **gecko profile threads**: GeckoMain,MediaDecoderStateMachine,MediaPDecoder
|
||||
* **link searchfox**: {searchfox}`testing/raptor/raptor/tests/custom/browsertime-media-playback-sw.toml#43`
|
||||
* **lower is better**: true
|
||||
* **measure**: powerUsage_cpu_package, powerUsage_cpu_cores, powerUsage_gpu, presentedFps
|
||||
* **page cycles**: 15
|
||||
* **page timeout**: 60000
|
||||
* **preferences**: media.autoplay.default=0 media.autoplay.blocking_policy=0 media.block-autoplay-until-in-foreground=false media.suspend-background-video.enabled=false media.hardware-video-decoding.enabled=false
|
||||
* **repository**: https://github.com/mozilla/perf-automation
|
||||
* **repository path**: benchmarks/media-playback
|
||||
* **repository revision**: 7046a4bd167a082989802856aaa86b77e84e939a
|
||||
* **sparse checkout**: true
|
||||
* **subtest lower is better**: false
|
||||
* **subtest unit**: uWh
|
||||
* **support class**: media_playback.py
|
||||
* **test script**: media_playback.js
|
||||
* **test url**: <http://\<host\>:\<port\>/index.html?clip=h264_1080p30&driver=single&raptor=1>
|
||||
* **type**: pageload
|
||||
* **unit**: uWh
|
||||
* **use live sites**: false
|
||||
::::
|
||||
|
||||
(mp-1080p30-vp9-hw-c)=
|
||||
|
||||
::::{dropdown} mp-1080p30-vp9-hw
|
||||
:class-container: anchor-id-mp-1080p30-vp9-hw-c
|
||||
|
||||
* **Command to Run Locally**
|
||||
|
||||
```
|
||||
./mach raptor -t mp-1080p30-vp9-hw
|
||||
```
|
||||
|
||||
**Owner**: Media Team
|
||||
|
||||
**Description**: Measures the power used by sustained 1080p30 VP9 playback with hardware decode forced, alongside the frame rate that shows whether playback kept up. This is the cross-browser track.
|
||||
|
||||
* **alert on**: [powerUsage_cpu_package](raptor-metrics.md#power-usage)
|
||||
* **alert threshold**: 5.0
|
||||
* **apps**: firefox, chrome
|
||||
* **benchmark webserver**: true
|
||||
* **browser cycles**: 1
|
||||
* **custom data**: true
|
||||
* **expected**: pass
|
||||
* **gecko profile features**: stackwalk,processcpu,power
|
||||
* **gecko profile interval**: 1
|
||||
* **gecko profile threads**: GeckoMain,MediaDecoderStateMachine,MediaPDecoder
|
||||
* **link searchfox**: {searchfox}`testing/raptor/raptor/tests/custom/browsertime-media-playback.toml#56`
|
||||
* **lower is better**: true
|
||||
* **measure**: powerUsage_cpu_package, powerUsage_cpu_cores, powerUsage_gpu, presentedFps
|
||||
* **page cycles**: 15
|
||||
* **page timeout**: 60000
|
||||
* **preferences**: media.autoplay.default=0 media.autoplay.blocking_policy=0 media.block-autoplay-until-in-foreground=false media.suspend-background-video.enabled=false media.hardware-video-decoding.force-enabled=true
|
||||
* **repository**: https://github.com/mozilla/perf-automation
|
||||
* **repository path**: benchmarks/media-playback
|
||||
* **repository revision**: 7046a4bd167a082989802856aaa86b77e84e939a
|
||||
* **sparse checkout**: true
|
||||
* **subtest lower is better**: false
|
||||
* **subtest unit**: uWh
|
||||
* **support class**: media_playback.py
|
||||
* **test script**: media_playback.js
|
||||
* **test url**: <http://\<host\>:\<port\>/index.html?clip=vp9_1080p30&driver=single&raptor=1>
|
||||
* **type**: pageload
|
||||
* **unit**: uWh
|
||||
* **use live sites**: false
|
||||
::::
|
||||
|
||||
(mp-1080p30-vp9-sw-c)=
|
||||
|
||||
::::{dropdown} mp-1080p30-vp9-sw
|
||||
:class-container: anchor-id-mp-1080p30-vp9-sw-c
|
||||
|
||||
* **Command to Run Locally**
|
||||
|
||||
```
|
||||
./mach raptor -t mp-1080p30-vp9-sw
|
||||
```
|
||||
|
||||
**Owner**: Media Team
|
||||
|
||||
**Description**: Measures the power used by sustained 1080p30 VP9 playback with software decode forced, alongside the frame rate that shows whether playback kept up. Firefox only, since the forcing preference is not applied to other browsers.
|
||||
|
||||
* **alert on**: [powerUsage_cpu_package](raptor-metrics.md#power-usage)
|
||||
* **alert threshold**: 5.0
|
||||
* **apps**: firefox
|
||||
* **benchmark webserver**: true
|
||||
* **browser cycles**: 1
|
||||
* **custom data**: true
|
||||
* **expected**: pass
|
||||
* **gecko profile features**: stackwalk,processcpu,power
|
||||
* **gecko profile interval**: 1
|
||||
* **gecko profile threads**: GeckoMain,MediaDecoderStateMachine,MediaPDecoder
|
||||
* **link searchfox**: {searchfox}`testing/raptor/raptor/tests/custom/browsertime-media-playback-sw.toml#47`
|
||||
* **lower is better**: true
|
||||
* **measure**: powerUsage_cpu_package, powerUsage_cpu_cores, powerUsage_gpu, presentedFps
|
||||
* **page cycles**: 15
|
||||
* **page timeout**: 60000
|
||||
* **preferences**: media.autoplay.default=0 media.autoplay.blocking_policy=0 media.block-autoplay-until-in-foreground=false media.suspend-background-video.enabled=false media.hardware-video-decoding.enabled=false
|
||||
* **repository**: https://github.com/mozilla/perf-automation
|
||||
* **repository path**: benchmarks/media-playback
|
||||
* **repository revision**: 7046a4bd167a082989802856aaa86b77e84e939a
|
||||
* **sparse checkout**: true
|
||||
* **subtest lower is better**: false
|
||||
* **subtest unit**: uWh
|
||||
* **support class**: media_playback.py
|
||||
* **test script**: media_playback.js
|
||||
* **test url**: <http://\<host\>:\<port\>/index.html?clip=vp9_1080p30&driver=single&raptor=1>
|
||||
* **type**: pageload
|
||||
* **unit**: uWh
|
||||
* **use live sites**: false
|
||||
::::
|
||||
|
||||
(mp-2160p30-av1-hw-c)=
|
||||
|
||||
::::{dropdown} mp-2160p30-av1-hw
|
||||
:class-container: anchor-id-mp-2160p30-av1-hw-c
|
||||
|
||||
* **Command to Run Locally**
|
||||
|
||||
```
|
||||
./mach raptor -t mp-2160p30-av1-hw
|
||||
```
|
||||
|
||||
**Owner**: Media Team
|
||||
|
||||
**Description**: Measures the power used by sustained 2160p30 AV1 playback with hardware decode forced, alongside the frame rate that shows whether playback kept up. Four times the pixels of the 1080p subtests, so playback accounts for more of the energy measured.
|
||||
|
||||
* **alert on**: [powerUsage_cpu_package](raptor-metrics.md#power-usage)
|
||||
* **alert threshold**: 5.0
|
||||
* **apps**: firefox, chrome
|
||||
* **benchmark webserver**: true
|
||||
* **browser cycles**: 1
|
||||
* **custom data**: true
|
||||
* **expected**: pass
|
||||
* **gecko profile features**: stackwalk,processcpu,power
|
||||
* **gecko profile interval**: 1
|
||||
* **gecko profile threads**: GeckoMain,MediaDecoderStateMachine,MediaPDecoder
|
||||
* **link searchfox**: {searchfox}`testing/raptor/raptor/tests/custom/browsertime-media-playback.toml#60`
|
||||
* **lower is better**: true
|
||||
* **measure**: powerUsage_cpu_package, powerUsage_cpu_cores, powerUsage_gpu, presentedFps
|
||||
* **page cycles**: 15
|
||||
* **page timeout**: 60000
|
||||
* **preferences**: media.autoplay.default=0 media.autoplay.blocking_policy=0 media.block-autoplay-until-in-foreground=false media.suspend-background-video.enabled=false media.hardware-video-decoding.force-enabled=true
|
||||
* **repository**: https://github.com/mozilla/perf-automation
|
||||
* **repository path**: benchmarks/media-playback
|
||||
* **repository revision**: 7046a4bd167a082989802856aaa86b77e84e939a
|
||||
* **sparse checkout**: true
|
||||
* **subtest lower is better**: false
|
||||
* **subtest unit**: uWh
|
||||
* **support class**: media_playback.py
|
||||
* **test script**: media_playback.js
|
||||
* **test url**: <http://\<host\>:\<port\>/index.html?clip=av1_2160p30&driver=single&raptor=1>
|
||||
* **type**: pageload
|
||||
* **unit**: uWh
|
||||
* **use live sites**: false
|
||||
::::
|
||||
|
||||
(mp-2160p30-h264-hw-c)=
|
||||
|
||||
::::{dropdown} mp-2160p30-h264-hw
|
||||
:class-container: anchor-id-mp-2160p30-h264-hw-c
|
||||
|
||||
* **Command to Run Locally**
|
||||
|
||||
```
|
||||
./mach raptor -t mp-2160p30-h264-hw
|
||||
```
|
||||
|
||||
**Owner**: Media Team
|
||||
|
||||
**Description**: Measures the power used by sustained 2160p30 H264 playback with hardware decode forced, alongside the frame rate that shows whether playback kept up. Four times the pixels of the 1080p subtests, so playback accounts for more of the energy measured.
|
||||
|
||||
* **alert on**: [powerUsage_cpu_package](raptor-metrics.md#power-usage)
|
||||
* **alert threshold**: 5.0
|
||||
* **apps**: firefox, chrome
|
||||
* **benchmark webserver**: true
|
||||
* **browser cycles**: 1
|
||||
* **custom data**: true
|
||||
* **expected**: pass
|
||||
* **gecko profile features**: stackwalk,processcpu,power
|
||||
* **gecko profile interval**: 1
|
||||
* **gecko profile threads**: GeckoMain,MediaDecoderStateMachine,MediaPDecoder
|
||||
* **link searchfox**: {searchfox}`testing/raptor/raptor/tests/custom/browsertime-media-playback.toml#64`
|
||||
* **lower is better**: true
|
||||
* **measure**: powerUsage_cpu_package, powerUsage_cpu_cores, powerUsage_gpu, presentedFps
|
||||
* **page cycles**: 15
|
||||
* **page timeout**: 60000
|
||||
* **preferences**: media.autoplay.default=0 media.autoplay.blocking_policy=0 media.block-autoplay-until-in-foreground=false media.suspend-background-video.enabled=false media.hardware-video-decoding.force-enabled=true
|
||||
* **repository**: https://github.com/mozilla/perf-automation
|
||||
* **repository path**: benchmarks/media-playback
|
||||
* **repository revision**: 7046a4bd167a082989802856aaa86b77e84e939a
|
||||
* **sparse checkout**: true
|
||||
* **subtest lower is better**: false
|
||||
* **subtest unit**: uWh
|
||||
* **support class**: media_playback.py
|
||||
* **test script**: media_playback.js
|
||||
* **test url**: <http://\<host\>:\<port\>/index.html?clip=h264_2160p30&driver=single&raptor=1>
|
||||
* **type**: pageload
|
||||
* **unit**: uWh
|
||||
* **use live sites**: false
|
||||
::::
|
||||
|
||||
(mp-2160p30-vp9-hw-c)=
|
||||
|
||||
::::{dropdown} mp-2160p30-vp9-hw
|
||||
:class-container: anchor-id-mp-2160p30-vp9-hw-c
|
||||
|
||||
* **Command to Run Locally**
|
||||
|
||||
```
|
||||
./mach raptor -t mp-2160p30-vp9-hw
|
||||
```
|
||||
|
||||
**Owner**: Media Team
|
||||
|
||||
**Description**: Measures the power used by sustained 2160p30 VP9 playback with hardware decode forced, alongside the frame rate that shows whether playback kept up. Four times the pixels of the 1080p subtests, so playback accounts for more of the energy measured.
|
||||
|
||||
* **alert on**: [powerUsage_cpu_package](raptor-metrics.md#power-usage)
|
||||
* **alert threshold**: 5.0
|
||||
* **apps**: firefox, chrome
|
||||
* **benchmark webserver**: true
|
||||
* **browser cycles**: 1
|
||||
* **custom data**: true
|
||||
* **expected**: pass
|
||||
* **gecko profile features**: stackwalk,processcpu,power
|
||||
* **gecko profile interval**: 1
|
||||
* **gecko profile threads**: GeckoMain,MediaDecoderStateMachine,MediaPDecoder
|
||||
* **link searchfox**: {searchfox}`testing/raptor/raptor/tests/custom/browsertime-media-playback.toml#68`
|
||||
* **lower is better**: true
|
||||
* **measure**: powerUsage_cpu_package, powerUsage_cpu_cores, powerUsage_gpu, presentedFps
|
||||
* **page cycles**: 15
|
||||
* **page timeout**: 60000
|
||||
* **preferences**: media.autoplay.default=0 media.autoplay.blocking_policy=0 media.block-autoplay-until-in-foreground=false media.suspend-background-video.enabled=false media.hardware-video-decoding.force-enabled=true
|
||||
* **repository**: https://github.com/mozilla/perf-automation
|
||||
* **repository path**: benchmarks/media-playback
|
||||
* **repository revision**: 7046a4bd167a082989802856aaa86b77e84e939a
|
||||
* **sparse checkout**: true
|
||||
* **subtest lower is better**: false
|
||||
* **subtest unit**: uWh
|
||||
* **support class**: media_playback.py
|
||||
* **test script**: media_playback.js
|
||||
* **test url**: <http://\<host\>:\<port\>/index.html?clip=vp9_2160p30&driver=single&raptor=1>
|
||||
* **type**: pageload
|
||||
* **unit**: uWh
|
||||
* **use live sites**: false
|
||||
::::
|
||||
|
||||
(mp-idle-c)=
|
||||
|
||||
::::{dropdown} mp-idle
|
||||
:class-container: anchor-id-mp-idle-c
|
||||
|
||||
* **Command to Run Locally**
|
||||
|
||||
```
|
||||
./mach raptor -t mp-idle
|
||||
```
|
||||
|
||||
**Owner**: Media Team
|
||||
|
||||
**Description**: Measures the power used over the same interval with nothing playing, as the baseline the media playback subtests are read against.
|
||||
|
||||
* **alert threshold**: 5.0
|
||||
* **apps**: firefox, chrome
|
||||
* **benchmark webserver**: true
|
||||
* **browser cycles**: 1
|
||||
* **custom data**: true
|
||||
* **expected**: pass
|
||||
* **gecko profile features**: stackwalk,processcpu,power
|
||||
* **gecko profile interval**: 1
|
||||
* **gecko profile threads**: GeckoMain
|
||||
* **link searchfox**: {searchfox}`testing/raptor/raptor/tests/custom/browsertime-media-playback.toml#72`
|
||||
* **lower is better**: true
|
||||
* **measure**: powerUsage_cpu_package, powerUsage_cpu_cores, powerUsage_gpu
|
||||
* **page cycles**: 15
|
||||
* **page timeout**: 60000
|
||||
* **preferences**: media.autoplay.default=0 media.autoplay.blocking_policy=0 media.block-autoplay-until-in-foreground=false media.suspend-background-video.enabled=false media.hardware-video-decoding.force-enabled=true
|
||||
* **repository**: https://github.com/mozilla/perf-automation
|
||||
* **repository path**: benchmarks/media-playback
|
||||
* **repository revision**: 7046a4bd167a082989802856aaa86b77e84e939a
|
||||
* **sparse checkout**: true
|
||||
* **subtest lower is better**: true
|
||||
* **subtest unit**: uWh
|
||||
* **support class**: media_playback.py
|
||||
* **test script**: media_playback.js
|
||||
* **test url**: <http://\<host\>:\<port\>/index.html?driver=idle&raptor=1>
|
||||
* **type**: pageload
|
||||
* **unit**: uWh
|
||||
* **use live sites**: false
|
||||
::::
|
||||
|
||||
(prefetch-conservative-c)=
|
||||
|
||||
::::{dropdown} prefetch-conservative
|
||||
|
||||
@@ -0,0 +1,185 @@
|
||||
/* This Source Code Form is subject to the terms of the Mozilla Public
|
||||
* License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
* file, You can obtain one at http://mozilla.org/MPL/2.0/. */
|
||||
|
||||
// Sustained video playback benchmark for measuring power. The workload lives in
|
||||
// the served page; this script brackets the measured interval.
|
||||
//
|
||||
// Power is accumulated between commands.measure.start() and stop(), so the page
|
||||
// warms up and parks at 'ready' with playback running, and is only told to begin
|
||||
// its measured interval once the window is open. Start-up therefore falls outside
|
||||
// it and what remains inside is steady-state playback.
|
||||
|
||||
const { logTest } = require("./utils/profiling");
|
||||
|
||||
// Each must stay under raptor's 120s no-output timeout for a pageload test, or
|
||||
// the process is killed for silence before the script can report anything.
|
||||
const STAGE_TIMEOUT_MS = 60000;
|
||||
const WARMUP_TIMEOUT_MS = 60000;
|
||||
const MEASURE_TIMEOUT_MS = 90000;
|
||||
const POLL_INTERVAL_MS = 100;
|
||||
|
||||
// executeAsyncScript appends its completion callback as the last argument, which
|
||||
// is the only channel back from the page, so every path out of these loops calls
|
||||
// it. Polling rather than listening also catches a state reached before injection.
|
||||
//
|
||||
// Fullscreen needs user activation, which a WebDriver click supplies, so this
|
||||
// waits for the page to build its stage and then clicks. It matters because a
|
||||
// video in the content area is scaled by the height of the browser chrome, which
|
||||
// would land in the power numbers being compared between browsers.
|
||||
const WAIT_STAGED_SCRIPT = `
|
||||
const notifyDone = arguments[arguments.length - 1];
|
||||
const deadline = Date.now() + ${STAGE_TIMEOUT_MS};
|
||||
(function poll() {
|
||||
const button = document.getElementById("gesture");
|
||||
const state = window.mediaPlaybackState;
|
||||
if (state === "error") {
|
||||
return notifyDone({ error: window.mediaPlaybackError });
|
||||
}
|
||||
// Positive conditions only, because navigate() returns before the document
|
||||
// exists, and browsertime fails the run on a click it cannot perform.
|
||||
if (button && (state === "warmup" || state === "ready")) {
|
||||
const box = button.getBoundingClientRect();
|
||||
return notifyDone({
|
||||
staged: true,
|
||||
needsGesture:
|
||||
document.fullscreenElement === null && box.width > 0 && box.height > 0,
|
||||
});
|
||||
}
|
||||
if (Date.now() > deadline) {
|
||||
return notifyDone({
|
||||
error: "timed out before the page was ready to click, state=" + state,
|
||||
});
|
||||
}
|
||||
setTimeout(poll, ${POLL_INTERVAL_MS});
|
||||
})();
|
||||
`;
|
||||
|
||||
// Waits for the page to finish its warmup. Playback is already running when this
|
||||
// resolves, so the window can be opened without a gap.
|
||||
const WAIT_READY_SCRIPT = `
|
||||
const notifyDone = arguments[arguments.length - 1];
|
||||
const deadline = Date.now() + ${WARMUP_TIMEOUT_MS};
|
||||
(function poll() {
|
||||
if (window.mediaPlaybackState === "ready") {
|
||||
if (
|
||||
typeof window.mediaPlaybackBeginMeasure !== "function" ||
|
||||
typeof window.mediaPlaybackMeasurementCount !== "number"
|
||||
) {
|
||||
return notifyDone({
|
||||
error:
|
||||
"the benchmark page does not offer the interface this script drives;" +
|
||||
" it and the pinned revision are out of sync",
|
||||
});
|
||||
}
|
||||
return notifyDone({ ready: true });
|
||||
}
|
||||
if (window.mediaPlaybackState === "error") {
|
||||
return notifyDone({ error: window.mediaPlaybackError });
|
||||
}
|
||||
if (Date.now() > deadline) {
|
||||
return notifyDone({
|
||||
error: "timed out waiting for warmup, state=" + window.mediaPlaybackState,
|
||||
});
|
||||
}
|
||||
setTimeout(poll, ${POLL_INTERVAL_MS});
|
||||
})();
|
||||
`;
|
||||
|
||||
// Opens the page's measured interval and waits for its results.
|
||||
const MEASURE_SCRIPT = `
|
||||
const notifyDone = arguments[arguments.length - 1];
|
||||
const deadline = Date.now() + ${MEASURE_TIMEOUT_MS};
|
||||
const before = window.mediaPlaybackMeasurementCount;
|
||||
try {
|
||||
window.mediaPlaybackBeginMeasure();
|
||||
} catch (e) {
|
||||
return notifyDone({ error: "failed to begin measurement: " + String(e) });
|
||||
}
|
||||
(function poll() {
|
||||
if (window.mediaPlaybackMeasurementCount > before) {
|
||||
return notifyDone(window.mediaPlaybackResult);
|
||||
}
|
||||
if (window.mediaPlaybackState === "error") {
|
||||
return notifyDone({ error: window.mediaPlaybackError });
|
||||
}
|
||||
if (Date.now() > deadline) {
|
||||
return notifyDone({
|
||||
error: "timed out measuring, state=" + window.mediaPlaybackState,
|
||||
});
|
||||
}
|
||||
setTimeout(poll, ${POLL_INTERVAL_MS});
|
||||
})();
|
||||
`;
|
||||
|
||||
module.exports = logTest(
|
||||
"media playback test",
|
||||
async function (context, commands) {
|
||||
context.log.info("Starting media playback test");
|
||||
|
||||
const url = context.options.browsertime.url;
|
||||
const post_startup_delay = context.options.browsertime.post_startup_delay;
|
||||
const page_cycles = context.options.browsertime.page_cycles;
|
||||
const page_cycle_delay = context.options.browsertime.page_cycle_delay;
|
||||
|
||||
await commands.navigate("about:blank");
|
||||
await commands.wait.byTime(post_startup_delay);
|
||||
|
||||
// Load the page once and bring it to steady-state playback.
|
||||
await commands.navigate(url);
|
||||
|
||||
const staged =
|
||||
await context.selenium.driver.executeAsyncScript(WAIT_STAGED_SCRIPT);
|
||||
if (!staged || staged.error) {
|
||||
context.log.error(
|
||||
`media playback setup failed: ${staged && staged.error}`
|
||||
);
|
||||
return false;
|
||||
}
|
||||
if (staged.needsGesture) {
|
||||
await commands.mouse.singleClick.bySelector("#gesture");
|
||||
}
|
||||
|
||||
const ready =
|
||||
await context.selenium.driver.executeAsyncScript(WAIT_READY_SCRIPT);
|
||||
if (!ready || ready.error) {
|
||||
context.log.error(
|
||||
`media playback warmup failed: ${ready && ready.error}`
|
||||
);
|
||||
return false;
|
||||
}
|
||||
|
||||
// Measure: one power interval per cycle, over the same playing video.
|
||||
for (let cycle = 0; cycle < page_cycles; cycle++) {
|
||||
context.log.info(`Starting cycle ${cycle + 1}/${page_cycles}`);
|
||||
|
||||
// stop() closes the power interval, so it has to run even if the script
|
||||
// throws.
|
||||
let result;
|
||||
await commands.measure.start();
|
||||
try {
|
||||
result =
|
||||
await context.selenium.driver.executeAsyncScript(MEASURE_SCRIPT);
|
||||
} finally {
|
||||
await commands.measure.stop();
|
||||
}
|
||||
|
||||
if (!result || result.error) {
|
||||
context.log.error(
|
||||
`media playback test failed: ${result && result.error}`
|
||||
);
|
||||
return false;
|
||||
}
|
||||
context.log.info(`media playback result: ${JSON.stringify(result)}`);
|
||||
|
||||
await commands.measure.addObject({ custom_data: result });
|
||||
|
||||
if (page_cycle_delay) {
|
||||
await commands.wait.byTime(page_cycle_delay);
|
||||
}
|
||||
}
|
||||
|
||||
context.log.info("Media playback test ended");
|
||||
return true;
|
||||
}
|
||||
);
|
||||
@@ -0,0 +1,125 @@
|
||||
# This Source Code Form is subject to the terms of the Mozilla Public
|
||||
# License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
|
||||
import filters
|
||||
from base_python_support import BasePythonSupport
|
||||
from cmdline import FIREFOX_APPS
|
||||
from logger.logger import RaptorLogger
|
||||
|
||||
LOG = RaptorLogger(component="perftest-support-class")
|
||||
|
||||
|
||||
class MediaPlayback(BasePythonSupport):
|
||||
"""Summarizes the media playback benchmark.
|
||||
|
||||
The energy this suite measures comes from add_additional_metrics, which has to
|
||||
be called explicitly from summarize_test.
|
||||
"""
|
||||
|
||||
# name -> (unit, lower_is_better)
|
||||
PAGE_METRICS = {
|
||||
"presentedFps": ("fps", False),
|
||||
}
|
||||
|
||||
def __init__(self, **kwargs):
|
||||
super().__init__(**kwargs)
|
||||
self.subtest_alert_on = []
|
||||
self.measured_a_cycle = False
|
||||
self.run_local = False
|
||||
self.saw_power = False
|
||||
|
||||
def setup_test(self, next_test, args):
|
||||
super().setup_test(next_test, args)
|
||||
# Runs before the manifest turns alert_on into a list, so it is still raw.
|
||||
alert_on = next_test.get("alert_on", "") or ""
|
||||
if isinstance(alert_on, str):
|
||||
alert_on = alert_on.replace("\n", ",").split(",")
|
||||
self.subtest_alert_on = [name.strip() for name in alert_on if name.strip()]
|
||||
self.run_local = getattr(args, "run_local", False)
|
||||
|
||||
def _should_alert(self, name):
|
||||
return name in self.subtest_alert_on and self.app in FIREFOX_APPS
|
||||
|
||||
def handle_result(self, bt_result, raw_result, **kwargs):
|
||||
for extras in raw_result.get("extras", []):
|
||||
data = extras.get("custom_data", {})
|
||||
if data:
|
||||
self.measured_a_cycle = True
|
||||
for name in self.PAGE_METRICS:
|
||||
value = data.get(name)
|
||||
if isinstance(value, (int, float)) and not isinstance(value, bool):
|
||||
bt_result["measurements"].setdefault(name, []).append(value)
|
||||
|
||||
def report_test_success(self):
|
||||
# Either half can go missing without anything raising, leaving a green
|
||||
# suite that measured nothing.
|
||||
if not self.measured_a_cycle:
|
||||
return False
|
||||
if self.platform == "Windows" and not self.run_local and not self.saw_power:
|
||||
LOG.error("no power data was collected; this suite exists to measure it")
|
||||
return False
|
||||
return True
|
||||
|
||||
def _merge_subtest(self, suite, subtest):
|
||||
"""summarize_test runs once per page cycle against a shared suite, so an
|
||||
appended subtest would appear once per cycle with one replicate each.
|
||||
"""
|
||||
name = subtest["name"]
|
||||
subtests = suite["subtests"]
|
||||
existing = None
|
||||
if isinstance(subtests, dict):
|
||||
existing = subtests.get(name)
|
||||
if existing is None:
|
||||
subtests[name] = subtest
|
||||
else:
|
||||
existing = next((s for s in subtests if s["name"] == name), None)
|
||||
if existing is None:
|
||||
subtests.append(subtest)
|
||||
if existing is not None:
|
||||
existing["replicates"].extend(subtest["replicates"])
|
||||
existing["value"] = round(filters.mean(existing["replicates"]), 3)
|
||||
|
||||
def summarize_test(self, test, suite, **kwargs):
|
||||
suite["type"] = "benchmark"
|
||||
if suite["subtests"] == {}:
|
||||
suite["subtests"] = []
|
||||
|
||||
for name, replicates in test["measurements"].items():
|
||||
if not replicates or self.is_additional_metric(name):
|
||||
continue
|
||||
if name not in self.PAGE_METRICS:
|
||||
continue
|
||||
unit, lower_is_better = self.PAGE_METRICS[name]
|
||||
subtest = self._build_standard_subtest(
|
||||
test,
|
||||
replicates,
|
||||
name,
|
||||
unit=unit,
|
||||
should_alert=self._should_alert(name),
|
||||
)
|
||||
# Assigned, not passed: _build_standard_subtest folds a False argument
|
||||
# into the test-level default.
|
||||
subtest["lowerIsBetter"] = lower_is_better
|
||||
self._merge_subtest(suite, subtest)
|
||||
|
||||
# cpuTime's replicates do not correspond to the windows this suite brackets.
|
||||
self.add_additional_metrics(test, suite, exclude=["cpuTime"], **kwargs)
|
||||
self.saw_power = any(
|
||||
s["name"].startswith("powerUsage_") and s.get("replicates")
|
||||
for s in suite["subtests"]
|
||||
)
|
||||
|
||||
# Rebuilt from the whole run each cycle, so the last copy is complete.
|
||||
deduped = {}
|
||||
for subtest in suite["subtests"]:
|
||||
deduped[subtest["name"]] = subtest
|
||||
|
||||
for name, subtest in deduped.items():
|
||||
subtest["shouldAlert"] = self._should_alert(name)
|
||||
# Median, not the base class's mean: a disturbed cycle would move a
|
||||
# mean. Replicates are left whole so the raw values stay inspectable.
|
||||
if subtest.get("replicates"):
|
||||
subtest["value"] = round(filters.median(subtest["replicates"]), 3)
|
||||
|
||||
suite["subtests"] = sorted(deduped.values(), key=lambda s: s["name"])
|
||||
@@ -12,6 +12,17 @@ metrics:
|
||||
- seekedColdLatency
|
||||
description: >
|
||||
Time (in ms) from issuing a buffered currentTime seek to the seeked event, measured with the video decoder warm (kept playing) and cold (released while paused and recreated per seek).
|
||||
"Power Usage":
|
||||
aliases:
|
||||
- powerUsage_cpu_package
|
||||
- powerUsage_cpu_cores
|
||||
- powerUsage_gpu
|
||||
description: >
|
||||
Energy in microwatt-hours consumed over the measured interval, read from the
|
||||
platform's own energy counters. The package channel covers the whole chip and is
|
||||
the steadiest of the three; the core and GPU planes are narrower and noisier.
|
||||
Sampled system-wide by a separate process rather than from inside the browser,
|
||||
which is what makes the measurement independent of which browser is under test.
|
||||
"cpu Time":
|
||||
aliases:
|
||||
- cpuTime
|
||||
@@ -280,6 +291,16 @@ suites:
|
||||
sample-python-support: "A sample test that uses a python support file to modify the test command."
|
||||
connect: "Verifies that connection time on a pageload is zero when a speculative connection has first been made"
|
||||
media-seek: "Measures buffered video seek latency (time from a currentTime seek to the seeked event) with the decoder warm, and also cold on Firefox, where the decoder can be forced to be released on pause."
|
||||
mp-1080p30-av1-hw: "Measures the power used by sustained 1080p30 AV1 playback with hardware decode forced, alongside the frame rate that shows whether playback kept up. This is the cross-browser track."
|
||||
mp-1080p30-av1-sw: "Measures the power used by sustained 1080p30 AV1 playback with software decode forced, alongside the frame rate that shows whether playback kept up. Firefox only, since the forcing preference is not applied to other browsers."
|
||||
mp-1080p30-h264-hw: "Measures the power used by sustained 1080p30 H264 playback with hardware decode forced, alongside the frame rate that shows whether playback kept up. This is the cross-browser track."
|
||||
mp-1080p30-h264-sw: "Measures the power used by sustained 1080p30 H264 playback with software decode forced, alongside the frame rate that shows whether playback kept up. Firefox only, since the forcing preference is not applied to other browsers."
|
||||
mp-1080p30-vp9-hw: "Measures the power used by sustained 1080p30 VP9 playback with hardware decode forced, alongside the frame rate that shows whether playback kept up. This is the cross-browser track."
|
||||
mp-1080p30-vp9-sw: "Measures the power used by sustained 1080p30 VP9 playback with software decode forced, alongside the frame rate that shows whether playback kept up. Firefox only, since the forcing preference is not applied to other browsers."
|
||||
mp-2160p30-av1-hw: "Measures the power used by sustained 2160p30 AV1 playback with hardware decode forced, alongside the frame rate that shows whether playback kept up. Four times the pixels of the 1080p subtests, so playback accounts for more of the energy measured."
|
||||
mp-2160p30-h264-hw: "Measures the power used by sustained 2160p30 H264 playback with hardware decode forced, alongside the frame rate that shows whether playback kept up. Four times the pixels of the 1080p subtests, so playback accounts for more of the energy measured."
|
||||
mp-2160p30-vp9-hw: "Measures the power used by sustained 2160p30 VP9 playback with hardware decode forced, alongside the frame rate that shows whether playback kept up. Four times the pixels of the 1080p subtests, so playback accounts for more of the energy measured."
|
||||
mp-idle: "Measures the power used over the same interval with nothing playing, as the baseline the media playback subtests are read against."
|
||||
prefetch-immediate: "Speculation Rules prefetch test (immediate eagerness): the inline speculationrules script prefetches the target as soon as it is parsed, then the link is clicked so a prefetch-capable browser serves the navigation from the prefetch cache."
|
||||
prefetch-eager: "Speculation Rules prefetch test (eager eagerness): the inline speculationrules script prefetches the target on the slightest hover, then the link is clicked so a prefetch-capable browser serves the navigation from the prefetch cache."
|
||||
prefetch-moderate: "Speculation Rules prefetch test (moderate eagerness): the inline speculationrules script prefetches the target after ~200 ms of sustained hover, then the link is clicked so a prefetch-capable browser serves the navigation from the prefetch cache."
|
||||
|
||||
@@ -61,6 +61,12 @@
|
||||
["include:tests/custom/browsertime-indexeddb.toml"]
|
||||
# Custom indexeddb test
|
||||
|
||||
["include:tests/custom/browsertime-media-playback.toml"]
|
||||
# Custom media playback power test, hardware-decode track
|
||||
|
||||
["include:tests/custom/browsertime-media-playback-sw.toml"]
|
||||
# Custom media playback power test, software-decode track
|
||||
|
||||
["include:tests/custom/browsertime-media-seek.toml"]
|
||||
# Custom media seek-latency test
|
||||
|
||||
|
||||
@@ -0,0 +1,49 @@
|
||||
# Media playback power benchmark, software-decode track. See
|
||||
# browsertime-media-playback.toml for everything not specific to this track.
|
||||
#
|
||||
# Separate from that file because a section's preferences replace the DEFAULT list
|
||||
# rather than extending it. Firefox-only: preferences are dropped for other apps.
|
||||
[DEFAULT]
|
||||
alert_threshold = 5.0
|
||||
apps = "firefox"
|
||||
benchmark_webserver = true
|
||||
browser_cycles = 1
|
||||
custom_data = true
|
||||
gecko_profile_features = "stackwalk,processcpu,power"
|
||||
gecko_profile_interval = 1
|
||||
gecko_profile_threads = "GeckoMain,MediaDecoderStateMachine,MediaPDecoder"
|
||||
lower_is_better = true
|
||||
measure = "powerUsage_cpu_package, powerUsage_cpu_cores, powerUsage_gpu, presentedFps"
|
||||
subtest_lower_is_better = false
|
||||
owner = "Media Team"
|
||||
page_cycles = 15
|
||||
page_timeout = 60000
|
||||
subtest_unit = "uWh"
|
||||
repository = "https://github.com/mozilla/perf-automation"
|
||||
repository_revision = "7046a4bd167a082989802856aaa86b77e84e939a"
|
||||
repository_path = "benchmarks/media-playback"
|
||||
sparse_checkout = true
|
||||
support_class = "media_playback.py"
|
||||
test_script = "media_playback.js"
|
||||
type = "pageload"
|
||||
unit = "uWh"
|
||||
use_live_sites = false
|
||||
preferences = [
|
||||
"media.autoplay.default=0",
|
||||
"media.autoplay.blocking_policy=0",
|
||||
"media.block-autoplay-until-in-foreground=false",
|
||||
"media.suspend-background-video.enabled=false",
|
||||
"media.hardware-video-decoding.enabled=false",
|
||||
]
|
||||
|
||||
["mp-1080p30-av1-sw"]
|
||||
alert_on = "powerUsage_cpu_package"
|
||||
test_url = "http://<host>:<port>/index.html?clip=av1_1080p30&driver=single&raptor=1"
|
||||
|
||||
["mp-1080p30-h264-sw"]
|
||||
alert_on = "powerUsage_cpu_package"
|
||||
test_url = "http://<host>:<port>/index.html?clip=h264_1080p30&driver=single&raptor=1"
|
||||
|
||||
["mp-1080p30-vp9-sw"]
|
||||
alert_on = "powerUsage_cpu_package"
|
||||
test_url = "http://<host>:<port>/index.html?clip=vp9_1080p30&driver=single&raptor=1"
|
||||
@@ -0,0 +1,79 @@
|
||||
# Media playback power benchmark, hardware-decode track.
|
||||
[DEFAULT]
|
||||
# Coarse because the reading includes the energy the browser spends just being
|
||||
# open, which playback is only a fraction of. Set before the suite had any trunk
|
||||
# history, so worth revisiting in either direction once it has some.
|
||||
alert_threshold = 5.0
|
||||
apps = "firefox, chrome"
|
||||
benchmark_webserver = true
|
||||
browser_cycles = 1
|
||||
custom_data = true
|
||||
gecko_profile_features = "stackwalk,processcpu,power"
|
||||
gecko_profile_interval = 1
|
||||
gecko_profile_threads = "GeckoMain,MediaDecoderStateMachine,MediaPDecoder"
|
||||
lower_is_better = true
|
||||
measure = "powerUsage_cpu_package, powerUsage_cpu_cores, powerUsage_gpu, presentedFps"
|
||||
# The one series the page produces is a frame rate, not energy. Power's own
|
||||
# direction is set per-channel by the support class.
|
||||
subtest_lower_is_better = false
|
||||
owner = "Media Team"
|
||||
# Reported as their median, so enough that a couple of disturbed windows cannot
|
||||
# move it.
|
||||
page_cycles = 15
|
||||
page_timeout = 60000
|
||||
subtest_unit = "uWh"
|
||||
# The workload is out-of-tree so other harnesses can drive the same page. Raptor
|
||||
# clones the revision named here and serves only that directory.
|
||||
repository = "https://github.com/mozilla/perf-automation"
|
||||
repository_revision = "7046a4bd167a082989802856aaa86b77e84e939a"
|
||||
repository_path = "benchmarks/media-playback"
|
||||
sparse_checkout = true
|
||||
support_class = "media_playback.py"
|
||||
test_script = "media_playback.js"
|
||||
type = "pageload"
|
||||
unit = "uWh"
|
||||
use_live_sites = false
|
||||
# Replaces the raptor profile's values rather than extending them, and is dropped
|
||||
# entirely for non-Firefox apps. That profile blocks autoplay outright, which
|
||||
# muting does not get past, and background-video suspend would otherwise fire
|
||||
# partway through a window and report the saving as real.
|
||||
preferences = [
|
||||
"media.autoplay.default=0",
|
||||
"media.autoplay.blocking_policy=0",
|
||||
"media.block-autoplay-until-in-foreground=false",
|
||||
"media.suspend-background-video.enabled=false",
|
||||
"media.hardware-video-decoding.force-enabled=true",
|
||||
]
|
||||
|
||||
["mp-1080p30-av1-hw"]
|
||||
alert_on = "powerUsage_cpu_package"
|
||||
test_url = "http://<host>:<port>/index.html?clip=av1_1080p30&driver=single&raptor=1"
|
||||
|
||||
["mp-1080p30-h264-hw"]
|
||||
alert_on = "powerUsage_cpu_package"
|
||||
test_url = "http://<host>:<port>/index.html?clip=h264_1080p30&driver=single&raptor=1"
|
||||
|
||||
["mp-1080p30-vp9-hw"]
|
||||
alert_on = "powerUsage_cpu_package"
|
||||
test_url = "http://<host>:<port>/index.html?clip=vp9_1080p30&driver=single&raptor=1"
|
||||
|
||||
["mp-2160p30-av1-hw"]
|
||||
alert_on = "powerUsage_cpu_package"
|
||||
test_url = "http://<host>:<port>/index.html?clip=av1_2160p30&driver=single&raptor=1"
|
||||
|
||||
["mp-2160p30-h264-hw"]
|
||||
alert_on = "powerUsage_cpu_package"
|
||||
test_url = "http://<host>:<port>/index.html?clip=h264_2160p30&driver=single&raptor=1"
|
||||
|
||||
["mp-2160p30-vp9-hw"]
|
||||
alert_on = "powerUsage_cpu_package"
|
||||
test_url = "http://<host>:<port>/index.html?clip=vp9_2160p30&driver=single&raptor=1"
|
||||
|
||||
["mp-idle"]
|
||||
test_url = "http://<host>:<port>/index.html?driver=idle&raptor=1"
|
||||
# No alert_on, and none intended: it is a baseline, and the noisiest series here
|
||||
# since it has no signal of its own to swamp the platform's variation. Nothing
|
||||
# plays, so no frame rate and no media thread.
|
||||
measure = "powerUsage_cpu_package, powerUsage_cpu_cores, powerUsage_gpu"
|
||||
subtest_lower_is_better = true
|
||||
gecko_profile_threads = "GeckoMain"
|
||||
Reference in New Issue
Block a user