sousa-gecko/testing/workers/JSON_FORMAT.md

Worker Pool Activity Data Format Documentation

This document describes the JSON file formats created by fetch-worker-data.js.

Overview

The script generates daily JSON files containing Taskcluster worker pool activity data, fetched from STMO (BigQuery). The format uses string tables, parallel arrays, and differential time compression to minimize file size.

Three file types are produced per day:

1. Index: index.json - List of available dates, used by the dashboard at https://tests.firefox.dev/workers.html to know which dates can be loaded.
2. Summary: workers-YYYY-MM-DD.json - Small file (~5 MB) with timestamps, resolution, task queue, and project. Loaded immediately by the dashboard for stats computation, time series rendering, and project filtering.
3. Tasks: workers-YYYY-MM-DD-tasks.json - Full task data (~20 MB) with all fields. Loaded on demand when the user clicks a Profiler button.

---

Index File (index.json)

{
  "dates": ["2026-03-04", "2026-03-03", "2026-03-02", ...]
}

Dates are sorted in descending order (newest first). Up to 21 days of history are maintained.

---

Summary File (workers-YYYY-MM-DD.json)

Small file for fast initial dashboard load. Contains only the fields needed to compute per-pool stats, draw time-series activity tracks, and populate project filter buttons.

Top-Level Structure

{
  "metadata": { ... },
  "tables": { ... },
  "tasks": { ... }
}

metadata

{
  "date": "2026-03-04",
  "generatedAt": "2026-03-05T03:12:45.123Z",
  "taskCount": 243684
}

tables

{
  "taskQueueIds": ["gecko-t/t-linux-docker-noscratch-amd", ...],
  "resolutions": ["completed", "failed", "exception - canceled", ...],
  "projects": ["mozilla-central", "try", "autoland", ...]
}

tasks

Parallel arrays of length taskCount:

{
  "scheduled": [1709510400000, 150, 0, 23, ...],
  "started": [5432, null, 1234, ...],
  "resolved": [65432, 3600000, 12345, ...],
  "resolutionIds": [0, 0, 2, ...],
  "taskQueueIdIds": [0, 1, 0, ...],
  "projectIds": [0, null, 1, ...]
}

See Time Compression and Resolutions below for encoding details.

---

Tasks File (workers-YYYY-MM-DD-tasks.json)

Full task data loaded on demand for Firefox Profiler integration.

Top-Level Structure

{
  "metadata": { ... },
  "tables": { ... },
  "tasks": { ... },
  "workerInfo": { ... },
  "taskGroupInfo": { ... }
}

metadata

Same structure as the summary file.

tables

All strings are deduplicated and stored once. Tables are sorted by frequency of use (most referenced entries first) to reduce JSON size (frequently used strings get smaller numeric indices).

{
  "labels": ["test-linux2404-64/opt-mochitest-plain-5", ...],
  "projects": ["mozilla-central", "try", "autoland", ...],
  "taskQueueIds": ["releng-hardware/gecko-t-linux-2404-wayland", ...],
  "resolutions": ["completed", "failed", "exception - canceled", ...],
  "workerGroups": ["us-central1-b", ...],
  "workerIds": ["7732042218797547089", ...],
  "priorities": ["low", "lowest", "medium", ...],
  "users": ["cron@mozilla-central", "user@example.com", ...],
  "taskGroupIds": ["YJJe4a0CRIqbAmcCo8n63w", ...]
}

tasks

Parallel arrays of length taskCount:

{
  "scheduled": [1709510400000, 150, 0, 23, ...],
  "started": [5432, null, 1234, ...],
  "resolved": [65432, 3600000, 12345, ...],
  "resolutionIds": [0, 0, 2, ...],
  "taskIds": ["YJJe4a0CRIqbAmcCo8n63w", "XPPf5b1DRJrcBndDp9o74x.1", ...],
  "labelIds": [0, 1, 2, ...],
  "priorityIds": [0, 0, 1, ...],
  "taskGroupIdIds": [0, 0, 1, ...],
  "userIds": [0, 1, 0, ...],
  "taskQueueIdIds": [0, 1, 0, ...],
  "workerIdIds": [0, null, 1, ...],
  "runCosts": [0.012244, 0, 0.003456, ...]
}

Task IDs

Task IDs include a .runId suffix only when the run ID is greater than 0:

• Run 0: "YJJe4a0CRIqbAmcCo8n63w"
• Run 1: "YJJe4a0CRIqbAmcCo8n63w.1"

Index References

• resolutionIds[i]: Index into tables.resolutions
• labelIds[i]: Index into tables.labels
• priorityIds[i]: Index into tables.priorities
• taskGroupIdIds[i]: Index into tables.taskGroupIds
• userIds[i]: Index into tables.users
• taskQueueIdIds[i]: Index into tables.taskQueueIds
• workerIdIds[i]: Index into tables.workerIds (null if never assigned)

Run Costs

runCosts[i] is the cost in USD, rounded to 6 decimal places. 0 when not available.

workerInfo

Per-worker arrays indexed by worker ID (index into tables.workerIds).

{
  "workerGroupIds": [0, 1, 0, ...]
}

• workerGroupIds[w]: Index into tables.workerGroups for worker w

Each worker belongs to exactly one worker group.

taskGroupInfo

Per-task-group arrays indexed by task group ID (index into tables.taskGroupIds).

{
  "projectIds": [0, 1, null, ...]
}

• projectIds[g]: Index into tables.projects for task group g (null for task groups with no project, typically decision tasks)

Each task group belongs to at most one project.

---

Shared Encoding Details

Time Compression

Both files use the same time encoding. Tasks are sorted by scheduled time.

• scheduled: Differential compression. scheduled[0] is an absolute millisecond timestamp. scheduled[i] (for i > 0) is the delta from scheduled[i-1]. Since tasks are sorted, deltas are small non-negative integers.
• started: Offset from the task's own scheduled time (i.e., queue wait time in ms). null if the task never started.
• resolved: Offset from the task's own scheduled time (i.e., total task lifetime in ms).

To reconstruct absolute timestamps:

const abs = new Array(n);
abs[0] = data.tasks.scheduled[0];
for (let i = 1; i < n; i++) {
  abs[i] = abs[i - 1] + data.tasks.scheduled[i];
}
// abs[i] is the absolute scheduled time for task i

const startedMs = data.tasks.started[i] !== null
  ? abs[i] + data.tasks.started[i] : null;
const resolvedMs = abs[i] + data.tasks.resolved[i];

Resolutions

The resolutions table merges the task state and reason_resolved fields:

• When both are the same: just the value (e.g., "completed", "failed")
• When different: "state - reason" (e.g., "exception - canceled", "exception - deadline-exceeded")

---

Example: Reconstruct a Full Task Row

Using the tasks file:

const d = data;
const i = 42;

// Reconstruct absolute scheduled time
let scheduledMs = d.tasks.scheduled[0];
for (let j = 1; j <= i; j++) {
  scheduledMs += d.tasks.scheduled[j];
}
const startedMs = d.tasks.started[i] !== null
  ? scheduledMs + d.tasks.started[i] : null;
const resolvedMs = scheduledMs + d.tasks.resolved[i];

const taskId = d.tasks.taskIds[i];
const label = d.tables.labels[d.tasks.labelIds[i]];
const resolution = d.tables.resolutions[d.tasks.resolutionIds[i]];
const priority = d.tables.priorities[d.tasks.priorityIds[i]];
const taskQueue = d.tables.taskQueueIds[d.tasks.taskQueueIdIds[i]];
const cost = d.tasks.runCosts[i];

// Task group and project
const tgIdx = d.tasks.taskGroupIdIds[i];
const taskGroupId = tgIdx !== null ? d.tables.taskGroupIds[tgIdx] : null;
const projectIdx = tgIdx !== null ? d.taskGroupInfo.projectIds[tgIdx] : null;
const project = projectIdx !== null ? d.tables.projects[projectIdx] : null;

// User
const userIdx = d.tasks.userIds[i];
const user = userIdx !== null ? d.tables.users[userIdx] : null;

// Worker (may be null if task never ran)
const wIdx = d.tasks.workerIdIds[i];
const workerId = wIdx !== null ? d.tables.workerIds[wIdx] : null;
const workerGroup = wIdx !== null
  ? d.tables.workerGroups[d.workerInfo.workerGroupIds[wIdx]] : null;

---

Data Characteristics

• Task count: ~240K tasks per day
• Summary file size: ~5 MB uncompressed per day
• Tasks file size: ~20 MB uncompressed per day
• String tables: Sorted by frequency (most used entries have lowest indices)
• History: 21 days maintained via CI artifact re-upload
• Source: STMO query 112377 (BigQuery)
• Schedule: Generated daily at 03:00 UTC via CI cron