Skip to content

Commit e1f4f25

Browse files
luluizCopilot
andcommitted
feat(benchmark): KPI scorecard, scenario exercises and reliable telemetry overhead A/B
- Scenario descriptors (description, tags, group, player) and exercised components in results, required run.scenario and summary.kpis, KPI scorecard/group/coverage views in the HTML report and CI summary, FPS ceiling calibration and GL renderer info - Electron host CPU converted to per-core-sum percent (was divided by logical CPU count) - Overhead A/B rework: in-process toggle with ABBA quartets on cumulative CPU time, direct (JS only) CPU attribution via CDP profiler + source maps, relaunch A/B with interleaved repeats for memory, median/IQR/bootstrap CI95 with within/inconclusive/exceeds verdicts, steady windows decoupled from page round trips, UA-memory and FPS-probe variants, report schema 3 - Benchmark frame loop runs in both arms (harness), its cost reported separately - CI runs the scenarios spec only; overhead A/B is run manually - README: specs comparison, overhead methodology, KPI and exercises docs Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent d73b87d commit e1f4f25

162 files changed

Lines changed: 21443 additions & 988 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/e2e-benchmark.yml‎

Lines changed: 9 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -1,25 +1,15 @@
11
name: E2E Benchmark
22

3-
# Runs the benchmark E2E suite (benchmark/e2e), which drives each scenario under test
4-
# (benchmark/e2e/runner/scenariosUnderTest.ts) on two platforms and collects host process metrics,
5-
# in-app telemetry (memory, pipeline/pool counters, phase events) and frame-time samples:
6-
# - e2e-benchmark: the standalone benchmark Electron shell (Playwright project "electron")
7-
# - e2e-benchmark-web: the benchmark web build in headless Chromium (Playwright project "web")
8-
# The real-pipeline scenario (mcap-multi-ingest) uses MCAP fixtures generated deterministically at
9-
# test time into benchmark/.artifacts/fixtures (no binaries are committed). Result JSONs
10-
# (legacy, benchmark-result.v1 and the report-only telemetry overhead A/B) and the scenario
11-
# checkpoint screenshots (benchmark/.artifacts/screenshots) are uploaded as artifacts, together
12-
# with the interim HTML viewer (benchmark/.artifacts/report/index.html).
13-
# Runs on pushes to the long-lived branches - not on a schedule or every PR since it's a
14-
# long-running, resource-sensitive suite whose results are only meaningful in a stable environment.
3+
# Runs the benchmark E2E suite (benchmark/e2e) on Electron and web, collecting process metrics,
4+
# in-app telemetry and frame times. Results, screenshots and the HTML report are uploaded as
5+
# artifacts. Runs only on pushes to long-lived branches: the suite is slow and needs a stable runner.
6+
# Runs the scenarios spec only; the telemetry overhead A/B (overhead spec) is long-running and is
7+
# run manually (see benchmark/README.md).
158

169
on:
1710
push:
1811
branches: [develop, main]
1912

20-
# Least privilege: both jobs only check out the repo. actions/cache and actions/upload-artifact
21-
# authenticate with the runtime token (not GITHUB_TOKEN) and $GITHUB_STEP_SUMMARY is a runner file,
22-
# so neither needs an extra permission.
2313
permissions:
2414
contents: read
2515

@@ -67,18 +57,13 @@ jobs:
6757
uses: coactions/setup-xvfb@v1
6858
with:
6959
working-directory: ./
70-
# The script sets CI=true and `--reporter=list,json`, and writes the json report to
71-
# benchmark/e2e/reports/results-electron.json via PLAYWRIGHT_JSON_OUTPUT_NAME (the json
72-
# reporter would otherwise print to stdout).
73-
run: yarn test:e2e:benchmark:ci
60+
run: yarn test:e2e:benchmark:ci scenarios
7461

7562
- name: Generate Benchmark Test Summary
7663
if: always()
7764
run: yarn test:e2e:benchmark:summary >> "$GITHUB_STEP_SUMMARY"
7865

7966
- name: Generate Benchmark HTML Report
80-
# Interim offline viewer (benchmark/.artifacts/report/index.html, Chart.js inlined) that
81-
# links the screenshots by relative path (../screenshots/), so both are uploaded below.
8267
if: always()
8368
run: |
8469
yarn test:e2e:benchmark:viewer
@@ -90,8 +75,7 @@ jobs:
9075
uses: actions/upload-artifact@v7
9176
with:
9277
name: benchmark-results-electron
93-
# Result JSONs, the checkpoint screenshots (end only by default) and the HTML report -
94-
# the generated MCAP fixtures are reproducible and not worth uploading.
78+
# Generated MCAP fixtures are reproducible, so they are not uploaded.
9579
path: |
9680
benchmark/.artifacts/**/*.json
9781
benchmark/.artifacts/screenshots/**
@@ -128,25 +112,19 @@ jobs:
128112
- run: yarn install --immutable
129113

130114
- name: Build benchmark web app
131-
# Static SPA in benchmark/.webpack, served by the suite's own static server
132-
# (benchmark/e2e/server/staticServer.ts) with COOP/COEP headers for cross-origin isolation.
133115
run: yarn benchmark:build:prod
134116

135117
- name: Install Playwright Chromium
136118
run: yarn playwright install --with-deps chromium
137119

138120
- name: Run Benchmark E2E tests (web)
139-
# Headless Chromium needs no display server. The script sets CI=true and writes the json
140-
# report to benchmark/e2e/reports/results-web.json (distinct from the Electron job's file).
141-
run: yarn test:e2e:benchmark:web:ci
121+
run: yarn test:e2e:benchmark:web:ci scenarios
142122

143123
- name: Generate Benchmark Test Summary
144124
if: always()
145125
run: yarn test:e2e:benchmark:summary >> "$GITHUB_STEP_SUMMARY"
146126

147127
- name: Generate Benchmark HTML Report
148-
# Interim offline viewer (benchmark/.artifacts/report/index.html, Chart.js inlined) that
149-
# links the screenshots by relative path (../screenshots/), so both are uploaded below.
150128
if: always()
151129
run: |
152130
yarn test:e2e:benchmark:viewer
@@ -158,8 +136,7 @@ jobs:
158136
uses: actions/upload-artifact@v7
159137
with:
160138
name: benchmark-results-web
161-
# Result JSONs, the checkpoint screenshots (end only by default) and the HTML report -
162-
# the generated MCAP fixtures are reproducible and not worth uploading.
139+
# Generated MCAP fixtures are reproducible, so they are not uploaded.
163140
path: |
164141
benchmark/.artifacts/**/*.json
165142
benchmark/.artifacts/screenshots/**

0 commit comments

Comments
 (0)