Repository navigation
Conversation
- Install the benchmark telemetry sink with renderer collectors (JS heap, FPS, late frames, long tasks, UA-specific memory) and a self-cost meter - Expose window.lichtblickTelemetry (isEnabled, now, drain); ?telemetry=off keeps the API but disables recording - Describe every scenario (group, tags, layout, data source, player, redacted params, panels) and accept ?ds.<param> overrides - Add window.lichtblickScenario.closeDataSource() and self-pausing scenarios with durationMs - Document scenarios and the telemetry contract in the benchmark README Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add benchmark-result.v1 types, builder and strict/lenient validator - Summarize process, renderer, phases, panels, pipeline and HydratedSourcePool series per run - Derive per-run KPIs and per-platform/per-group aggregates (geomean or worst case) - Register the benchmark-e2e Jest project and its tsconfig Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Drive each scenario with runScenario: 1 Hz host polling and telemetry drain on the page clock, FPS calibration, checkpoint screenshots - Collect Electron process memory and CPU (per-core sum) and fail runs on crashes, reloads or error boundaries - Write the v1 result next to the legacy artifact and extend the markdown CI summary with v1, panels and pipeline tables - Run the Electron project in the e2e-benchmark workflow Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add yarn test:e2e:benchmark:viewer: a self-contained HTML report with per-run charts, phases and screenshots - Show a KPI scorecard per platform and an exercises card / coverage matrix per scenario - Add the scorecard and exercises lines to the CI summary and upload the report in CI - Add chart.js as a benchmark dev dependency Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add the web Playwright project: headless Chromium against a static server with COOP/COEP (cross-origin isolated) - Collect browser process CPU via CDP and RSS via ps for web runs - Add the web scripts and the e2e-benchmark-web CI job Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Generate deterministic MCAP fixtures with @mcap/core and serve them from the benchmark server - Add the mcap-multi-ingest scenario: time-overlapping MCAP files through the real worker pipeline with one hydrated source at a time - Assert the pool invariant, over-budget periods and open/first-message events in the scenarios spec Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Open
4 tasks done
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. 🗂️ Base branches to auto review: 2
Please check the settings in the CodeRabbit UI or the
 |
This was referenced Nov 4, 2026
- PanelExtensionAdapter: do not count a tick without messages as a skipped render while a render is pending. - Document that range-delivered bytes are approximate when data is merged into the middle of allFrames (count stays exact). - IterablePlayer: end the preload phase with reason "cancelled" when the last full-preload subscription is removed; emit the datasource close start/end pair at most once per player. - HydratedSourcePool: settle the over-budget state right after victims are evicted instead of after their close() settles. - WorkerSerializedIterableSource: bound the final telemetry drain with one shared deadline so the worker is always disposed; stop the bridge timer when enabling worker telemetry fails. - WorkerSerializedIterableSourceWorker: drop the redundant publish in terminate(); the final drain publishes on dispose. - CurrentLayoutProvider: emit layout.applied only when the selection was accepted (not for unsupported working layout versions). - BufferedTelemetrySink: reject counter totals that overflow to Infinity (record and ingest paths); keep the sink's self-metric series separate from caller series and reject caller metrics using the reserved self-metric names. - Tests for each change; assert initialize() is called once in the stale-init worker source test. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
User-Facing Changes
None (benchmark tooling only). The product app is not changed.
Description
Split of #1393 into three stacked PRs that can be reviewed separately.
📚 Review sequence (part 2 of 3)
feat(telemetry): memory and pipeline telemetry contract and product instrumentationdevelopfeat(benchmark): telemetry harness, result schema v1, viewer and web runnerfeature/benchmark-telemetry-product([1/3])feat(benchmark): telemetry overhead A/B and AI benchmark analystfeature/benchmark-telemetry-harness([2/3])Review and merge in order 1 → 2 → 3. Each PR passes type checks, lint and unit tests on its own. This PR targets the [1/3] branch, so the diff shows only its own changes. It will be retargeted to
developafter [1/3] merges. #1393 stays open as a draft umbrella for reference. Its final tree equals the tip of [3/3].🎯 Main gain within the overall plan
Overall plan (#1393):
benchmark-result.v1. Electron and web runners produce them, and an offline HTML viewer and a CI job summary read them. Overhead target for telemetry: ≤ 2 % CPU (relative) and ≤ 20 MiB, checked by an A/B.What this PR adds: the measurement harness around the contract from [1/3]:
benchmark-result.v1file per runWhat it unblocks: comparable, versioned results that are stored as CI artifacts on every push to
develop/main. [3/3] builds the overhead A/B on this harness. Baselines, gates and a dashboard can later use the same schema.🧩 What was done (review commit by commit)
Total: 166 files, +21039 / −205. Source (non-test
.ts/.tsx): 107 files, +11338 / −187. Tests and builders: 48 files, +8931. Other files (md, json, yml, mjs, lock, gitignore): 11 files, +770 / −18. There are nopackages/**changes.a7a88d907src/telemetry/installBenchmarkTelemetry.tscallssetTelemetrySink; this is the only activation point.Renderer collectors in
src/telemetry/collectors/:•
performance.memory• long tasks
• rAF FPS (
frameProbe/FrameLoop.ts)• UA-specific memory when
crossOriginIsolated• in-page self-cost
Also: scenario descriptors (
describe(),scenarios/registry.ts) andBenchmarkStats.listenerDispatchDurationMs.0a787a7fae2e/result/schemaV1.ts(+types): validator forbenchmark-result.v1.Summaries: panels, pipeline, phases (including
layoutApplied/panelsReady) and memory slope.kpis.tsbuilds the per-run KPIs and the platform scorecard (geomean / worst).Pure functions; review the tests next to them.
e71777f42runner/runScenario.tsdrives a run. It writes the v1 result and the legacy artifact.electronHostCollector.ts:app.getAppMetrics()at 1 Hz.crashDetection.ts: a crash, OOM or reload fails the scenario.calibration.ts: FPS ceiling and GL renderer.Also: screenshots and the error-boundary guard.
.github/workflows/e2e-benchmark.yml(Electron job) and the job summary script.6f4729c2de2e/report/htmlReport*.ts: a self-contained offline HTML page with Chart.js inlined.It shows a KPI scorecard per metric family (geomean / worst per platform), an "Exercises" card per run and a coverage matrix.
CI uploads the report in the artifacts. Adds
chart.jsto the benchmark workspace.cf263203dserver/staticServer.tsserves the build with COOP/COEP, so the page iscrossOriginIsolatedand UA-specific memory is available.runner/webHostCollector.tsreads per-process metrics through CDPSystemInfo, with apsfallback.Adds the CI web job.
7b5651a1dfixtures/generateMcapFixtures.tsgenerates the multi-file MCAP fixtures at runtime; they are not committed.The scenario
mcap-multi-ingestruns the real pipeline withmaxHydratedSources=1, so pool evictions and over-budget periods happen and the invariant is checked.Adds
@mcap/coreto the benchmark workspace.Key design notes
sinewave,pointcloud,transform,transformpreloadingandmcap-multi-ingest.chart.jsand@mcap/coreare added to thebenchmarkworkspace only. Both packages were already in the lockfile throughsuite-base/suite-desktop.e2e-benchmark.ymlruns the scenarios on Electron and web on push todevelop/mainonly, because the suite is slow. It uploads the JSON results, screenshots and the HTML report as artifacts and writes a job summary.run.modeonly accepts"scenario".🧪 How to test
Expected results:
1 passed.*.benchmark-result.v1.jsonfile is written tobenchmark/.artifacts/.crossOriginIsolated: true.yarn test benchmarkgives 46 suites / 489 tests.✅ Validation evidence
The E2E benchmark workflow only runs on push to
develop/main, so this evidence comes from local runs. Setup: macOS, Apple M4 Pro, ANGLE Metal, 2026-11-04, prod builds made from the tip7b5651a1d.yarn benchmark:build:prodyarn benchmark:desktop:build:prodyarn format:ciyarn tsc --noEmit(root,-p benchmark/e2e,-p benchmark/src)yarn lint:unused-exportsyarn test benchmarkyarn test:e2e:benchmark:vieweryarn test:e2e:benchmark:summary(on a clean.artifacts/)git diff --stat <[1/3]> HEADpackages/**filesReal runs (each wrote a
*.benchmark-result.v1.json)scenarios -g sinewavescenarios -g mcap-multi-ingestscenarios -g sinewavescenarios -g mcap-multi-ingestscenarios -g cross-originrun.mode: "scenario", 0 dropped records and 0 runner events..artifacts/folder from Add memory telemetry per process, pipeline and panel to benchmarks #1393 needs cleaning; see the caveat above.panel.ready, deeper panel and pipeline instrumentation, network, CI gates and baselines, and a historical dashboard.Checklist
Benchmark web runner (headless Chromium, cross-origin isolated): sinewave, mcap-multi-ingest and cross-origin specs passed.
Benchmark Electron runner: sinewave and mcap-multi-ingest passed.
46 suites / 489 tests in
benchmark/.Per-module
constants.ts,types.ts/*.types.tsandhtmlReport.style.ts.