Skip to content

CI: Playwright failures are undiagnosable — trace.zip is not uploaded and the stack-log tail is too short #487

Closed

Description

@mforce

A Playwright failure on CI currently cannot be diagnosed from its own artifacts. Hit while investigating #486, where the decisive evidence — did PUT /me/language reach the server before the reload? — was unrecoverable, so the root cause had to be inferred from source and timing rather than observed.

Two independent gaps, both in .github/workflows/e2e-smoke.yml.

1. The trace is generated but never uploaded

.github/workflows/e2e-smoke.yml:212-214:

path: |
  tools/simulation/ui/test-results/**/*.png
  tools/simulation/ui/test-results/**/*.webm

Playwright wrote trace.zip and error-context.md for the failing test — the reporter names both in the log — but the glob takes only .png and .webm. The trace is the one artifact carrying the network log, which is what answers "was the request issued, and what did it return". Without it, a race between an un-awaited write and a navigation is unfalsifiable after the fact.

Fix: add **/trace.zip and **/*.md to the glob. Retention already 7 days, if-no-files-found: ignore already set, so nothing else changes. Watch the size — the trace for one failing test in #486's run would have been well within limits, but a run failing many tests uploads one per test.

2. The container log dump is truncated below the useful window

.github/workflows/e2e-smoke.yml:221-223:

docker compose -p cluckwork-sim … logs --tail 200 || true

200 lines is shared across all services, and the app emits one JSON line per HTTP request. Measured on #486's run: the dump covered 01:01:00-01:01:19 — the last 19 seconds — while the test that failed ran at roughly 01:00:21-01:00:33. The relevant window had already scrolled off.

Fix: raise the tail substantially (or drop --tail and let the step capture the run; the sim run is ~3 minutes, so the volume is bounded). Worth also dumping per-service so app logs are not interleaved out of reach.

Why this is worth doing on its own

Neither gap is visible until something fails, and by then the evidence is already gone — the failure has to recur before it can be diagnosed. Both fixes are a few lines, and the payoff lands on the next flake rather than this one.

Note the asymmetry that makes it worse: the log dump is if: failure(), so a passing run keeps nothing at all. Comparing a failing run against a passing one — the exact move that isolated #486 as timing rather than code — is only possible on the reporter's duration output today.

Related: #486 is the flake that surfaced this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions