Skip to content

[agent-cli] Stabilize setup and reload E2E tests - #251

Open
vonovak wants to merge 1 commit into
mainfrom
codex/fix-windows-e2e
Open

vonovak wants to merge 1 commit into
mainfrom
codex/fix-windows-e2e

Conversation

@vonovak

@vonovak vonovak commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fix two independent test-isolation issues exposed by the same CI run. Both affect code paths used on all platforms.

 interactive setup test
+  disable real docs downloads
 reload verification test
+  use the known stub URL and show CLI output on unexpected exit codes

Bug fix

Interactive setup: real docs downloads exceed the test deadline

The interactive helper bypasses the shared launcher to send keyboard input. It missed the launcher's AGENT_CLI_NO_DOCS_SYNC default, so the agent-selection test downloaded real documentation after installing its stub plugin. In the failing run, the child had not exited within the helper's 15-second deadline.

This is an intermittent issue across platforms. The same test passed in 1.02 seconds on Linux and in 3 seconds on Windows in the previous run. These timings do not isolate download speed from extraction or other runner work.

Reproduce the setup issue

Keep the added skipped-docs assertion, but remove the fixture's AGENT_CLI_NO_DOCS_SYNC flag. From packages/@expo/agent-cli, run:

AGENT_CLI_DOCS_URL=http://127.0.0.1:1 bun run test:e2e -- agents-install-test -t 'toggle preselected'

The attempted download makes the docs status failed instead of skipped. Restoring the flag makes this regression check pass. This demonstrates the unintended network dependency without relying on a slow external download.

Reload verification: discovery can time out before the reload runs

The reload evidence table starts a fake dev server and knows its URL, but asks the CLI to discover it through a lock. Discovery's 800 ms probe deadline can expire when the stub responds slowly, returning NO_DEV_SERVER instead of the reload report. The original assertion then fails with undefined !== false.

This failure mode was reproduced locally. The original CI run omitted the error payload, so its exact cause remains inferred.

Reproduce the reload issue

With the original lock-based test, temporarily delay the stub's first targets response by 900 ms and run the reloadTargets stale case. Discovery returns NO_DEV_SERVER. Passing --dev-server-url with the same delayed stub lets the test complete. The temporary delay is excluded from this patch.

How

  • Setup: apply the shared launcher's docs-sync opt-out to the interactive fixture and assert that syncing was skipped. Separate tests still cover docs syncing.
  • Reload: pass the stub URL directly, preserving the reconnect/stale/gone assertions. Check the expected exit code with captured stdout/stderr so unexpected CLI errors are visible. Separate tests still cover lock-based discovery.

Alternatives considered

Increasing the test timeout would leave setup's real docs downloads in place and would not change reload discovery's separate 800 ms deadline.

Verification

Evidence

  • Setup: the skipped-docs regression check failed without the flag and passed with it. The fixed test passed in 724 ms in the Windows verification run.
  • Reload: the 900 ms delayed-stub reproduction failed through discovery and passed with the explicit URL. The fixed stale-target case passed in 2,157 ms in the Windows verification run.

The Windows verification run finished with 400 passed, 2 failed and 7 skipped. Both original failing tests passed. The two other failures were in unchanged cloud-binding-test.ts: iOS tunnel reconnection and a 300-second guarded-argv timeout. Those failures are outside this patch.

Local checks on macOS with Node 24.22.1:

  • bun run test:e2e -- agents-install-test runtime-reload-test docs-test: 68 passed.
  • bun run test -- src/runtime/__tests__/devServerLockDiscovery-test.ts src/runtime/__tests__/devServerDiscovery-test.ts: 35 passed.
  • Package build, typecheck and lint; formatting of both changed files; git diff --check: passed.

Merge Danger

Door: two-way

Blast Radius: tests

The patch changes two E2E test files and is straightforward to revert.

@vonovak vonovak changed the title [agent-cli] Isolate setup and reload e2e tests from incidental I/O [agent-cli] Stabilize setup and reload E2E tests on Windows Oct 30, 2026
@vonovak vonovak changed the title [agent-cli] Stabilize setup and reload E2E tests on Windows [agent-cli] Stabilize setup and reload E2E tests Oct 30, 2026
@vonovak
vonovak requested a review from Kudo October 30, 2026 18:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant