Repository navigation
The local pre-push gate refuses what CI refuses #319
Description
Part of #194.
Part of #311 (Group D, CI consistency). Owner: Codex prepares and verifies the candidate; Anders accepts integration. Status: in progress (failure investigation below). Task base and PR target: develop. Independent of other sub-issues.
Intended behavior
just ci-local-prepush refuses every warning and lint error that CI refuses for the crates a change affects, within a bounded time.
Current behavior
- CI sets
RUSTFLAGS: "-D warnings"for every job (ci.yml:28) and runscargo clippy --workspace --all-targets -- -D warnings(ci.yml:206). - The local gate (
scripts/ci-local-prepush.py:509-518) runscargo check --workspace --all-targetswithout-D warnings. It runs clippy only for the touched packages (-p <touched>). - A change that adds a warning in an untouched crate that depends on the change, for example an unused import after an API change, passes locally and fails in CI.
- Release Python self-tests run in CI (
ci.ymlsource-gate, "release artifact source proofs"). Locally onlycheck-product-dataruns.
Done means
- The local gate sets
RUSTFLAGS=-D warningsfor its cargo calls, as CI does. Test: a fixture inscripts/ci-local-prepush-test.py. - Clippy covers the touched packages and their reverse dependencies in the same workspace. Test: a fixture where a warning appears only in a dependent crate fails the gate.
- When
scripts/release-*,install.shorpublish-release.shchange, the gate runs the release self-tests (release-platform-input-test.py,ci-release-policy-test.py). Test: a fixture.
Acceptance record
PR#326 is ready for review at6d6330e4a414dcb0dbe5b35b828fb6f94e05468d, tree4923852246b10d6f59c6d54fd4df917a9ae941d2. Approved target: develop; candidate base7184e0444079881e62ef99a6c3807547119a4534.
Local proof passed65 gate fixtures,102 product-data cases, required formatting and diff checks. The changed CI helpers select zero Rust consumers and therefore zero compilation. Independent development review passed the published diff. Different-model review also passed the exact candidate. Wider policy-input coverage remains in#317.
Hosted run37973992785 passed on the first attempt: all11 executed jobs and Linux x86-64, Linux ARM64 and macOS installed journeys. The release job was skipped for the PR. Elapsed time was71m01s; macOS took65m23s. Trusted cache-write/warm-read and budget measurements remain in#311/#310.
Develop later advanced to37e58ea1235e35a4c7f4cd310ae164d119997877 through#327. Its17 changed paths and this patch's3 paths are disjoint. Git calculated clean combined treef12334311107922fa8ab85e89b1a33364ea179ba. The successful hosted run tested the earlier7184 base; current combined installed acceptance is still an integration check. The reviewed candidate remains unchanged.
Human review is requested from irzhywau on this ready PR. Remaining gates: required human review, current-base acceptance and approved integration into develop. Keep this issue open through merge. Anders accepts integration; the next CI issue follows its dependencies and the one-active-task rule.
Failure investigation (10 Oct): every late failure, its cause and its guard
Principle: a CI result is a function of (tree, commands, environment). Every difference between local and CI in any of the three is a hole. Each failure below found one. Generated from the failure ledger (elastos-lesson render-issue).
ledger: 66 incidents, 25 open automated guards, rendered 2026-10-11T16:31Z
| # | Date | Where | Failure | Root cause | Class | Guard | Guard status |
|---|---|---|---|---|---|---|---|
| 1 | 2026-10-10 | remote | release-platform-input-test: missing release-platforms.json (remote, 25 min in) |
a new file read by a script; three test fixtures copy a fixed file list; the gate ran release tests only for some paths | commands-gap | shared fast list runs on every push (done); fixtures copy what the script imports (2) | open |
| 2 | 2026-10-10 | local | prepare-release-platform-test 16/25 (local, after merge) | security fix made a script import the signer; fixture list not updated; the agent ran a different test | commands-gap | same as 1 | open |
| 3 | 2026-10-10 | remote | release_cmd tests (remote) | message changed in a library file; the tests asserting it live in the binary target; the gate runs only the target that owns the changed file | commands-gap | gate runs every test target of a touched package (3) | open |
| 4 | 2026-10-10 | remote | setup profile test (remote) | the same rule lived in two places (Rust test and product-data check); the merge updated one | duplicate-truth | duplicate removed (done) | open |
| 5 | 2026-10-10 | ci | Chat CI: "pending delivery copy is inaccurate" (CI) | just product-ui-browser smokes run only in CI |
commands-gap | add to the shared lists if fast, else Linux leg (4) | open |
| 6 | 2026-10-10 | ci | Browser CI: two image tests fail on Linux only (CI) | Mac gate passed; code differs per OS | host-dependent | Linux leg of the gate before every push (5) | open |
| 7 | 2026-10-10 | remote | 50 behaviour failures (remote) | remote box lacked zip, jq that GitHub runners have |
box-tools | tool preflight from one tool list (6) | open |
| 8 | 2026-10-10 | remote | three remote runs hung 90 min | box lacked Chromium libraries; a test started Chromium before its cleanup; no timeouts | box-tools | libraries provisioned, test fixed, --test-force-exit, step timeouts (done) |
open |
| 9 | 2026-10-10 | remote | remote run never started (busy lock, dead ssh) | tooling: killed client left the remote run holding the lock | tooling-fragile | step timeouts; the client clears its own run (done) | open |
| 10 | 2026-10-10 | push | pre-push gate refused a 3-line main with no tests |
gate stricter than CI for unchanged targets | gate-rule | fixed with a fixture test (done) | open |
| 11 | 2026-10-10 | remote | 1,225 unrelated files committed by an agent | git add -A in a worktree holding copied untracked files |
stray-commit | agents add explicit paths; integrator audits added files (7) | open |
| 12 | 2026-10-10 | remote | long remote runs used as the first signal after merges | process: the integrator skipped the fast local checks | process-order | merge → fast list + behaviour + package tests locally → only then the remote run (7) | advice only |
| 13 | 2026-10-10 | local | behaviour test turn_watchdog_deadline (agent, local) |
the agent ran the behaviour suite at the same time as a cargo test build; the test has a wall-clock deadline (passes 3/3 alone and 1511/1511 in the full suite) | concurrency-load | run the behaviour suite alone, as CI does (rules file) | advice only |
| 14 | 2026-10-10 | local | 7 custody tests 'missing test binary env' (agent, local) | plain cargo test -p elastos-server lacks the provider binaries the gate builds first |
test-env-missing | rules file says so; the gate builds them (prepare_process_providers) | advice only |
| 15 | 2026-10-10 | remote | remote gate exit 127, 'python3: No such file' (remote) | an agent rewrote the shared remote-check tool to install test tools only from the new scripts/ci-tools.txt, which the commit under test predates, and dropped python3 from the base set |
tooling-fragile | base tool set always installed; the list only adds; agents never edit shared tools (rules file) | open |
| 16 | 2026-10-10 | remote | home summary test expects Browser (remote, Linux x86) | the Browser-hiding change made the result depend on the host CPU; the existing test assumed Browser is always listed; checked only on the Mac (arm64) | host-dependent | the test derives the expectation from the same platform rule; host-dependent changes are checked on x86 Linux before pushing | open |
| 17 | 2026-10-10 | remote | gate refused elastos-componentize with zero tests (remote, 41 min in) | my earlier gate fix covered the per-target zero-test check but missed the second, package-level check of the same rule | gate-rule | the duplicate check is removed; rule: when fixing a rule, git grep every place that enforces it |
open |
| 18 | 2026-10-10 | remote | Chromium libraries missing on a fresh remote box | the 'installed' marker sat on the reusable /work volume while the libraries live in the fresh container; failures were silenced | box-tools | marker in the container's own filesystem; failure printed | open |
| 19 | 2026-10-10 | push | GitHub refused the push: files over 100 MB in history (after a 35-min gate) | the accidental output/ files (#11) were deleted in a later commit, but they stayed in history; nothing checked history size before the gate |
stray-commit | history rewritten (tree unchanged); the gate checks every pushed blob first, in seconds; /output/ ignored |
closed (fails without the fix) |
| 20 | 2026-10-11 | release | release package: Linux ARM64 static TURN build fails (linux/mman.h; then coturn adds -I/usr/include/) |
the Browser helper source builds run only in the release workflow; #307 added them and nothing ever built them on Linux | release-only-path | reproduce in an ubuntu:24.04 ARM64 container in minutes; fix; a CI job builds the upstream assets whenever their recipes or build scripts change (8) | advice only |
| 21 | 2026-10-11 | release | release package: Mac step's JSON broken ('Invalid numeric literal') | build tools inherit stdout, which the shell captures as JSON | release-only-path | stdout guard in release-upstream-assets.py; same CI job (8) | advice only |
| 22 | 2026-10-11 | remote | catalog contract test 38 != 39 (remote, Linux) | a host-dependent rule (Browser per platform) was applied to the catalog; the test built its catalog for darwin-arm64 but read the registry for the host | host-dependent | the registry takes the platform like the catalog; rule: any function whose output depends on the host takes the platform as an argument, and tests pass it explicitly | open |
| 23 | 2026-10-11 | local | Agent checklist commands are absent from the Chat source | scripts/test-source-gate.sh is absent from candidate 1433c9fd and fetched base 792942b5; the supplied basic gate's scripts/home-entropy-check.mjs is absent from the candidate. Current CI lists its source checks directly |
brief-base-mismatch | Resolve commands against the approved source; report unavailable literals and run the current CI command list under the required environment before push (proof in progress) | advice only |
| 24 | 2026-10-11 | local | Additional pre-push source check: 112/113 release-platform-input cases pass; one nested bootstrap test times out after 180 seconds | Exact install-bootstrap-test.py --bash /bin/bash timeout; cause under review |
concurrency-load | Retain the 14 passed commands and raw failure; run the exact focused bootstrap diagnostic and six remaining commands; resolve the cause and guard before push | advice only |
| 25 | 2026-10-11 | local | Chat candidate 1433c9fd: full unchanged canonical pre-push gate and ordinary push passed with four test threads, including 2,750 server library passes, zero failures and 27 ignored, plus 228 server binary passes |
First local package attempt omitted canonical fixture preparation. The retry built all three current-source providers but retained three startup failures and an interrupted FIFO case; those three cases then passed in isolation. The earlier default-package results remain false | test-env-missing | Retain both failed package attempts and the isolated diagnostic beside the full canonical gate and verified publication. This gate result does not relabel the interrupted default-package attempt as passed | advice only |
| 26 | 2026-10-11 | local | Browser readiness fixture: eleven SyntaxErrors in the full local behavior suite at 36571a53 |
The fixture evaluates extracted ES module text as a script; the newly exported retirement helper kept its export prefix |
fixture-shape | Reviewed 1f36ad20 strips that prefix before evaluation. All 28 focused cases pass; the complete behavior suite covers the fixture before integration. #217 owns the exact proof |
advice only |
| 27 | 2026-10-11 | local | Browser retirement deadline at 1f36ad20: one missing rejection among 1522 behavior tests |
Sleep can consume the retirement budget; a subsequent 1 ms probe or late ready reply could escape the deadline result | deadline-logic | Reviewed 550f60aa checks the remaining budget after sleep and after the probe. Two deterministic real-socket regressions preserve socket identity and prove both boundaries. Final 1524/1524 behavior tests pass alone; source gate and ownership smokes pass. Target fault proof remains open in #217 |
advice only |
| 35 | 2026-10-11 | local | codex-review.sh exit 1: 'Not inside a trusted directory' when WORKTREE is not a git repo | the wrapper assumed every review target is a git worktree | tooling-fragile | codex-review.sh passes --skip-git-repo-check | closed (fails without the fix) |
| 39 | 2026-10-11 | release | Release package run 37860198897/1: admit the exact source | release dispatched (8 Oct, alpha.11/12) with a source commit that was not on develop | release-input | release-package.yml admit step refuses a commit not on develop | closed (fails without the fix) |
| 40 | 2026-10-11 | local | scripts/test-source-gate.sh (full run on macOS, 2026-10-11 09:19) left an untracked tmp1nqh5htg/snapshot/.local/state/gh/device-id in the worktree; the pre-push gate then refused the dirty tree. Some test runs the real gh with a relative HOME/temp path; culprit not yet identified. | a source-gate test runs the real gh with a relative HOME, leaving gh/device-id in the worktree | test-tooling | the gate refuses a dirty tree (caught it); culprit test to be found | advice only |
| 41 | 2026-10-11 | local | Approved transplant onto develop kept the current native build tests but initially omitted two vsock policy fixture files and the new linux-amd64 Node recipe expectation; focused producer suite found both before gate. Updating exact fixtures and platform set. | transplant onto develop dropped two fixture files and a platform expectation | merge-resolution-loss | focused producer suite before the gate | advice only |
| 47 | 2026-10-11 | local | Artifact preflight incorrectly compared the current multi-architecture ARM recipe to the alpha.15 ARM receipt and tried a retired Browser entropy script. Exact per-architecture comparison confirms both input sets match reviewed 550f60aa; the retained amd64 image matches 23a95d6d. The alpha.15 ARM re | artifact preflight compared the multi-arch recipe with a single-arch receipt and called a retired script | tooling-fragile | none: one-off operator script, corrected by hand | n/a |
| 50 | 2026-10-11 | local | Canonical source gate on merged 4ed8009 failed release-signer-test test_pinned_community_network_is_bound_like_the_catalog: alpha.15 exact advertised-versus-approved artifact binding rejects the older Chat fixture as an unbound artifact. Update the fixture to the current signed packaging contract; | an older Chat fixture failed alpha.15's stricter artifact binding after the merge | merge-resolution-loss | source gate runs release-signer-test (caught it) | open |
| 51 | 2026-10-11 | local | Canonical source gate on merged 4ed8009 timed out InstallerAdmissionTest install-bootstrap-test.py after its 180-second child budget while other lanes ran heavy builds. Earlier C had the same nested deadline risk. Preserve the gate budget, identify any owned child cleanup and test the changed isol | install-bootstrap test hit its 180 s budget while other lanes built | concurrency-load | builds serialized through the lease; rerun alone before calling it a failure | advice only |
| 52 | 2026-10-11 | release | Release package run 38149169301/1: aarch64-linux N and N+1 | coturn's configure enables any optional backend whose headers the host has; GitHub's ubuntu-24.04-arm runner ships MySQL headers, our ubuntu:24.04 container repro did not, so the repro passed and the release failed (32 min in) | host-dependent | build script turns every optional coturn backend off (TURN_NO_*); the new CI guard runs the exact release command on the real GitHub ARM runner whenever these scripts change | open |
| 53 | 2026-10-11 | local | A hand-run signer fixture check used the default Python 3.9 environment instead of the prescribed Python 3.12 PATH and failed protected-tool ancestor admission. The prescribed Python 3.12 run passes all 68 tests; production permission guards remain unchanged. | hand-run signer check used Python 3.9 instead of the prescribed 3.12 PATH | test-env-missing | agent-checks setup line; the gate sets PATH itself | advice only |
| 54 | 2026-10-11 | local | The focused installer resume check was selected through module import using InstallationTests, but the method belongs to a later test class. unittest rejected the nonexistent method before executing it; select the actual class from source instead of inferring the class from an earlier declaration. | focused test selected through the wrong unittest class | test-tooling | none: operator selection slip; rerun with the right class passed | n/a |
| 56 | 2026-10-11 | local | local gate fails with Homebrew defaults (Node 22, Python 3.14) where CI (Node 26, Python 3.12) passes: browser-viewer-restore, release-publish.test.mjs, update-hop-compare-test.py | the gate assumes the CI toolchain versions but never checks them | test-env-missing | gate checks Node major against .nvmrc and Python 3.12 first, with the PATH fix (ci/merge-test-guard) | open |
| 57 | 2026-10-11 | remote | pre-push stopped: command failed: ./scripts/test-behaviour.sh | two browser_config tests assume a host where the local launcher exists; on Linux x86 they now see Browser enabled/disabled differently (browser_config.rs:899, :972) | host-dependent | tests pass the platform and launcher presence explicitly (R4); Linux remote check before push | open |
| 58 | 2026-10-11 | remote | behaviour suite: browser-vm-shutdown-order.test.mjs evaluates extracted launcher code; removePrivateSessionInputs is not defined in the eval context (3 tests) | the test evaluates an extracted function; the new helper it calls was not extracted with it | fixture-shape | import the real module instead of evaluating extracted text (R20) | advice only |
| 59 | 2026-10-11 | user | called Sasha (SashaMIT) 'she' twice; Sasha is a man | a personal fact lived only in conversation memory, which is summarised and loses details | context-loss | people facts in the generated agent brief and in my session summary's critical context | advice only |
| 60 | 2026-10-11 | user | told Anders #274 mainly makes updates smaller and that updates re-download every app; code already skips unchanged apps and providers dropped the version stamp in alpha.8 | stated code behaviour from memory before checking the code | unverified-claim | check the code (or a scout) before any claim used for a decision | advice only |
| 61 | 2026-10-11 | user | told Anders #183's other 12 items are not needed for public use; three are public-safety items | summarised an issue without reading each item | unverified-claim | read every Done-means item before recommending scope | advice only |
| 62 | 2026-10-11 | user | kept 'no relays in 0.8.0' although it breaks the Carrier principle (any DID reaches any DID on any network); Anders had to point it out | carried an old scope cut forward without checking it against first principles | unverified-claim | check each scope cut against the ElastOS principles before planning | advice only |
| 65 | 2026-10-11 | user | planned to tie local-network announcing to the Discovery switch without checking how same-Wi-Fi Chat finds peers | designed from the issue text, not the code | unverified-claim | check the code path before changing a network default | advice only |
| 66 | 2026-10-11 | local | ran the 16-minute Linux remote check on the Browser candidate before its local gate had passed (own rule R18) | skipped the fast local step under time pressure | process-order | the queue enforces the Linux pass; local gate first is still advice | advice only |
Done means (added 10 Oct)
- (done) One list of CI's fast checks,
scripts/test-source-gate.sh, run by CI,just verifyand every push. - (2) A policy test fails when a CI fast-job step runs a command that is not in the shared lists or an explicit long-job list with its reason.
- (3) The gate runs every test target (lib, bins, integration tests) of each touched package.
- (4) The product UI browser smokes run in the shared lists, or the long-job list says why not.
- (5) Before every push, the gate also runs on Linux (
elastos-remote-check --slot N) for changes that touch Rust or scripts. - (6) One tool list: the shared lists check every external program the tests call before running anything and stop with "missing tool: X".
- (7) Agent and integration rules recorded where every task brief reads them.
- (8) CI builds the release upstream assets (TURN, crosvm, debugfs, Python, Node) for Linux ARM64 and macOS whenever
scripts/browser-upstream-build.py,scripts/release-upstream-*.py|jsonorscripts/prepare-release-platform.shchange, so release-only paths never meet their first run at release time. Proof: ci: build the release upstream assets on the real runners; coturn without host-detected backends #346 (https://github.com/Elacity/elastos-runtime/blob/aa3d61e4478a1a1a63dbef0dae97c23a0f9f4851/.github/workflows/ci.yml#L563)
Activity
andersalm commentedon Oct 9, 2026
This is the active CI consistency task while #311 Group A is repaired privately. Approved base and PR target remain develop
7184e0444079881e62ef99a6c3807547119a4534.I will cover the three recorded acceptance gaps and a related build-cost defect: the local gate currently cleans and checks the complete Runtime workspace even for documentation or generic CI tooling with no Rust consumer. Basic formatting and product checks remain required; Rust compilation will follow the affected package graph. Fixtures will prove that shared contracts and embedded product inputs still widen that graph, and that dependent warnings and release self-tests fail locally. The installed-journey wait correction recorded in #194 will have a separate commit. No candidate has been published yet.
andersalm commentedon Oct 9, 2026
Candidate prepared on
fix/319-ci-prepush-and-journeyat6d6330e4a414dcb0dbe5b35b828fb6f94e05468d, tree4923852246b10d6f59c6d54fd4df917a9ae941d2, from approved develop7184e044. The branch is clean and two commits ahead of its base. The local gate uses the CI warning flags, lints reverse consumers, and runs release self-tests when their inputs change. Documentation and generic CI tooling retain the basic checks without unrelated Rust clean/build work. The second commit repairs #194's initial model-preparation deadline while keeping the existing Retry deadline and journey assertions.Independent agent source review approves the complete diff (SHA-256
f042484ebbea0ae0c29d3889b448bbcfa7c7176df41db518699bd060bc01e82b). The full fixture suite passed 64 of 65 tests; the remaining fixture expected two consumer workspaces where the corrected graph selects three. Its exact workspace assertions were updated and the focused regression passes.git diff --checkand the journey script syntax check pass. The push hook will run the complete 65-test candidate gate before publication. Hosted journey proof remains pending. This issue stays open for PR review and merge; next work in this lane waits for that gate.andersalm commentedon Oct 9, 2026
PR #326 is published for review at candidate
6d6330e4a414dcb0dbe5b35b828fb6f94e05468d(tree4923852246b10d6f59c6d54fd4df917a9ae941d2). The push hook passed all 65 local-gate fixtures, all 102 product-data tests, the required Runtime and chain-provider formatting gates, and diff checks against develop7184e044. Consumer selection chose zero Rust compilation for this CI-helper candidate. Independent source review remains approved for the published diff. The branch is clean and matches origin. Hosted installed-journey proof and required review/merge remain pending; the issue stays open in QA / Review.This lane keeps #319 as its sole active candidate. The next CI task must use the issue order and current blockers after this candidate is accepted.
ci: pre-push gate runs the CI behaviour suite from one shared script (#…
ci: behaviour suite runs offline; inputs install is a separate step (#…
andersalm commentedon Oct 11, 2026
@andersalm Browser's slot-3 Linux gate passed the behavior and source suites, workspace check/Clippy, and 2,676 server library tests at
32122b8b. It then refusedelastos-componentizebecause the unchanged binary target has zero unit tests. The current gate skips an untouched empty target but still rejects its package when the total executed count is zero (crate_units, final guard). This appears to be the remaining form of failure 10 above.The Browser branch remains local. Please resolve the gate's unchanged-empty-package case, with a fixture that retains rejection for a touched empty target. I will retain this failure and run the required tool on the current Browser candidate
588e76a8; #217 owns its acceptance record.andersalm commentedon Oct 11, 2026
@andersalm slot-3 on the frozen Browser candidate
1701d2a6443630503ee42aa6ae7c310e049f4ba8stops before source checks at the new 50 MB history guard. The two rejected files entered commit1b940a424in the original approved train ancestry: the 144,811,072-byte model and 85,119,584-byte Kubo executable underoutput/. Commit712533070later removed them. Browser work adds neither artifact, but its normal merges retain that history.The guard is working as designed. The repair is to transplant the reviewed Browser commits onto the current clean train and preserve the old local branch until tree comparison proves the result. This needs history-rewrite approval under AGENTS.md. The current branch remains local. The owned x86 machine continues the exact-source installed journey with the existing verified image; Mac checks wait for the shared lease. #217 holds the candidate and installed gates.
SashaMIT commentedon Oct 11, 2026
Local gate on a Mac with Homebrew defaults (Node 22, Python 3.14) fails where CI (Node 26, Python 3.12) passes: 5
browser-viewer-restoretests andrelease-publish.test.mjsunder Node 22, andupdate-hop-compare-test.py("fileobj not provided for non zero-size regular file") under Python 3.14. The same on clean develop 60a1275.With Node 26 + Python 3.12 first on PATH the gate passes (verified for 12c06a6).
Could the gate check or pin the Node and Python versions it needs?
andersalm commentedon Oct 11, 2026
@SashaMIT agreed: the gate will check first that Node matches .nvmrc and Python is 3.12, and stop with the PATH fix (branch ci/merge-test-guard). Recorded in the failure ledger.
andersalm commentedon Oct 13, 2026
Postponed to 0.8.1: CI cleanup/hardening; no workflow or deletion changes on release day.