CS-10673 + CS-10708: Bootstrap as seed issue, wire issue-driven loop into factory:go#4379
CS-10673 + CS-10708: Bootstrap as seed issue, wire issue-driven loop into factory:go#4379
Conversation
Integrate findings from CS-10672 implementation directly into the relevant plan sections rather than appending notes: - Selection algorithm uses backlog (not ready) to match darkfactory.gts - Priority enum includes critical - Issue lifecycle uses backlog → in_progress → done/blocked/review - Exhausted issues tracked via exclusion set to prevent re-picking - RealmIssueStore uses absolute darkfactory module URL (env-sensitive) - Boxel linksToMany uses dotted-key format for blockedBy relationships - Loop outcome disambiguation table for edge cases - Migration path references actual file organization and CS-10708 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add a modular validation pipeline that runs after every inner-loop agent turn. The pipeline runs all steps concurrently via Promise.allSettled() and aggregates results. Each step is a separate module implementing the ValidationStepRunner interface — adding a new step is one file plus one line in createDefaultPipeline(). Key changes: - ValidationPipeline class implementing the Validator interface with concurrent step execution and per-step exception capture - TestValidationStep wrapping executeTestRunFromRealm() with detailed failure data read back from the completed TestRun card - NoOpStepRunner placeholders for parse, lint, evaluate, instantiate (child tickets CS-10713 through CS-10716) - Step-specific failure shapes via details field on ValidationStepResult - formatForContext() on each step for LLM-friendly failure output - Max iterations + failing validation now blocks the issue with the reason and failure context in the issue description - Extract fetchRealmFilenames() from pullRealmFiles() for reuse - Optional updateIssue() on IssueStore for max-iteration blocking - Updated phase-2-plan.md with validation architecture documentation Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…es is standalone pullRealmFiles is dead code (only used in its own tests) and doesn't need to be refactored. fetchRealmFilenames stands on its own for the test validation step's file discovery. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ersistence warning - Fix test-step error mapping: fall back to handle.errorMessage when details.failures is empty (infrastructure errors with readable TestRun) - Remove incorrect "no-op" log suffix in smoke test - Improve createDefaultPipeline test to actually verify 5 steps in order - Add warning log when IssueStore lacks updateIssue for max-iteration blocking Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
… lint - Make IssueStore.updateIssue required (not optional) - Implement updateIssue on RealmIssueStore: read card, mutate status/description, write back to realm - Update all MockIssueStore implementations to include updateIssue - Fix prettier formatting and non-null assertion lint errors Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Update orchestrator pseudocode to reflect max-iterations→blocked behavior - Document IssueStore.updateIssue and orchestrator realm writes - Update Issue Lifecycle section with max-iteration blocking - Clean up --jwt reference in boxel-cli auth section - Fix NoOpStepRunner to declare parameter names (TS strict mode) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The import was ../setup-logger (resolving to scripts/setup-logger which doesn't exist). Fixed to ../../src/setup-logger matching all other smoke test scripts. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The validation pipeline's TestValidationStep already exercises executeTestRunFromRealm internally, so the separate direct call was redundant. The smoke test now goes straight from writing test files to running the ValidationPipeline. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…into factory:go Replace the deterministic bootstrapProjectArtifacts() + runFactoryImplement() flow with a seed-issue-driven approach. The entrypoint creates a single bootstrap Issue in the realm, then delegates to runIssueLoop() which uses the IssueScheduler, ValidationPipeline, and ContextBuilder to orchestrate the full agentic loop. Key changes: - Add 'bootstrap' to IssueTypeField enum in darkfactory.gts - Create factory-seed.ts with idempotent createSeedIssue() - Create factory-issue-loop-wiring.ts to construct Phase 2 infrastructure - Create realm-issue-relationship-loader.ts for ContextBuilder.buildForIssue() - Create bootstrap-implement.md prompt template and bootstrap skill - Route bootstrap issues to dedicated prompt in tool-use agent - Relocate LoopAgent/AgentRunResult types to factory-agent-types.ts - Handle missing project in buildForIssue() with stub for bootstrap issues - Update entrypoint summary types (seedIssue, issueLoop) - Update all tests and smoke tests for new flow - Update phase-2-plan.md with implementation decisions Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 33b3d8921d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
Pull request overview
This PR refactors factory:go to bootstrap via a single seed Issue and then run an issue-driven agent loop, including wiring for context loading and a modular validation pipeline (with real QUnit test execution as the first implemented validation step).
Changes:
- Replace hard-coded bootstrap/implement flow with seed-issue creation +
runIssueLoop()orchestration. - Add validation pipeline framework (
ValidationPipeline+ step runners) and implement test validation via realm.test.gtsdiscovery + TestRun card parsing. - Add realm relationship loader to build per-issue context from
project/relatedKnowledgerelationships and update supporting tests + smoke tests.
Reviewed changes
Copilot reviewed 34 out of 34 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/software-factory/src/factory-entrypoint.ts | Switch entrypoint to create a seed issue and optionally run the issue loop. |
| packages/software-factory/src/factory-issue-loop-wiring.ts | New wiring module that constructs tools/agent/context/validator and runs runIssueLoop(). |
| packages/software-factory/src/factory-seed.ts | New idempotent createSeedIssue() helper writing Issues/bootstrap-seed. |
| packages/software-factory/src/issue-loop.ts | Add validator formatting hook + logic to block issues on max-iterations with failing validation; re-export validation pipeline. |
| packages/software-factory/src/issue-scheduler.ts | Add IssueStore.updateIssue() and implement read-modify-write updates in RealmIssueStore. |
| packages/software-factory/src/factory-context-builder.ts | Allow bootstrap issues to proceed without a linked project (stub project). |
| packages/software-factory/src/realm-issue-relationship-loader.ts | New loader to traverse issue relationships and load Project/Knowledge cards. |
| packages/software-factory/src/validators/validation-pipeline.ts | New concurrent validation pipeline with default 5-step composition (NoOps + test step). |
| packages/software-factory/src/validators/test-step.ts | New test validation step running QUnit and extracting failure details from TestRun cards. |
| packages/software-factory/src/validators/noop-step.ts | New placeholder step runner for unimplemented steps. |
| packages/software-factory/src/realm-operations.ts | Add fetchRealmFilenames() via _mtimes for test discovery. |
| packages/software-factory/src/factory-agent-types.ts | Extend validation result typing (details) and relocate loop-agent types. |
| packages/software-factory/src/factory-agent-tool-use.ts | Route bootstrap issues to a new bootstrap prompt template. |
| packages/software-factory/src/factory-agent-mocks.ts | Update imports to relocated loop-agent types. |
| packages/software-factory/src/factory-implement.ts | Update imports for relocated loop-agent types. |
| packages/software-factory/src/factory-skill-loader.ts | Add bootstrap skill priority + bootstrap issue skill resolution. |
| packages/software-factory/src/factory-prompt-loader.ts | Add assembleBootstrapPrompt(). |
| packages/software-factory/src/cli/factory-entrypoint.ts | Update CLI logging for issue-loop summaries. |
| packages/software-factory/realm/darkfactory.gts | Add bootstrap to IssueType enum. |
| packages/software-factory/realm/tsconfig.json | Add realm tsconfig for strict TS checking. |
| packages/software-factory/prompts/bootstrap-implement.md | New bootstrap prompt template. |
| packages/software-factory/.agents/skills/software-factory-bootstrap/SKILL.md | New skill describing bootstrap artifact creation. |
| packages/software-factory/docs/phase-2-plan.md | Update Phase 2 design doc to reflect implemented behavior. |
| packages/software-factory/tests/index.ts | Register new test modules. |
| packages/software-factory/tests/validation-pipeline.test.ts | New unit tests for ValidationPipeline + NoOp runner. |
| packages/software-factory/tests/test-step.test.ts | New unit tests for TestValidationStep. |
| packages/software-factory/tests/issue-loop.test.ts | Update tests for max-iteration blocking behavior + updateIssue calls. |
| packages/software-factory/tests/issue-scheduler.test.ts | Update mock store to implement updateIssue. |
| packages/software-factory/tests/factory-entrypoint.test.ts | Update entrypoint tests to assert seed-issue behavior and issue-loop integration. |
| packages/software-factory/tests/factory-entrypoint.integration.test.ts | Update integration summary expectations to seed-issue + loop semantics. |
| packages/software-factory/tests/factory-implement.test.ts | Update imports for relocated loop-agent types. |
| packages/software-factory/tests/factory-seed.spec.ts | New Playwright spec verifying seed issue creation/idempotency against a live realm. |
| packages/software-factory/scripts/smoke-tests/smoke-test-realm.ts | Convert smoke test to validate via ValidationPipeline (with real QUnit execution). |
| packages/software-factory/scripts/smoke-tests/issue-loop-smoke.ts | Add scenarios for ValidationPipeline integration and bootstrap seed flow. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Delete Phase 1 files:
- src/factory-bootstrap.ts (deterministic bootstrap)
- src/factory-loop.ts (old runFactoryLoop)
- src/factory-implement.ts (old runFactoryImplement + TestRunner wiring)
- tests/factory-bootstrap.{test,spec}.ts
- tests/factory-loop.test.ts
- tests/factory-implement.test.ts
- tests/factory-agent.{test,integration.test}.ts (old declarative agent)
- scripts/smoke-tests/factory-agent-smoke.ts
Trim factory-agent.ts to barrel re-exports only (remove OpenRouterFactoryAgent,
validateAgentActions, parseActionsFromResponse).
Move inferDarkfactoryModuleUrl() from factory-bootstrap.ts to factory-seed.ts.
Update README to describe the Phase 2 issue-driven agentic loop architecture.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Refactors the software-factory “factory:go” flow from a deterministic bootstrap + single-issue implement loop into an issue-driven agentic loop seeded by a bootstrap Issue, with a modular validation pipeline executed after each agent turn.
Changes:
- Replaces the legacy
factory-bootstrap/factory-implement/factory-looppath with seed-issue creation (Issues/bootstrap-seed) andrunIssueLoop()wiring. - Introduces a modular
ValidationPipeline(parse/lint/evaluate/instantiate placeholders + real test step) and adds unit + smoke coverage. - Adds realm-backed relationship loading for issue context (
project,relatedKnowledge) and a bootstrap prompt/skill for seed issue processing.
Reviewed changes
Copilot reviewed 45 out of 45 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/software-factory/tests/validation-pipeline.test.ts | Adds unit coverage for ValidationPipeline behavior and formatting. |
| packages/software-factory/tests/test-step.test.ts | Adds unit coverage for the test validation step (discovery, execution, details formatting). |
| packages/software-factory/tests/issue-scheduler.test.ts | Updates mocks to satisfy the expanded IssueStore interface. |
| packages/software-factory/tests/issue-loop.test.ts | Updates tests for max-iteration blocking behavior + IssueStore updates. |
| packages/software-factory/tests/index.ts | Switches test suite entry to new issue-loop/validation tests and removes legacy loop tests. |
| packages/software-factory/tests/factory-seed.spec.ts | Adds Playwright spec validating seed issue creation/idempotency in a live realm. |
| packages/software-factory/tests/factory-loop.test.ts | Removes legacy single-issue loop tests. |
| packages/software-factory/tests/factory-implement.test.ts | Removes legacy implement wiring tests. |
| packages/software-factory/tests/factory-entrypoint.test.ts | Updates entrypoint tests to reflect seed-issue creation and issue-loop execution. |
| packages/software-factory/tests/factory-entrypoint.integration.test.ts | Updates integration summary shape to seedIssue + issueLoop semantics. |
| packages/software-factory/tests/factory-bootstrap.test.ts | Removes deterministic bootstrap unit tests (superseded by seed issue flow). |
| packages/software-factory/tests/factory-bootstrap.spec.ts | Removes bootstrap Playwright spec (replaced by seed issue spec). |
| packages/software-factory/tests/factory-agent.integration.test.ts | Removes legacy OpenRouterFactoryAgent integration tests (declarative agent removed). |
| packages/software-factory/src/validators/validation-pipeline.ts | Introduces ValidationPipeline + factory (createDefaultPipeline) and step-runner contract. |
| packages/software-factory/src/validators/test-step.ts | Implements test validation step using realm test execution + TestRun card detail extraction. |
| packages/software-factory/src/validators/noop-step.ts | Adds NoOp placeholder validation steps for unimplemented phases. |
| packages/software-factory/src/realm-operations.ts | Adds fetchRealmFilenames() helper used by test validation. |
| packages/software-factory/src/realm-issue-relationship-loader.ts | Loads Project/Knowledge cards via issue relationship links for context building. |
| packages/software-factory/src/issue-scheduler.ts | Expands IssueStore with updateIssue() and implements realm-backed read-modify-write updates. |
| packages/software-factory/src/issue-loop.ts | Updates loop to block issues on max-iterations + failing validation and re-exports validator utilities. |
| packages/software-factory/src/factory-skill-loader.ts | Adds bootstrap-skill routing for issueType: bootstrap. |
| packages/software-factory/src/factory-seed.ts | Adds idempotent seed issue creation + darkfactory module URL inference. |
| packages/software-factory/src/factory-prompt-loader.ts | Adds bootstrap-specific prompt assembly. |
| packages/software-factory/src/factory-loop.ts | Removes legacy single-issue orchestrator loop implementation. |
| packages/software-factory/src/factory-issue-loop-wiring.ts | Adds “Phase 2” wiring: auth, tools, agent, schemas, validator, and runIssueLoop() invocation. |
| packages/software-factory/src/factory-implement.ts | Removes legacy implement wiring module. |
| packages/software-factory/src/factory-entrypoint.ts | Switches entrypoint to create seed issue then (in implement mode) run issue loop. |
| packages/software-factory/src/factory-context-builder.ts | Allows bootstrap issues to proceed without a linked project via a stub project. |
| packages/software-factory/src/factory-bootstrap.ts | Removes deterministic project artifact bootstrap module. |
| packages/software-factory/src/factory-agent.ts | Converts to a barrel re-export; removes legacy declarative agent + validation helpers. |
| packages/software-factory/src/factory-agent-types.ts | Adds step result details and relocates loop agent types here. |
| packages/software-factory/src/factory-agent-tool-use.ts | Adds bootstrap prompt selection based on issue type + brief URL. |
| packages/software-factory/src/factory-agent-mocks.ts | Updates LoopAgent imports to new location. |
| packages/software-factory/src/cli/factory-entrypoint.ts | Updates CLI log output to reflect issue-loop execution results. |
| packages/software-factory/scripts/smoke-tests/smoke-test-realm.ts | Converts smoke test to run the ValidationPipeline (including real QUnit execution). |
| packages/software-factory/scripts/smoke-tests/issue-loop-smoke.ts | Adds pipeline + bootstrap-seed scenarios and updates max-iteration behavior expectations. |
| packages/software-factory/scripts/smoke-tests/factory-agent-smoke.ts | Removes legacy OpenRouter agent smoke test script. |
| packages/software-factory/realm/tsconfig.json | Adds dedicated TS config for realm code compilation. |
| packages/software-factory/realm/darkfactory.gts | Extends IssueType enum with bootstrap. |
| packages/software-factory/README.md | Updates documentation to describe the issue-driven loop + validation pipeline architecture. |
| packages/software-factory/prompts/bootstrap-implement.md | Adds bootstrap issue prompt template used by the tool-use agent. |
| packages/software-factory/package.json | Removes legacy smoke:agent script; keeps factory:go and new smoke targets. |
| packages/software-factory/docs/phase-2-plan.md | Updates plan doc to match implemented scheduler/validation/seed issue behavior. |
| packages/software-factory/.agents/skills/software-factory-bootstrap/SKILL.md | Adds bootstrap skill guidance and schema expectations for seed issue processing. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
The brief may require multiple card definitions (main card + supporting field/linked cards). Update the bootstrap skill, prompt template, and seed issue to reflect this: - Issue #1 covers all card definitions and tests, not just one - Issue #2 creates catalog specs for "entry point" cards (the ones discoverable in the catalog), using judgment from the brief - At least one card must have a catalog spec Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove hardcoded 2-article assumption. The agent should create as many knowledge articles as needed to keep each one cohesive with a clear guiding principle — a complex brief may warrant several, a simple one may need just one. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Always create these two baseline articles, plus additional ones as the brief warrants (e.g., visual design details, domain knowledge). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…guidance - Route validationResults to iterate prompt so agent receives structured failure context for self-correction (not just the implement prompt) - RealmIssueRelationshipLoader now fetches the full issue card from the realm before traversing relationships, since SchedulableIssue only has scheduling fields (no relationships or issueType) - Add issueType to SchedulableIssue and mapper so context builder can detect bootstrap issues without a separate realm fetch - Clarify in bootstrap prompt and skill that update_issue writes the full card (not a partial patch) — agent must read existing attributes first - Fix helper function signatures to accept generic objects Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…er interface Add missing _targetRealmUrl parameter to run() and _result parameter to formatForContext() on MockStepRunner and ThrowingStepRunner. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- factory-seed.spec.ts: fixture realm has pre-existing issues, so use find() instead of asserting exactly 1 issue from listIssues() - factory-target-realm.spec.ts: update from old summary.bootstrap shape to new summary.seedIssue shape, verify seed issue exists in realm instead of Project card (Project is now created by the agent, not the entrypoint) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Read ${issueId}.json (the source file) instead of the indexed card
document, which can have stripped relationships during indexing.
This prevents data corruption when writing back the modified document.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Summary
bootstrapProjectArtifacts()+runFactoryImplement()with seed-issue-driven agentic looprunIssueLoop()withIssueScheduler,ValidationPipeline, andContextBuilderprojectandrelatedKnowledgerelationships for context loadingfactory-loop.ts,factory-implement.ts,factory-bootstrap.ts, old declarative agent, and all associated testsArchitecture
New files
src/factory-seed.ts—createSeedIssue()(idempotent)src/factory-issue-loop-wiring.ts— constructs Phase 2 infrastructure and callsrunIssueLoop()src/realm-issue-relationship-loader.ts—RealmIssueRelationshipLoaderforContextBuilder.buildForIssue()prompts/bootstrap-implement.md— bootstrap prompt template.agents/skills/software-factory-bootstrap/SKILL.md— bootstrap skill with card schemastests/factory-seed.spec.ts— Playwright spec for seed issue in real realmDeleted files (Phase 1 retirement)
src/factory-loop.ts,src/factory-implement.ts,src/factory-bootstrap.tstests/factory-loop.test.ts,tests/factory-implement.test.ts,tests/factory-bootstrap.test.ts,tests/factory-bootstrap.spec.tstests/factory-agent.test.ts,tests/factory-agent.integration.test.tsscripts/smoke-tests/factory-agent-smoke.tsTry it out
End-to-end test (requires a running realm server)
cd packages/software-factory MATRIX_URL=http://localhost:8008 \ MATRIX_USERNAME=your-username \ MATRIX_PASSWORD=your-password \ pnpm factory:go -- \ --brief-url https://cardstack.com/boxel-motion/Wiki/sticky-note \ --target-realm-url http://localhost:4201/your-username/smoke-test-realm/ \ --realm-server-url http://localhost:4201/(Substitute your Matrix credentials. The realm server must be running via
mise run dev-allor equivalent.)What you'll see in the CLI
Issues/bootstrap-seedwritten to the realmIssues/bootstrap-seed(bootstrap, critical priority).gtscard definition(s),.test.gtstestsSpec/*.jsoncatalog spec(s), sample instancesall_issues_done, 3 outer cyclesWhat you'll see in the Boxel app
Navigate to your
smoke-test-realmworkspace. You should see:Projects/<slug>-mvp) — projectName, objective, scope, success criteria from briefIssues/bootstrap-seed) — status:done, issueType:bootstrapdone, withprojectandrelatedKnowledgerelationshipsdone,blockedByissue Bring in a demo from ember-animated #1, with project/knowledge relationships.gtsfiles) — the cards the agent implemented.test.gtsfiles) — QUnit tests co-located with card definitionsSpec/<card-name>.json) — for entry point cards, withlinkedExamplesto sample instancesTest Runs/folder) — TestRun cards with pass/fail detailsLinear tickets
Test plan
pnpm lintpasses (software-factory)pnpm test— 342 tests pass (Phase 1 tests removed, all remaining pass)pnpm smoke:issue-loop— 8 scenarios, 28 assertions pass (new bootstrap scenario)factory-seed.spec.ts— seed issue creation in real realmfactory-target-realm.spec.ts— end-to-end seed issue creation🤖 Generated with Claude Code