Skip to content

fix(ontogeny): harden runtime consistency and lifecycle contracts - #391

Open
imbajin wants to merge 68 commits into
apache:ontogenyfrom
hugegraph:ontogeny
Open

imbajin wants to merge 68 commits into
apache:ontogenyfrom
hugegraph:ontogeny

Conversation

@imbajin

@imbajin imbajin commented Oct 29, 2026 •

Copy link
Copy Markdown
Member

Before → After

Before: The pending Ontogeny proposal needs further review fixes for runtime consistency, lifecycle handling and compatibility.

After: Apply the follow-up fixes and regression coverage from the existing fork branch: object/version and idempotency consistency, transaction/outbox lifecycle, policy and identity handling, typed synchronization, and console/deployment behavior.

Dependency and scope

Depends on 🔗 AI #380. This stacked PR targets apache:ontogeny, not main, and contains 55 commits beyond that proposal's current head (155 changed files). It preserves the existing hugegraph:ontogeny source branch and avoids duplicating the original platform introduction as a second PR into main. Recheck the base and remaining diff after #380 merges; do not assume squash or branch deletion preserves this comparison.

Validation and upgrade boundaries

The author reports backend/frontend CI, PostgreSQL concurrency checks and isolated HugeGraph projection smoke tests. These results have not been independently reproduced by this submission check. Existing databases may require explicit migrations/backfills, and automatic outbox processing can deliver queued external events. See the source PR's detailed compatibility and deployment notes before adoption. This submission is not release, live-cluster or UI acceptance.

Source and detailed evidence: 🔗 hugegraph/hugegraph-ai #90

Visual overview

Ontogeny 架构与提议中的堆叠分支关系

The diagram illustrates the shared architecture and the proposed stacked relationship between #391 and #380; it does not indicate that either PR has merged or that release acceptance has passed.

orion added 30 commits October 29, 2026 16:28

@bitflicker64 bitflicker64 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: yes. Summary: The automatic outbox loop scans all delivered history on every tick, one failing derivation row stalls derived values for its whole object type, and the bootstrap admin password drops to six digits while the password floor for every account drops to 6.

Corpus store + external extraction agent (stdin/stdout JSON or HTTP),
schema compilation, deterministic binder with reference resolution,
gated task flow (extract -> review -> confirm; nothing writes before
confirm), /admin/documents API surface and the /documents console
workspace (source switcher, extractor status, staged runs, extraction
graph editor). Platform wiring: kernel, registry, stores, agent broker,
builder ops and API factory updated for the new subsystem; web shell,
API client and shared components updated accordingly.

The semantic kind registry (core/kinds.py) lands here because the whole
kernel imports it; its docs and console board follow in the taxonomy
commit.
…uring

New semiconductor lifecycle showcase (10 objects, 8 links, 16 actions,
4 functions, lifecycle Cedar policies, product/fab/cqe copilots, eval
suite, guided demo story) that also serves as the test-suite golden
package: it carries the platform coverage the retired package owned —
marking (tapeout-version.mask_cost), a derived property (fab_lag_h),
all seven effect kinds (webhook/set-link/archive-object included),
equipment reassignment and an intentional RSI demo gap (field-failure
lot_id left unmodeled on purpose).

product-manufacturing is removed together with its docs and tools; the
document-intelligence FMEA case moves to the module FMEA agent
(tools/doc_agent_fmea_module.py) dispatched by the shared demo router.
All domain-touching tests are re-targeted at chip-lifecycle seed data
with their platform assertions preserved; the two superseded contract
files are dropped (covered by test_chip_lifecycle.py).
…integration

- Kind taxonomy: core/kinds.py docs counterpart (00-kind-taxonomy zh/en),
  frozen-table contract test, console arch board reading the registry via
  kindVisuals; board caption counts corrected (9 platform resources of 11
  kinds).
- Python 3.12+: ontogeny requires-python/classifiers raised, CI matrix
  pins bumped, py312 per-file ruff target with 58 mechanical modernizations
  (datetime.UTC alias, TimeoutError), README floor statement and test
  badges updated.
- Selective CI: ontogeny registered in scripts/ci.py (module suite +
  3.12-3.14 full matrix), ontogeny-ci.yml converted to the reusable
  ontogeny.yml workflow wired into ci.yml preflight/result, selection
  tests updated; docs/quality/ci.md documents the Ontogeny suite.
- Docs: reference manual gains the documents API group and /documents
  route (104 operations / 92 paths, counted from live OpenAPI), the UI
  tour gains the document-intelligence page (zh/en, real screenshot),
  quickstart/UI-tour sample-domain references now chip-lifecycle, the
  obsolete merge-plan scratch doc is removed, web e2e and bench target
  the chip domain.
…ence + chip-lifecycle

Merge the mainline's 55-commit hardening wave (sandbox bounds, approval
policies against unmasked state, idempotency scoping, cross-domain table
guards, typed JSON columns, watermark boundaries, staged builder saves,
PostgreSQL-gated concurrency suites, Python 3.12 floor) with this branch's
document intelligence subsystem and chip-lifecycle golden domain.

Conflict resolution kept both sides' semantics:
- service.py: mainline _install_snapshot (cache snapshot + rollback) with
  the layout-store rebind re-applied inside it (switching domains must
  rebind the canvas layout -- unfixed on the mainline)
- api_factory: mainline try/lifespan structure + the document-extract
  skip in the demo populate loop
- db.py: both sides' additive column migrations (documents tables +
  outbox cursor receipts)
- builder_ops: mainline staged/journaled save rewritten onto the
  core/kinds.py KIND_DIRS registry (module-scope import -- the mainline's
  new _materialize also consumes it)
- AgentBroker lost its task-handler surface on the mainline: document
  extraction now runs the doc-agent protocol directly (no session per
  extraction payload); the broker-path test now pins the direct run
- tests: the mainline's new suites (effect ordering, agent atomicity,
  idempotency scope, marked filters, typed storage, PG gates, ...) are
  re-targeted at chip-lifecycle seed data with their platform assertions
  intact; PLANNER-era helper aliases kept for cross-suite imports
- docs: mainline monorepo README + this branch's badges/wording; the
  mainline's hardened quickstart (deploy/credentials/auth defaults) with
  the minimal-package section retained; validator list takes the
  mainline's markings-on-PK additions
- CI: the reusable ontogeny.yml keeps the mainline's frontend lint and
  PostgreSQL concurrency job, parameterized through the selective-CI
  inputs; scripts/tests floor assertions follow the parameterized wiring

Verified on Python 3.12: 898 backend tests passed (31 by-design skips:
live-LLM/PG-gated), web 278 + build, ruff clean, CI selection self-tests
green, demo smoke (document extraction 7/7 + tapeout chain) green.
The apache/skywalking-eyes preflight gate scans every tracked file
outside the licenserc ignore list (.md/.sql/.cedar/.json/.github etc.),
and the files introduced with the document-intelligence and
chip-lifecycle work shipped without the header -- which failed the
preflight license step and skipped every selected module job on PR #90.

Headers added to: chip-lifecycle package YAML (objects/links/actions/
functions/policies/projections/stores/evals/agents/layout/ontology),
the supplier-risk test fixture package, the deterministic doc agents
(demo/reference/openai/module-FMEA) and the shared document test
fixtures. SQL and Cedar sources stay headerless: the ignore list
covers them and '#' breaks both syntaxes.

Local reproduction of the licenserc scan (same ignore rules) now
reports zero missing headers; full suite re-verified 898 passed.

@bitflicker64 bitflicker64 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: yes. Summary: Document extraction drops non-snake_case properties, rewrites numeric-looking strings, breaks a store when a removed file returns, re-raises rejected candidates and mixes attempt ids with run ids; the three earlier comments still apply at this head.

The earlier review comments below still apply at 0e0b6ab; this head does not fix them.

  • Important: this anti-join starts at the first outbox row and nothing prunes ontogeny_outbox or ontogeny_outbox_receipt, so each 1-second tick runs four scans whose cost grows with all delivered history (every sync adds one row per source row). A local SQLite run with 1M delivered events spends about 1 s per idle tick. Keep a per-consumer floor below which every ID is acknowledged and add OutboxRow.id > floor to the query. (#391 (comment))
  • Important: a row whose function raises fails the whole map_rows batch, and its pending entry now persists, so every later drain includes it and no row of that type gets new derived values while it stays pending. Retry a failed batch row by row and leave only the rows that still fail pending. (#391 (comment))
  • Important: the generated admin password has 10^6 values, the username defaults to admin, and /auth/login has no attempt limit, while MIN_PASSWORD = 6 lowers the floor for every account. Generate a high-entropy bootstrap secret such as secrets.token_urlsafe(24) and keep MIN_PASSWORD = 8. (#391 (comment))

Comment thread ontogeny/server/ontogeny/stores/repo.py Resolved
@imbajin

imbajin commented Nov 8, 2026

Copy link
Copy Markdown
Member Author

Follow-up at head 066a796247c5fd8afaa31f3178ad64a42d829ec2: the eight previously reported threads have been addressed and individually replied to. This does not establish that the entire stacked PR has been reviewed or is ready to merge.

Remaining confirmed boundaries:

  • Document IDs are derived from the relative path and content without a store identifier. Two stores containing the same relative path and bytes can therefore attempt to insert the same global document ID. Changing existing IDs requires a compatibility/migration design; this has not been fixed here.
  • Incorrect historical attempt/run associations created before this fix are not backfilled; old action revision snapshots also do not gain derived fields automatically.
  • For the outbox migration, stop the old worker before upgrading and use one service process per database. The first dispatch after startup reconciles historical receipts once.
  • This PR targets apache:ontogeny and depends on feat: Introduce Ontogeny — a self-improving operational ontology platform built on HugeGraph #380, which remains open. Recheck the base and complete diff after that dependency changes. The CI for this exact head is still running.

Validation: 156 targeted backend tests passed and 4 PostgreSQL-dependent tests were skipped in that run; a separate run against a real PostgreSQL 16 instance passed 12 outbox/lifespan tests, and the frontend access tests passed (19 tests). The combined backend run emitted three aiosqlite event-loop-closed warnings. No production deployment or old-data migration was validated.

@imbajin

imbajin commented Nov 8, 2026

Copy link
Copy Markdown
Member Author

Independent follow-up score: 9.2/10 for the reviewed fixes in 84d2b5d and 066a796 only. This is one independent reviewer’s assessment of those changes, including outbox migration/replay, derivation retry and ingestion fixes; it is not a score for the full 155-file PR. No new blocker was found in that reviewed repair scope.

The document ID collision, historical data and deployment boundaries remain. The exact-head CI is still running, and #380 remains open.

@imbajin

imbajin commented Nov 8, 2026

Copy link
Copy Markdown
Member Author

Follow-up: the cross-store document ID collision noted above is fixed in bef3b9237b1505e2657dc680c2dce1770d271597. New IDs are scoped to the store; existing same-store legacy IDs and references remain intact. Listing, preview, and ingestion share the mapping. A true concurrent scan regression covers identical paths and content in two stores.

Rollback is not seamless: back up the database and corpus together before upgrading, and restore the matching backup if downgrading; that loses changes since the backup. If both legacy and scoped records already exist for the same document, scanning stops before altering references and requires deliberate reconciliation. This change does not guess or rewrite historical attempt/run associations. CI for this exact head is pending.

@imbajin

imbajin commented Nov 8, 2026

Copy link
Copy Markdown
Member Author

Follow-up at d97afa1eb6b6238e0d136ec5d2bfc470bc885375: historical action snapshots are now evaluated with derived fields assembled against the applicable live or candidate schema, without rewriting audit history. Missing required historical inputs, incompatible parameters and expression errors are reported as unknown rather than a passing replay/shadow result; explicit null, defaults, archive results and retargeted object types are handled separately. Regression coverage includes historical struct fields and parameters.

This does not reconstruct or migrate the ambiguous historical attempt/run associations described above: the original rows do not contain enough information to identify their provenance reliably. Existing limitations on rollback, single-process dispatch, and the open #380 dependency still apply. The 77 evolve and query-service tests passed after integrating the concurrent branch changes; CI for this new head is pending.

@imbajin

imbajin commented Nov 8, 2026

Copy link
Copy Markdown
Member Author

Independent review follow-up

Reviewed head d97afa1eb6b6238e0d136ec5d2bfc470bc885375 in two focused, independent lanes:

  • Data consistency: 9.5/10 for outbox migration/replay, cross-store document identity and legacy compatibility.
  • Historical replay: 9.5/10 for missing versus null fields, nested structures and parameters, archive/type changes, and rule errors.

No further actionable findings were identified in those scopes. These scores are not a review of all files in the PR. The reviewers performed static inspection and did not independently run tests. The implementation team reports 77 related evolve/query tests passing, plus targeted PostgreSQL, SQLite and document-ingestion tests.

Limits: stop the old worker before upgrading and run only one service process per database. Historical delivery gaps require audit and scoped replay, which can duplicate delivery; old audit records lack schema versions and cannot safely be backfilled automatically. Current-head CI is still running, and #380 remains an upstream dependency.

@lplalbert

Copy link
Copy Markdown

Independent verification at d97afa1 + four findings from a cross-platform check

Reproduction (Linux)

Environment: Ubuntu 22.04 x86_64, Python 3.12.13 (uv-managed), SQLite/aiosqlite, single process. Ran the regression suites for the eight reviewed fixes plus the historical-replay fix:

tests/test_outbox_receipts.py tests/test_outbox_lifespan.py tests/test_derivation_retry.py tests/test_documents_ingest.py tests/test_documents_tasks.py tests/test_documents_binder.py tests/test_stores.py tests/test_auth_identity.py tests/test_auth_rbac.py

Result: 104 passed, 2 skipped (the two ONTOGENY_PG_DSN-gated tests in test_outbox_receipts.py), 0 failed (390s). This reproduces the reported validation for this scope, including the cross-store document-ID and concurrent-scan regressions from bef3b92 and the historical-replay coverage from d97afa1.

Not covered here: the ONTOGENY_PG_DSN tests (no PostgreSQL in this environment — the "12 outbox/lifespan tests on PostgreSQL 16" remain unreproduced by me), the frontend tests, deployment, and the combined-suite scale.

The same nine files were also run on Windows 10 (zh-CN, Python 3.12) to check platform assumptions. That surfaced the findings below. One further Windows failure, test_credential_symlink_is_not_followed, is an environment limitation (symlink creation needs privilege there), not product behavior.

Finding 1 — os.fchmod makes admin bootstrap fail on Windows (Important)

ontogeny/server/ontogeny/auth/service.py:81 (new in this stack): _write_bootstrap_password calls os.fchmod, which does not exist on Windows. With the secure default (ONTOGENY_ADMIN_PASSWORD unset), bootstrap raises AttributeError: module 'os' has no attribute 'fchmod' and the admin account is never created.

Evidence (same nine files, Windows 10 zh-CN): 50 passed, 2 skipped, 9 failed, 45 errors. 52 of the 54 non-passing results trace to this one line (45 fixture errors + 7 test failures) — including the cross-store document-ID and concurrent-scan regressions, which fail before reaching their assertions. With ONTOGENY_ADMIN_PASSWORD set (bypassing the file path): 96 passed, 2 skipped, 6 failed, 2 errors; four of the remainder deliberately exercise the credential-file path and hit fchmod again.

The except BaseException: path.unlink(missing_ok=True) cleanup does work — no credential file is left behind — so the defect is availability, not leakage.

Suggested fix: gate the call (if os.name == "posix": os.fchmod(...)) and note in the docstring that Windows stdlib cannot enforce 0o600 (on POSIX the os.open mode is umask-masked and fchmod is what makes the bits exact — that intent should stay). If Windows is explicitly out of scope for the server, an actionable error at bootstrap would still beat an AttributeError from stdlib internals.

Finding 2 — upload reports OS-native relative paths while the scan pipeline uses POSIX (Important)

documents/engine.py:1652 builds the written list with str(target.relative_to(root)), so on Windows the response contains supplier-notes\initech.md. The scan/identity pipeline normalizes the same value to POSIX (documents/corpus.py:101, path.relative_to(base).as_posix()), and test_upload_then_ingest asserts the POSIX form (assert out["written"] == ["supplier-notes/initech.md"], failing at tests/test_documents_ingest.py:427 on Windows).

Same concept, two representations across boundaries of one feature. Suggested fix: written.append(target.relative_to(root).as_posix()), matching the corpus scan.

Finding 3 — locale-dependent text IO in tests (Minor)

Several tests read/write files without encoding=. On a non-UTF-8 default locale (observed: Windows zh-CN/GBK) this raises UnicodeDecodeError on non-ASCII content. Two failures observed: tests/test_documents_tasks.py:229 (agent YAML) and tests/test_documents_ingest.py:669 (shipped note file). Product code passes encoding="utf-8" consistently; ~50 bare read_text()/write_text() sites remain under tests/. Adding encoding="utf-8" there is cheap and matches the boundary discipline in server/.

Note — the occasional aiosqlite event-loop-closed warnings (hypothesis, not confirmed)

I saw this warning class once (3× PytestUnhandledThreadExceptionWarning: RuntimeError: Event loop is closed) but could not reproduce it on demand (a repeat of the same set: zero warnings; a script that drops a doc_context engine after its loop closes: clean). Source-derived observation: tests/doc_fixtures.py::doc_context yields its ServiceContext without teardown while tests/conftest.py::client ends with await sc.engine.dispose(); ServiceContext has no shutdown method at all. Unifying teardown there is cheap, but I could not confirm it removes the warnings — flagging as a hypothesis, not a verified fix.

@bitflicker64 bitflicker64 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: yes. Summary: The earlier outbox, derivation, bootstrap-password and ingestion fixes hold at this head, but the demo still accepts the X-Ontogeny-Principal admin header the docs now call disabled, a superseded document task can be re-confirmed over newer data, reviewer overrides bypass type checks, agent-path rule and policy refusals no longer reach the revision audit, and --reseed leaves a credential file that stops the next start. The Minor items are smaller lifecycle, message and docs issues.

Comment thread ontogeny/README.md

> [!NOTE]
> The API is session/bearer authenticated. The `X-Ontogeny-Principal` dev header is **enabled by default** for evaluation (`ONTOGENY_DEV_AUTH=0` disables it) — turn it off in production, where the agent surfaces (REST + `/mcp`) then require a signed-in driver and every session records WHO drove it.
> The API is session/bearer authenticated. The `X-Ontogeny-Principal` dev header is disabled by default; set `ONTOGENY_DEV_AUTH=1` only for development or evaluation. With it disabled, the agent surfaces (REST + `/mcp`) require a signed-in driver and every session records WHO drove it.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important: ontogeny serve --demo, which the Docker CMD and the quickstart both run, still sets ONTOGENY_DEV_AUTH=1 through DemoPaths.as_env, so any client that sends X-Ontogeny-Principal: {"id":"x","is_admin":true} passes require_admin while this note says the header is disabled by default. Stop --demo from enabling dev auth unless the operator sets ONTOGENY_DEV_AUTH=1.

task = await session.get(DocumentTaskRow, task_id)
if task is None:
raise NotFoundError(f"no task {task_id}")
if task.status == "confirmed":

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important: confirm refuses only confirmed tasks, so a superseded revision can be confirmed again and rewrites its older graph over the newer confirmed one, leaving both tasks confirmed; TasksTab.tsx shows the Confirm button for it. Refuse confirm, save_graph and run unless the task is still open for review.

try:
props[prop] = coerce_value(ptype.ptype(), value, prop=prop)
except Exception:
props[prop] = value

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important: when coerce_value rejects a reviewer override, this fallback writes the raw value, so {"risk_level": "CATASTROPHIC"} is stored on an enum[LOW, MEDIUM, HIGH] property and effective_date: "not a date" fails later as a 500. Raise a DocumentError on a coercion failure instead of storing the raw value.

params,
target_id,
expected_revision=args.get("expected_revision"),
commit_rejections=False,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important: with commit_rejections=False inside begin_nested(), a rule-rejected or policy-denied agent write rolls back its rejected_rule/denied_policy revision with the savepoint, here and in the approval path of decide, so agent refusals no longer reach the audit stream that REST refusals still write. Record the refusal revision after the savepoint rollback.

try:
fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
except FileExistsError as exc:
raise StoreError(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important: ontogeny serve --demo --copy-package --reseed deletes ontogeny.db but keeps admin-bootstrap-password, so the next start finds no admin, raises here and exits with Application startup failed. Remove the stale credential file in prepare_demo when --reseed deletes the database.

wanted_ids = set(doc_ids or [])
async with self.sc.sessionmaker() as session:
docs = (await session.execute(select(DocumentRow).where(DocumentRow.store == store_name))).scalars().all()
selected = [d for d in docs if d.id in wanted_ids or Path(d.path).resolve() in paths]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: this selects removed rows whose path matches, so uploading a revised file under the same name returns a second task bound to the removed document id that reads the new bytes. Select only active documents.

issues = [{"message": "issues must be an array"}] if issues is not None else []
status = chunk.get("status", "final")
if status not in _VALID_STATUS:
status = "failed"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: status is reassigned before the issue is built, so the message always reads invalid status 'failed' and drops the value the agent sent. Build the message from the original value first.

raise SkillError(f"skill examples must be a list of at most {MAX_EXAMPLES} items")
if len(json.dumps(examples, ensure_ascii=False, default=str)) > MAX_EXAMPLE_CHARS:
raise SkillError("skill examples are too large")
language = str(skill.get("language") or "en")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: the snapshot always stores language, defaulting to en, so skill.get("language") or cfg.language in tasks.py never falls back to the store's documents.language (default zh). Leave language unset when the caller gives none.

disabled={createDefault.isPending}
data-testid="documents-create-default"
>
{createDefault.isPending ? t('documents.default.creating') : t('documents.default.action')}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: documents.default.action and documents.default.creating exist in neither en.ts nor zh.ts, so the create-default button shows the raw key. Add both keys to the two dictionaries.

Comment thread ontogeny/README.md
```

`tools/doc_agent_reference.py` is a deterministic reference agent that makes the
Product Manufacturing demo runnable without an LLM; `tools/doc_agent_openai.py` shows the

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: this names the Product Manufacturing demo, which this PR deletes in favour of chip-lifecycle, and line 23 reads Baselineal guardrails. Name the chip-lifecycle demo and fix the heading.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants