Repository navigation
fix(search): restore CJK recall for core observation search - #1553
Conversation
Three structural gaps made core search near-zero-recall for CJK users (rohitg00#1539): the BM25 document never included the user's own words, query entity extraction was ASCII-only, and the vector-embedded text left concepts out. - extractTerms: index source.userPrompt (capped at 2000 chars) and a bounded prefix of source.assistantResponse (1000 chars) so a query in the user's language can hit the record it came from - extractEntitiesFromQuery: emit segmented CJK runs (>= 2 chars) via the existing segmentCjk so the graph leg fires for non-English queries - vector embed text: share observationEmbedText (title + narrative + concepts) across every embedding site — observe (restore + synthetic), compress, remember, evolve (relations), and both migrate-vector-index rebuild phases — so concepts anchor non-English content in the vector leg - COMPRESSION_SYSTEM: keep concepts in the user's language, bilingual when the observation mixes languages The prompt directive's effect needs a live model to observe; the other three are covered by red-to-green tests in the search-index, query-expansion, compression-validation and vector-index-populate test files. Fixes rohitg00#1539 Signed-off-by: akushonkamen <akushonkamen@163.com>
Review follow-up to 3166caa: indexRecords (search.ts:404, :413) and the keyword rebuild's backfill queue (search.ts:602, :633) still embedded title + narrative/content. Those are the shared channels for rebuildIndex, reconcileIndex, snapshot restore, replay and import (rebuildIndex clears the vector index first), and the backfill queue feeds embedBackfillJobs — so any cold-start rebuild or backfill would have re-embedded the whole store without concepts, including records that live writes had already embedded with the concept-aware text. All four sites now go through observationEmbedText (whitespace-trimmed, so records without concepts embed the exact text they did before), which also makes the previous commit's 'every embedding site' claim true in fact: the migrate-vector-index rebuild it cited has no production caller (no src/ importer), while these search.ts channels are the live rebuild paths. Regression tests assert the concept-bearing text on both channels: indexRecords via a recording embedding provider, and the keyword rebuild via the returned backfill jobs. Part of rohitg00#1539 Signed-off-by: akushonkamen <akushonkamen@163.com>
|
@akushonkamen is attempting to deploy a commit to the rohitg00's projects Team on Vercel. A member of the Team first needs to authorize it. |
|
Caution Review failedThe pull request is closed. Walkthrough
Merge Risk:
|
|
I ran a live-model check of the compression-prompt directive in this PR before adopting it. Setup, results, and two small suggestions below. Setup6 real CJK observations from a production store (6-8k Han characters each, Chinese prose mixed with code identifiers). Same model, same temperature, only the concept rules swapped. 2 rounds per variant, so 12 compressions per variant.
ResultsConcepts aggregated over the 6 observations x 2 rounds:
What the numbers say
Two suggestions1. Make "one language per 2. Document that existing data does not update on its own. This PR changes the write-side text ( Minor and optional: Happy to share the raw runs if that would help. |
|
Heads-up: this PR now conflicts with GitHub reports Files this PR targets that moved on
The other files this PR touches are unchanged on I can re-run the before/after CJK recall measurement against a rebased branch on a production store if that helps review. |
|
@akushonkamen thanks for tracking down the CJK recall gap in #1539. Main moved under this branch: #1546 changed failed compression recovery and #1569 changed structured capture redaction, and the branch now conflicts in Please merge current main into the branch and keep the redaction and compression-recovery behavior from main intact while restoring the raw-field indexing. CI has not run on this fork PR yet; it will run after the update. |
|
Merged the current Conflict resolution, per file:
Verification (run locally):
|
|
fix coderabbit and devin suggestions @akushonkamen |
|
Thanks @rohitg00. I'll address the open review findings on this branch: in |
|
Thanks for addressing the source-field indexing gaps in #1539. We independently encountered a related issue in AgentMemory v0.9.31 while integrating OMP (Oh My Pi). Our regression demonstrated that a retained final answer could appear in conversation replay but remain absent from keyword search when its relevant terms existed in I see that this PR adds bounded indexing for Would you consider including bounded indexing of We have a tested local correction and a synthetic migration regression covering index rebuilding, repeated restart, and rollback. We can share the relevant reproduction or patch if useful. Separately, we observed that compact search results may omit the answer text needed for useful retrieval. That appears to be a distinct search-preview concern and does not necessarily need to be addressed in this PR. Transparency note: I used AI coding assistants (Codex and OMP) to help investigate this issue, develop the local patch, and run regression tests. The findings and test results were reviewed before sharing them here. |
|
Thank you for the independent regression — a retained final answer being searchable only through the compressed narrative because its terms live in source.toolOutput is exactly the class of gap this PR's bounded indexing leaves open. Yes, please do share the reproduction (and the patch if you have one handy): if extending coverage to source.toolOutput stays within the same bounded-shape indexing approach, I'd rather fold it into this PR than leave the partial fix. Agreed the compact-preview omission is a separate concern — worth its own issue. |
|
Thanks for the quick response and for being open to incorporating this into #1553. I'll prepare the minimal reproduction and tested patch for Agreed on keeping the compact-preview issue separate. I'll follow up on that independently. |
Signed-off-by: Jacques L'Heureux <jm_lheureux@yahoo.com>
|
Thanks for offering to fold this into #1553. The tested contribution is available here: Commit: jmlheureux@f05d50e The patch is based on this PR's current head (8fe4f88) and adds bounded indexing of source.toolOutput for conversation records, with regression tests. Compact search previews are unchanged. Validation: 41 focused tests passed; 3,855 full-suite tests passed with 4 skipped. Build and synthetic worker migration, rebuild, repeated restart, and rollback checks passed. TypeScript diagnostics matched the unchanged PR baseline. The patch changes four files and is intended for incorporation into this existing PR. Codex and OMP assisted with implementation and validation; the results were reviewed before publication. |
…pr1553 by jmlheureux)
|
@jmlheureux Merged your branch (codex/tool-output-pr1553, f05d50e) into the PR branch as-is — your authorship is preserved in the history (merge commit 7421c64). Reviewed the patch: it's exactly the bounded-indexing shape we wanted (conversation toolOutput only, TOOL_OUTPUT_TERM_CAP = 1000 via slice), no out-of-scope changes, and the regression tests are solid. Verified locally: all 41 indexing-related tests pass (search-index, index-keyword-rebuild, tool-output-index-regression). Full suite: 3840 passed / 4 skipped, with 15 failures that are pre-existing on our head 8fe4f88 in this environment (fs-watcher, openclaw-connect, Docker/timing-dependent suites) and unrelated to your changes — we confirmed the same tests fail on the pre-merge baseline. Thanks for the careful work and for including the boundary tests around toolInput exclusion and the term cap. |
The embedding input is clipped at 16,000 characters. With concepts after the narrative, a long memory lost its concepts, including the CJK terms this change adds. Concepts now follow the title, and the parts are joined without empty gaps.
Core search was near-zero-recall for CJK users (#1539): a query in the user's own language could fail to return the very record it produced. Four structural gaps compounded:
Root cause
extractTerms,src/state/search-index.ts:338) never includedsource.userPrompt/source.assistantResponse, so the user's own words were unindexed — only an English title/narrative could match.extractEntitiesFromQuery,src/functions/query-expansion.ts:133) relied on a capitalized-word rule that never fires for scripts without capitalization; a pure-CJK query yielded zero entities and the graph leg stayed silent.conceptsout, so non-English concepts could not anchor records in the vector leg.COMPRESSION_SYSTEMgave no language instruction, so compressed concepts could drift to English even for non-English sources.Fix
extractTerms: indexsource.userPrompt(capped at 2000 chars) and a bounded prefix ofsource.assistantResponse(1000 chars) so a query in the user's language can hit the record it came from.extractEntitiesFromQuery: emit segmented CJK runs (>= 2 chars) via the existingsegmentCjkso the graph leg fires for non-English queries.observationEmbedText(title + narrative + concepts,src/state/memory-utils.ts:57) used by every embedding site — observe (restore + synthetic), compress, remember, evolve (src/functions/relations.ts:228), and bothmigrate-vector-indexrebuild phases — so concepts anchor non-English content in the vector leg.indexRecords(src/functions/search.ts:404,:413) and the keyword rebuild's backfill queue (:602,:633) — the shared channels forrebuildIndex,reconcileIndex, snapshot restore, replay and import — also embed throughobservationEmbedText(whitespace-trimmed, so records without concepts embed the exact text they did before). Without this, any cold-start rebuild or backfill would have re-embedded the whole store without concepts.COMPRESSION_SYSTEM: keep concepts in the user's language; bilingual (original + English) when the observation mixes languages. This directive's effect requires a live model to observe and could not be asserted locally — flagged for the reporter to verify end-to-end.Red-to-green tests live in four existing files (
test/search-index.test.ts,test/query-expansion.test.ts,test/compression-validation.test.ts,test/vector-index-populate.test.ts; +8 cases) plustest/index-keyword-rebuild.test.ts(+8 cases) for the follow-up channels.Validation
36eb328, during development):timeout 300 npx vitest run test/search-index.test.ts test/query-expansion.test.ts test/compression-validation.test.ts test/vector-index-populate.test.ts→ 8 failed (exactly the new CJK cases) / 71 passed.timeout 600 npx vitest run --exclude test/integration.test.tsafternpm run build(the integration test needs a live server on:3111and is skipped per CONTRIBUTING) → 3,357 passed / 0 failed / 4 skipped.npm run buildpasses.npx tsc --noEmitreports 26 errors, all pre-existing (identical count on a stashed baseline), zero in files changed here.Fixes #1539.
Summary by CodeRabbit