Skip to content

fix(search): restore CJK recall for core observation search - #1553

Merged
rohitg00 merged 9 commits into
rohitg00:mainfrom
akushonkamen:fix/1539-cjk-recall
Nov 4, 2026
Merged

rohitg00 merged 9 commits into
rohitg00:mainfrom
akushonkamen:fix/1539-cjk-recall

Conversation

@akushonkamen

@akushonkamen akushonkamen commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Core search was near-zero-recall for CJK users (#1539): a query in the user's own language could fail to return the very record it produced. Four structural gaps compounded:

Root cause

  • The BM25 document (extractTerms, src/state/search-index.ts:338) never included source.userPrompt/source.assistantResponse, so the user's own words were unindexed — only an English title/narrative could match.
  • Query entity extraction (extractEntitiesFromQuery, src/functions/query-expansion.ts:133) relied on a capitalized-word rule that never fires for scripts without capitalization; a pure-CJK query yielded zero entities and the graph leg stayed silent.
  • Every vector-embedded text concatenated only title + narrative, leaving concepts out, so non-English concepts could not anchor records in the vector leg.
  • COMPRESSION_SYSTEM gave no language instruction, so compressed concepts could drift to English even for non-English sources.

Fix

  • extractTerms: index source.userPrompt (capped at 2000 chars) and a bounded prefix of source.assistantResponse (1000 chars) so a query in the user's language can hit the record it came from.
  • extractEntitiesFromQuery: emit segmented CJK runs (>= 2 chars) via the existing segmentCjk so the graph leg fires for non-English queries.
  • New shared observationEmbedText (title + narrative + concepts, src/state/memory-utils.ts:57) used by every embedding site — observe (restore + synthetic), compress, remember, evolve (src/functions/relations.ts:228), and both migrate-vector-index rebuild phases — so concepts anchor non-English content in the vector leg.
  • Follow-up: indexRecords (src/functions/search.ts:404, :413) and the keyword rebuild's backfill queue (:602, :633) — the shared channels for rebuildIndex, reconcileIndex, snapshot restore, replay and import — also embed through observationEmbedText (whitespace-trimmed, so records without concepts embed the exact text they did before). Without this, any cold-start rebuild or backfill would have re-embedded the whole store without concepts.
  • COMPRESSION_SYSTEM: keep concepts in the user's language; bilingual (original + English) when the observation mixes languages. This directive's effect requires a live model to observe and could not be asserted locally — flagged for the reporter to verify end-to-end.

Red-to-green tests live in four existing files (test/search-index.test.ts, test/query-expansion.test.ts, test/compression-validation.test.ts, test/vector-index-populate.test.ts; +8 cases) plus test/index-keyword-rebuild.test.ts (+8 cases) for the follow-up channels.

Validation

  • Red (baseline 36eb328, during development): timeout 300 npx vitest run test/search-index.test.ts test/query-expansion.test.ts test/compression-validation.test.ts test/vector-index-populate.test.ts → 8 failed (exactly the new CJK cases) / 71 passed.
  • Green (re-verified at this head before submission), same command → 79 passed / 0 failed.
  • Relevant subset (+relations+migrate+keyword-rebuild): 7 files, 103 passed / 0 failed.
  • Full suite timeout 600 npx vitest run --exclude test/integration.test.ts after npm run build (the integration test needs a live server on :3111 and is skipped per CONTRIBUTING) → 3,357 passed / 0 failed / 4 skipped.
  • npm run build passes. npx tsc --noEmit reports 26 errors, all pre-existing (identical count on a stashed baseline), zero in files changed here.

Fixes #1539.

Summary by CodeRabbit

  • Search Improvements
    • Search uses titles, narratives, and concepts when indexing memories and observations, including concepts in other languages.
    • Chinese, Japanese, and Korean query text can contribute searchable terms, including in mixed-language queries.
    • Search can find captured prompts, responses, and retained conversation answers. Indexed source text is limited to a set amount.
  • Content Processing
    • Compression guidance encourages retaining search terms in the original language, including for observations that mix languages.
  • Bug Fixes
    • Improved consistency when indexing and refreshing memories and observations for search.

Devin Review

Three structural gaps made core search near-zero-recall for CJK users
(rohitg00#1539): the BM25 document never included the user's own words, query
entity extraction was ASCII-only, and the vector-embedded text left
concepts out.

- extractTerms: index source.userPrompt (capped at 2000 chars) and a
  bounded prefix of source.assistantResponse (1000 chars) so a query in
  the user's language can hit the record it came from
- extractEntitiesFromQuery: emit segmented CJK runs (>= 2 chars) via the
  existing segmentCjk so the graph leg fires for non-English queries
- vector embed text: share observationEmbedText (title + narrative +
  concepts) across every embedding site — observe (restore + synthetic),
  compress, remember, evolve (relations), and both migrate-vector-index
  rebuild phases — so concepts anchor non-English content in the vector
  leg
- COMPRESSION_SYSTEM: keep concepts in the user's language, bilingual
  when the observation mixes languages

The prompt directive's effect needs a live model to observe; the other
three are covered by red-to-green tests in the search-index,
query-expansion, compression-validation and vector-index-populate test
files.

Fixes rohitg00#1539

Signed-off-by: akushonkamen <akushonkamen@163.com>
Review follow-up to 3166caa: indexRecords (search.ts:404, :413) and the
keyword rebuild's backfill queue (search.ts:602, :633) still embedded
title + narrative/content. Those are the shared channels for
rebuildIndex, reconcileIndex, snapshot restore, replay and import
(rebuildIndex clears the vector index first), and the backfill queue
feeds embedBackfillJobs — so any cold-start rebuild or backfill would
have re-embedded the whole store without concepts, including records
that live writes had already embedded with the concept-aware text. All
four sites now go through observationEmbedText (whitespace-trimmed, so
records without concepts embed the exact text they did before), which
also makes the previous commit's 'every embedding site' claim true in
fact: the migrate-vector-index rebuild it cited has no production
caller (no src/ importer), while these search.ts channels are the live
rebuild paths.

Regression tests assert the concept-bearing text on both channels:
indexRecords via a recording embedding provider, and the keyword
rebuild via the returned backfill jobs.

Part of rohitg00#1539

Signed-off-by: akushonkamen <akushonkamen@163.com>
@vercel

vercel Bot commented Oct 13, 2026

Copy link
Copy Markdown

@akushonkamen is attempting to deploy a commit to the rohitg00's projects Team on Vercel.

A member of the Team first needs to authorize it.

@coderabbitai

coderabbitai Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

Caution

Review failed

The pull request is closed.

Walkthrough

Embedding generation now includes concepts for memories and observations. Keyword indexing includes bounded text from selected source fields. Query entity extraction now segments CJK text.

Changes: Search recall

Layer / File(s) Summary
Concept-aware embedding inputs
src/state/memory-utils.ts, src/prompts/compression.ts, src/functions/{compress,migrate-vector-index,observe,relations,remember,search}.ts, test/{compression-validation,index-keyword-rebuild,vector-index-populate}.test.ts
observationEmbedText combines titles, concepts, and narratives. Embedding paths use the helper for observations and memories. The compression prompt specifies concepts in the source language and, for mixed-language observations, in both languages. Tests check concept inclusion in embedding inputs.
Raw source fields in keyword indexing
src/state/search-index.ts, test/{search-index,index-keyword-rebuild,tool-output-index-regression}.test.ts
Keyword extraction indexes up to 2,000 characters of source.userPrompt and 1,000 characters of source.assistantResponse and conversation source.toolOutput. Tests cover field selection, length caps, serialization, and keyword-index rebuilds.
CJK query entity extraction
src/functions/query-expansion.ts, test/query-expansion.test.ts
extractEntitiesFromQuery segments Han, Hiragana, Katakana, and Hangul runs and adds segments at least two characters long. Tests cover pure CJK and mixed CJK/ASCII queries.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Bug fix · Severity of issue fixed: Medium

Suggested reviewers: rohitg00

Merge Risk: Low · up to 2a53b

The changes are mergeable with bounded follow-up: existing vectors retain their older search behavior, unusually long edited titles can exclude concepts from embeddings, and bilingual prompt guidance needs a stronger regression assertion.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Linked Issues check Warning [#1539] Four objectives apply to this PR. Three are implemented at head. (1) Bounded raw-source indexing: extractTerms in src/state/search-index.ts indexes capped source.userPrompt (2,000 chars)… Extend the COMPRESSION_SYSTEM language directive in src/prompts/compression.ts to cover <subtitle> as well as <concepts>. Apply the same rule: use source-language text for non-English content, and use both original-language and Engl…
Passed Passed checks: 4 passed
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title "fix(search): restore CJK recall for core observation search" matches the main change. The PR adds CJK-aware indexing and query entity extraction, includes concepts in shared embedding text,…
Out of Scope Changes check Passed All changed areas trace to [#1539]. The source.toolOutput indexing is limited to bounded string output of conversation records. The issue names source.toolOutput as an unindexed raw field (gap 1…
Docstring Coverage Passed Docstring coverage is inconclusive: files from earlier reviews still need to be rechecked within the check's analysis limits. Not blocking.
Full details: Title check

Explanation

The title "fix(search): restore CJK recall for core observation search" matches the main change. The PR adds CJK-aware indexing and query entity extraction, includes concepts in shared embedding text, and instructs compression to keep source-language concepts. These changes all aim to restore CJK recall in core observation search, as linked issue #1539 requests. The title is specific and concise, and it follows the conventional commit style. The title does not mention the tool-output indexing change, but the PR objectives describe that change as a secondary contribution, so the title remains a fair summary.

Full details: Out of Scope Changes check

Explanation

All changed areas trace to [#1539]. The source.toolOutput indexing is limited to bounded string output of conversation records. The issue names source.toolOutput as an unindexed raw field (gap 1), so this change serves a stated objective. The observationEmbedText call-site updates in observe, compress, remember, evolve, migrate-vector-index, and src/functions/search.ts implement the vector-text objective. The src/prompts/compression.ts concept rule implements the language objective. The src/state/memory-utils.ts helper and the src/state/search-index.ts caps support these changes. Test files cover the changed behavior. The compact search preview work named in the PR objectives is not present in the whole-PR changed-file summary, so it does not count as out-of-scope work here. The full-suite failures reported by the author are not attributable to this diff from the available evidence.

Full details: Linked Issues check

Explanation

[#1539] Four objectives apply to this PR. Three are implemented at head. (1) Bounded raw-source indexing: extractTerms in src/state/search-index.ts indexes capped source.userPrompt (2,000 chars) and source.assistantResponse (1,000 chars). It also indexes capped string toolOutput for toolName: "conversation" records, which matches the issue's named source.toolOutput gap. (2) CJK query entities: extractEntitiesFromQuery in src/functions/query-expansion.ts emits segmentCjk runs of at least two characters. (3) Concepts in vector text: observationEmbedText in src/state/memory-utils.ts (title + narrative + concepts) is used at observe, compress, remember, evolve, migrate-vector-index, indexRecords, and the keyword-rebuild backfill. Tests cover these paths. (4) Compression language directive: src/prompts/compression.ts now requires original-language concepts, and both original-language and English concepts for mixed-language observations. The whole-PR diff for that file changes only the concepts rule (+1/-1). The issue's Suggested fix 2 states that concepts (and subtitle) must include search terms in the user's language. At head, the &lt;subtitle&gt; line in COMPRESSION_SYSTEM is unchanged and no test asserts a subtitle rule. This objective is unmet for subtitle. The live-model effect of the directive is a separate, non-automated check and does not affect this verdict.

Resolution

Extend the COMPRESSION_SYSTEM language directive in src/prompts/compression.ts to cover &lt;subtitle&gt; as well as &lt;concepts&gt;. Apply the same rule: use source-language text for non-English content, and use both original-language and English text for mixed-language observations. Add an automated prompt regression test in test/compression-validation.test.ts that asserts the subtitle language directive is present in COMPRESSION_SYSTEM. Live-model verification remains a separate manual check.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Tip

🎯 Customize your risk assessments

Guide how CodeRabbit assesses merge risk and security architecture risk. Highlight sensitive changes and explain your security boundaries using Merge Risk Instructions and Security Risk Instructions in your review settings. Security architecture risk assessments require the Advanced plan or above.

✅ Automatic approvals, with your rules

Let CodeRabbit approve pull requests once required threads are resolved, pre-merge checks pass, and your risk limits are met. Enable Automatic approvals in review settings, with overrides for teams or file paths.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

coderabbitai[bot]

This comment was marked as resolved.

@7889545

7889545 commented Oct 14, 2026

Copy link
Copy Markdown

I ran a live-model check of the compression-prompt directive in this PR before adopting it. Setup, results, and two small suggestions below.

Setup

6 real CJK observations from a production store (6-8k Han characters each, Chinese prose mixed with code identifiers). Same model, same temperature, only the concept rules swapped. 2 rounds per variant, so 12 compressions per variant.

  • B = the wording in this PR (include both the original-language and English terms).
  • C = our previous wording, which required both languages on every concept.
  • D = B plus one explicit sentence: emit each language as its own <concept> element.

Results

Concepts aggregated over the 6 observations x 2 rounds:

variant concepts both languages present CJK-only Latin-only mixed scripts in one <concept> comma-joined
B (this PR) 46 6/6 obs 15 11 20 0
C (both-languages required) 38 6/6 obs 2 4 32 22
D (B + one language per element) 65 6/6 obs 23 30 12 0

mixed = one <concept> element holds both Han and Latin characters. comma-joined = one <concept> element holds a comma-separated list, e.g. 配置表改进方案, config table improvement.

What the numbers say

  • The directive does what it claims: two-language coverage becomes 6/6 per observation, and CJK-only plus Latin-only coverage is preserved.
  • It does not say how the two languages should be distributed, so the model decides per element. Even without a both-languages requirement, 20 of 46 concepts in row B mix scripts in one element. Adding a must-be-bilingual rule makes it worse: 22 of 38 concepts in row C are comma-joined lists.
  • That matters here: observationEmbedText joins concepts with spaces, so a comma-joined element becomes one literal string such as 配置表改进方案, config table improvement, which matches neither a Han query nor an English one. It is a strictly worse token than either single-language concept would have been.

Two suggestions

1. Make "one language per <concept>" an explicit contract. One sentence is enough, for example: "Emit each language as its own <concept> element - never two languages inside one element, and never a comma-separated list." In variant D this drove comma-joined to 0 and produced 53 single-language concepts versus 15 in B, with two-language coverage unchanged at 6/6.

2. Document that existing data does not update on its own. This PR changes the write-side text (observationEmbedText) that compress, observe, remember, relations and search all consume, and it ships migrate-vector-index.ts plus changes to indexRecords / runKeywordRebuild. Those are operational steps, not automatic ones: vectors already stored were embedded from the old text, and existing records have no source.userPrompt in BM25. Without that note in the PR description, an upgrade looks complete after merge while recall over old records is still served from the old index.

Minor and optional: USER_PROMPT_TERM_CAP = 2000 and ASSISTANT_RESPONSE_TERM_CAP = 1000 look reasonable. One sentence on the index-size and rebuild-time basis would help reviewers judge them.

Happy to share the raw runs if that would help.

@7889545

7889545 commented Oct 17, 2026

Copy link
Copy Markdown

Heads-up: this PR now conflicts with main.

GitHub reports mergeable: false / mergeable_state: dirty for #1553 as of 2026-10-17. The branch was last pushed 2026-10-14; main is now at 6a182c4 (2026-10-16).

Files this PR targets that moved on main after your last push:

The other files this PR touches are unchanged on main since 2026-10-13 (src/functions/query-expansion.ts, src/prompts/compression.ts, src/state/memory-utils.ts), so none of the CJK work here is superseded — it just needs a rebase. src/functions/search.ts looks like the most likely conflict point, since both #1559 and this PR edit it.

I can re-run the before/after CJK recall measurement against a rebased branch on a production store if that helps review.

@rohitg00

Copy link
Copy Markdown
Owner

@akushonkamen thanks for tracking down the CJK recall gap in #1539. Main moved under this branch: #1546 changed failed compression recovery and #1569 changed structured capture redaction, and the branch now conflicts in src/functions/compress.ts and src/functions/observe.ts.

Please merge current main into the branch and keep the redaction and compression-recovery behavior from main intact while restoring the raw-field indexing. CI has not run on this fork PR yet; it will run after the update.

@akushonkamen

Copy link
Copy Markdown
Contributor Author

Merged the current main (14dfd6e) into this branch to resolve the conflicts the maintainer flagged.

Conflict resolution, per file:

  • src/functions/compress.ts and src/functions/observe.ts: took main's side in full — this keeps fix: retain and recover failed observation compression #1546 (failed-compression recovery: compression-state/compression-recovery, revision-guarded writes) and fix: preserve structured capture redaction #1569 (structured capture redaction) intact — then reapplied our CJK recall fix on top: the three vector-embed sites now build their text via observationEmbedText() (title + narrative + concepts) instead of title + " " + narrative.
  • test/compression-validation.test.ts: the two embed-text assertions from this PR are preserved, but adapted to main's new vectorIndexAddGuarded signature (it now takes a write-back validator callback, so the assertions expect one additional argument). The tested semantics are unchanged: concepts still reach the vector embed text.

Verification (run locally):

  • npx tsc --noEmit: 26 errors, byte-identical to the set produced by a clean checkout of origin/main — all pre-existing upstream, none introduced by the merge.
  • npm test (vitest, integration excluded): 3840 passed / 8 failed / 4 skipped. The 3 failing files (fs-watcher, observation-source, openclaw-connect) fail identically on a clean origin/main checkout, so they are pre-existing upstream failures, not regressions from this merge. test/compression-validation.test.ts passes (41/41).

devin-ai-integration[bot]

This comment was marked as resolved.

coderabbitai[bot]

This comment was marked as resolved.

@rohitg00

Copy link
Copy Markdown
Owner

fix coderabbit and devin suggestions @akushonkamen

@akushonkamen

Copy link
Copy Markdown
Contributor Author

Thanks @rohitg00. I'll address the open review findings on this branch: in src/functions/remember.ts, update observationEmbedText() so concepts are reserved space within the 16,000-character embedding limit instead of being truncated away when the narrative is long, and in src/state/search-index.ts, version the embedding text produced by the rebuild flow so a change invalidates existing vectors and queues a full backfill that doesn't skip those IDs. I'll also review the two Devin flags (including the one not posted inline) and fix anything still valid, keeping main's redaction and compression-recovery behavior intact. Will push a revision shortly.

@jmlheureux

Copy link
Copy Markdown
Contributor

Thanks for addressing the source-field indexing gaps in #1539.

We independently encountered a related issue in AgentMemory v0.9.31 while integrating OMP (Oh My Pi).

Our regression demonstrated that a retained final answer could appear in conversation replay but remain absent from keyword search when its relevant terms existed in source.toolOutput rather than the compressed narrative.

I see that this PR adds bounded indexing for source.userPrompt and source.assistantResponse, which addresses part of the underlying problem.

Would you consider including bounded indexing of source.toolOutput as well?

We have a tested local correction and a synthetic migration regression covering index rebuilding, repeated restart, and rollback. We can share the relevant reproduction or patch if useful.

Separately, we observed that compact search results may omit the answer text needed for useful retrieval. That appears to be a distinct search-preview concern and does not necessarily need to be addressed in this PR.

Transparency note: I used AI coding assistants (Codex and OMP) to help investigate this issue, develop the local patch, and run regression tests. The findings and test results were reviewed before sharing them here.

@akushonkamen

Copy link
Copy Markdown
Contributor Author

Thank you for the independent regression — a retained final answer being searchable only through the compressed narrative because its terms live in source.toolOutput is exactly the class of gap this PR's bounded indexing leaves open. Yes, please do share the reproduction (and the patch if you have one handy): if extending coverage to source.toolOutput stays within the same bounded-shape indexing approach, I'd rather fold it into this PR than leave the partial fix. Agreed the compact-preview omission is a separate concern — worth its own issue.

@jmlheureux

Copy link
Copy Markdown
Contributor

Thanks for the quick response and for being open to incorporating this into #1553.

I'll prepare the minimal reproduction and tested patch for source.toolOutput, review them against the current PR changes, and share them here.

Agreed on keeping the compact-preview issue separate. I'll follow up on that independently.

Signed-off-by: Jacques L'Heureux <jm_lheureux@yahoo.com>
@jmlheureux

Copy link
Copy Markdown
Contributor

Thanks for offering to fold this into #1553. The tested contribution is available here:

Commit: jmlheureux@f05d50e
Branch: https://github.com/jmlheureux/agentmemory/tree/codex/tool-output-pr1553

The patch is based on this PR's current head (8fe4f88) and adds bounded indexing of source.toolOutput for conversation records, with regression tests. Compact search previews are unchanged.

Validation: 41 focused tests passed; 3,855 full-suite tests passed with 4 skipped. Build and synthetic worker migration, rebuild, repeated restart, and rollback checks passed. TypeScript diagnostics matched the unchanged PR baseline.

The patch changes four files and is intended for incorporation into this existing PR. Codex and OMP assisted with implementation and validation; the results were reviewed before publication.

@akushonkamen

Copy link
Copy Markdown
Contributor Author

@jmlheureux Merged your branch (codex/tool-output-pr1553, f05d50e) into the PR branch as-is — your authorship is preserved in the history (merge commit 7421c64). Reviewed the patch: it's exactly the bounded-indexing shape we wanted (conversation toolOutput only, TOOL_OUTPUT_TERM_CAP = 1000 via slice), no out-of-scope changes, and the regression tests are solid.

Verified locally: all 41 indexing-related tests pass (search-index, index-keyword-rebuild, tool-output-index-regression). Full suite: 3840 passed / 4 skipped, with 15 failures that are pre-existing on our head 8fe4f88 in this environment (fs-watcher, openclaw-connect, Docker/timing-dependent suites) and unrelated to your changes — we confirmed the same tests fail on the pre-merge baseline. Thanks for the careful work and for including the boundary tests around toolInput exclusion and the term cap.

devin-ai-integration[bot]

This comment was marked as resolved.

coderabbitai[bot]

This comment was marked as resolved.

The embedding input is clipped at 16,000 characters. With concepts after
the narrative, a long memory lost its concepts, including the CJK terms
this change adds. Concepts now follow the title, and the parts are
joined without empty gaps.
coderabbitai[bot]

This comment was marked as resolved.

@rohitg00
rohitg00 merged commit 26fd312 into rohitg00:main Nov 4, 2026
12 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Core observation search drops CJK recall: raw fields are not indexed, query-entity extraction ignores CJK, and vector text excludes concepts

4 participants