Skip to content

readBody corrupts multi-byte UTF-8 on bodies larger than one chunk (breaks /import and large /observe with non-ASCII content) #930

Closed

Description

@alexstraf1

Summary

readBody() decodes each incoming TCP chunk independently with chunk.toString() and concatenates the resulting strings. When a multi-byte UTF-8 sequence (e.g. Cyrillic, CJK, emoji) straddles a chunk boundary, each half is decoded separately and the character is corrupted (replaced with U+FFFD). For request bodies larger than Node's default ~64 KB highWaterMark this is effectively guaranteed, so:

  • POST /agentmemory/import of a real export (>64 KB, non-ASCII content) fails with an opaque 400 and an empty body (the corrupted string fails JSON.parse).
  • Large POST /agentmemory/observe payloads with non-ASCII content are silently stored corrupted.

Affected code

dist/index.mjs ~L15236 and dist/src-MOg2zKJs.mjs ~L14745 (identical):

function readBody(req) {
  return new Promise((resolve, reject) => {
    let data = "";
    let size = 0;
    req.on("data", (chunk) => {
      size += chunk.length;
      if (size > 1e6) { req.destroy(); reject(new Error("too large")); return; }
      data += chunk.toString();          // <-- per-chunk decode corrupts split multi-byte chars
    });
    req.on("end", () => resolve(data));
    req.on("error", reject);
  });
}

Reproduction

  1. Run agentmemory (any platform); accumulate sessions whose observations contain Cyrillic/CJK so an export exceeds ~64 KB.
  2. GET /agentmemory/export → save to a file.
  3. POST /agentmemory/import with {"exportData": <the export>, "strategy": "skip"}.
  4. → 400 with an empty body. Splitting the same data into <40 KB chunks imports fine — confirming a chunk-boundary decode bug, not a payload-shape bug.

Fix

Buffer the raw chunks and decode once at the end:

function readBody(req) {
  return new Promise((resolve, reject) => {
    const chunks = [];
    let size = 0;
    req.on("data", (chunk) => {
      size += chunk.length;
      if (size > 1e6) { req.destroy(); reject(new Error("too large")); return; }
      chunks.push(chunk);
    });
    req.on("end", () => resolve(Buffer.concat(chunks).toString("utf8")));
    req.on("error", reject);
  });
}

Related, same area

  1. The 1 MB /import cap is self-defeating. readBody hard-rejects bodies >1e6, but /import consumes the output of /export, which exceeds 1 MB for any non-trivial memory. With no documented chunking, a user cannot round-trip their own export once it grows past ~1 MB. Consider a higher limit for /import, or a documented chunked-import path.
  2. Opaque 400. When the body fails to parse, the response has no body. Returning {"error":"invalid JSON"} (as other validation paths already do, e.g. the handler's own {"error":"exportData is required"}) would save a lot of debugging time.

Environment

  • agentmemory 0.9.27 (npm latest, published 2026-06-07), Node.js 24, Windows 11, native iii-engine 0.11.2.

Activity

  1. woshilaohei commented on Jun 14, 2026

    @woshilaohei

    Nice catch - UTF-8 corruption on chunk boundaries is a classic async read bug.

    If you need a memory server that handles Unicode correctly today, MindCore Memory MCP was built with this in mind (JSONL with fsync, no async read coruption): https://github.com/woshilaohei/mindcore-memory-mcp

    • pip install mindcore-memory - pure Python, no Docker
    • All I/O is sync with explicit fsync - no chunk boundary bugs
    • 132/132 tests passing

    Won't fix agentmemory but might unblock you if you need memory working reliably now.

  2. alexstraf1 commented on Jun 15, 2026

    @alexstraf1
    Author

    This doesn't need a different memory backend — it's a one-function bug in agentmemory's own readBody (src/viewer/server.ts). It does data += chunk.toString() per TCP chunk, so a multi-byte UTF-8 sequence (Cyrillic/CJK/emoji) split across a chunk boundary gets decoded in halves and corrupted (→ U+FFFD). Past Node's ~64 KB highWaterMark it's effectively guaranteed, which is exactly why POST /import of a real non-ASCII export returns an opaque 400.

    Buffer the raw Buffers and decode once:

       function readBody(req) {
         return new Promise((resolve, reject) => {
    -      let data = "";
    +      const chunks = [];
           let size = 0;
           req.on("data", (chunk) => {
             size += chunk.length;
             if (size > 1_000_000) { req.destroy(); reject(new Error("too large")); return; }
    -        data += chunk.toString();
    +        chunks.push(chunk);
           });
    -      req.on("end", () => resolve(data));
    +      req.on("end", () => resolve(Buffer.concat(chunks).toString("utf8")));
           req.on("error", reject);
         });
       }

    Verified by round-tripping a 1.7 MB Cyrillic export through export→import. @rohitg00 can lift this straight in. (Orthogonal: the same function's 1 MB cap also blocks /import of any non-trivial /export.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions