Skip to content

feat: allow a cheaper model for compression than for summarization #899

Open

Description

@somedev7

Problem

All four LLM-backed pipelines — per-observation compression, session summarization, consolidation synthesis, and reflection — share a single model setting (OPENAI_MODEL / ANTHROPIC_MODEL / etc.). But their cost profiles and quality bars are very different.

Measured on one active day of real usage (several parallel Claude Code sessions, AGENTMEMORY_AUTO_COMPRESS=true, v0.9.27):

Pipeline LLM calls Share
Observation compression 1 908 ~87%
Graph extraction 147 ~7%
Session summarization 123 ~6%
Consolidation + reflect 37 ~2%

Compression is the bulk of spend, and — as the README's own "Cost-aware model selection" section puts it — it is "a summarization task with relatively loose quality bars". Summaries, consolidation output, and lessons, on the other hand, are exactly what gets injected into future sessions, so quality there matters most.

Today the only lever is downgrading the single shared model, which trades away quality on the 2-6% of calls where it matters to save money on the 87% where it doesn't. I couldn't split the workload because there is no per-task model setting (closest prior art: #613 documented cost tiers, #767 proposes a token budget — neither separates models per task).

The codebase is actually already shaped for this: MemoryProvider has distinct compress() and summarize() methods, and AgentMemoryConfig.compressionModel exists but always mirrors provider.model.

Proposed solution

A single provider-agnostic env var:

OPENAI_MODEL=your-main-model                  # summaries, consolidation, lessons
AGENTMEMORY_COMPRESS_MODEL=your-cheap-model   # per-observation compression etc.

I have a working implementation with tests and will open a PR.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions