[nicoloboschi]

Why

The decision model's request ceiling (HINDSIGHT_API_DECISION_MODEL_MAX_REQUEST_TOKENS) and the decision reranker's caps (2k-token query, 4k shared context, 4k per memory) were counted in HINDSIGHT_API_TOKENIZER_ENCODING, which is o200k_base by default. The endpoint counts its window with the served model's own tokenizer, and on mixed-script text the two drift apart a lot. Example: 東京 café naïve 🎉 repeated is ~1.6× more tokens by Jev's count. So a request sized to fit could still overflow the window (the halving in #5794 catches that, at the cost of extra round trips), and a memory "capped at 4k" could still be well over 4k for the model.

What

• New setting HINDSIGHT_API_DECISION_MODEL_TOKENIZER: the vocabulary both the ceiling and the reranker's caps are counted in.
  • Default for typesafe: jev, toktok's estimate of Jev's count.
  • Default for the other providers: HINDSIGHT_API_TOKENIZER_ENCODING, so nothing changes for them.
  • Set it when an endpoint serves a different model than the provider's default, e.g. qwen3.5.
  • An unknown name fails at startup, with the list of names that work.
• token_encoding:
  • truncate_to_tokens takes an optional encoding, like count_tokens and truncate_many_to_tokens already do.
  • jev is accepted as a named encoding. It has no ids, so it only goes through toktok's batch_count / truncate, never a loaded tokenizer, and it can't be the global encoding.
  • qwen3.5 is added to BUNDLED_ENCODINGS; the installed toktok ships it.
• DecisionModel._split and the decision reranker's caps count with the configured vocabulary.

A standalone file for the decision-rerank questions

New hindsight_api/engine/decision_rerank_interface.py. It holds everything that decides what the decision reranker sends and how it reads the answers:

• the state (query, plus the shared context)
• one Noul per candidate, with the wording
• the caps
• splitting into requests that fit the ceiling
• turning answers into position scores

It is meant to be used outside hindsight-api too. A service that reranks over the System One API can load this one file by path and send exactly what recall sends. To make that work:

• it imports the standard library only, nothing from hindsight_api;
• the caller passes the tokenizer in, as count and truncate functions;
• it builds and reads plain System One JSON and sends nothing itself.

DecisionModelReranker and DecisionModel._split now call it, so there is one copy of this logic. Behaviour is unchanged; the existing reranker and splitting tests pass untouched apart from where the constants are imported from. The module docstring says to treat its names and signatures as an API.

Behaviour change

typesafe users now count in Jev's vocabulary instead of o200k_base. On English text the counts are close. On mixed-script text Jev counts higher, so:

• a large pool may be split into one or two more requests up front, instead of overflowing and being halved
• very long memories are cut a little earlier

No change for openai or cloudflare, or when the setting is unset for them.

Tests

• test_token_encoding.py: every named encoding, jev included, counts and truncates, and the batched cut matches the single one. An unknown name lists what would work.
• test_decision_model_config.py:
  • typesafe defaults to jev, and the setting overrides it.
  • An unknown name fails at setup.
  • Providers with no known vocabulary use the global one.
  • A mixed-script pool that fits one request by o200k_base splits under jev, and every part fits by Jev's count.
• test_decision_reranker.py: the per-memory cap is counted in the client's vocabulary.
• test_decision_rerank_interface.py: the file loads by path in a fresh python -I, with no hindsight_api, toktok or pydantic imported; the caller's tokenizer sets the caps.

Full hindsight-api-slim suite run locally: the same set of tests fails before and after this change. All of them need LLM keys or local ML models that aren't set up locally, and none of them involve this code.