Skip to content

fix(core): tighten session title prompt - #57648

Merged
rekram1-node merged 2 commits into
devfrom
title-prompt
Oct 21, 2026
Merged

rekram1-node merged 2 commits into
devfrom
title-prompt

Conversation

@rekram1-node

@rekram1-node rekram1-node commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Session titles average ~6 words and some aren't titles at all. The title request sends the user's first message as a bare user turn, so on long, instruction-style prompts the small model answers or starts the task instead of naming it:

model first message title we stored
claude-haiku-5-5 long compaction-test question Here's how I see it. I don't have access to the …
gpt-6-luna transform.ts investigation We need title only. Need capture investigation empty content …
deepseek-v4.1-flash patch tool comparison I'll start by exploring the codebase to find both …
deepseek-v4.1-flash Cloudflare model IDs … for Cloudflare AI gateway</think>Anthropic model IDs …

The prompt's examples also teach filler endings (…investigation, …implementation, …review), and casing drifts between sentence and Title Case across models.

Change

 SessionTitle.generate
   text =
-    firstUser.text                              // initial title
-    "Original request: … Recent conversation: …" // regeneration
+    messagePrompt(firstUser.text)                // <message>…</message> + "Reply with only the title."
+    conversationPrompt("Original request: …")    // <conversation>…</conversation>, still within 8,000 chars
   llm.stream(system: PROMPT_TITLE, user: text)

PROMPT_TITLE is rewritten in plain prose: aim for three to five words, keep the detail that distinguishes the session (version, provider, comparison target) and drop generic words to make room, sentence case with identifiers kept verbatim, and the input is something to name, not answer. Examples now model short titles.

Before / after, same first message and model:

before after
Patch Tool Comparison: v1 vs Current Patch tool vs v1 apply_patch
Unit test for issue #22020 in session instruction handling Issue #22020 instruction test
Resolve dist build conflicts by regenerating build outputs during rebase Rebase dist build conflicts
Pre-release review of v2.0.32..9d90d89 Pre-release changelog and API audit
Available Tools Inquiry Available tools list

Benchmark

First prompts from 30 real sessions (8 short, 12 medium, 10 long), replayed on six OpenCode-provider models at their lowest reasoning variant (claude-haiku-5-5#none, gpt-5-nano#minimal, gpt-6-luna#none, deepseek-v4.1-flash#low, minimax-m3, gemini-3.5-flash-lite#minimal), 2 runs each = 360 titles per prompt.

prompt mean words p90 answered / leaked / empty Title Case
current 5.73 (5.98 incl. failures) 8 11 9
current + wrapped input 5.60 8 0 4
new prompt, raw input 4.36 6 12 8
this PR 4.51 6 0 0
  • Wrapping the input is what removes the failures; the new system prompt alone still let the model answer 12 times.
  • A blind ranking by two judges (claude-opus-5-5, gpt-6.1-sol) scored the new titles level with the current ones (average rank within 0.1, judges split on order). A tighter 3–4 word target was shorter still but ranked clearly worse because it dropped the distinguishing detail.
  • Regeneration was spot-checked on 6 real sessions of 14–93 messages across all six models: no answers or leaks.

Out of scope

  • Generated titles and /rename still publish the same session.renamed event, so the two can't be told apart afterward.
  • gpt-5-nano stays the wordiest model (5.6 words) and sometimes starts lowercase.

Verification

  • bun test test/session-runner.test.ts test/session-title.test.ts test/session-generate.test.ts in packages/core: 255 pass
  • bun test in packages/core: 6684 pass, 8 fail, all ShellTool bash process-substitution cases that fail on this macOS machine and pass on Linux CI; unrelated to this change
  • bun typecheck in packages/core: pass

@opencode-agent opencode-agent Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No problems found that should block this. I checked the request the title model receives on the base and with this change. The first message is now wrapped in <message> and followed by "Reply with only the title." Regeneration context is wrapped in <conversation> and still fits within 8,000 characters. The space left for recent messages can't go negative, since the original request is capped at 2,000 characters. session-title.test.ts passes (20/20) and packages/core typechecks. I couldn't confirm that real models stop answering, because there are no provider credentials here, so that rests on your benchmark.

Public API unchanged: only the built-in title prompt and the text sent to the model change. Title hooks still receive the request and now see the wrapped text in event.messages.

Fixes #54185
Related: #53173

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant