Skip to content

Add MCP-managed dictation context for Deepgram keyterms #240

Open

Description

@Juliusolsson05

Problem

Agent Code wraps speech-to-text input in <stt ...> because dictation can contain transcription mistakes. That warning is useful, but it is reactive: the model sees a possibly-wrong transcript after Deepgram has already guessed.

Deepgram supports vocabulary/context biasing (keyterm for current Nova-3/Flux flows, and keywords with boosts for older model flows). Agent Code should expose a safe feedback loop where agents can inspect and improve the dictation context used for future speech-to-text.

Example failures this should address:

  • MCP being heard as unrelated words
  • pull request / PR being mangled
  • project names, branch names, issue numbers, provider names, and repo-local terms being misheard
  • casing/formatting knowledge such as Codex, Claude, Deepgram, OpenAI, GitHub, TypeScript

Proposed shape

Add an optional built-in MCP domain, tentatively dictation_context, that lets an active agent read and update a bounded dictation vocabulary/context store.

The MCP server should expose tools like:

  • dictation_context_read
    • returns currently active keyterms/keywords, formatting rules, model/provider mode, and limits
  • dictation_context_add_terms
    • adds short technical/domain terms with optional aliases and rationale
  • dictation_context_remove_terms
    • removes stale or bad terms
  • dictation_context_record_correction
    • records that a misheard phrase should map to a preferred term in future context
  • optionally dictation_context_suggest_from_workspace
    • proposes terms from repo names, branches, package names, open issues, and common Agent Code concepts

Deepgram integration

Current code path:

  • streaming Deepgram URL is built in packages/agent-voice-dictation/src/speech/deepgram/streaming.ts
  • Agent Code starts streaming in src/main/ipc/dictation.ts
  • current streaming URL only sets model

Desired behavior:

  • For Flux / Nova-3-compatible flows, append repeated keyterm=<term> query params.
  • For older Deepgram model flows, use keywords=<term>:<boost> only where that API is appropriate.
  • Batch fallback should receive equivalent context, not only the streaming preview.
  • Dictation debug logs should record which context version/hash was used, but must not leak API keys or full private audio.

Storage / safety constraints

  • Store context under Agent Code state, not inside project files by default.
  • Keep a small, bounded active list, for example top 100 terms.
  • Terms should be short words or noun phrases, not arbitrary prompt text.
  • Normalize duplicates and casing.
  • Track provenance: user-added, agent-added, inferred, timestamp, optional rationale.
  • Agents should be allowed to improve vocabulary, not inject broad behavioral instructions into dictation.
  • Provide a way to inspect and clean up stale terms.

Acceptance criteria

  • A built-in MCP domain exists for dictation context and can be enabled for selected agents.
  • Agents can read the active dictation vocabulary/context through MCP.
  • Agents can add/remove bounded terms through MCP with validation.
  • Deepgram streaming and batch transcription receive the active context.
  • Debug traces show the context version/hash and term count used for each dictation session.
  • Existing dictation works unchanged when no context terms are configured.

Related code

  • packages/agent-voice-dictation/src/speech/deepgram/streaming.ts
  • packages/agent-voice-dictation/src/speech/deepgram/index.ts
  • src/main/ipc/dictation.ts
  • src/mcp/runtime/createBuiltInMcpServer.ts
  • src/mcp/shared/types.ts

Activity

  1. Juliusolsson05 commented on Sep 27, 2026

    @Juliusolsson05
    OwnerAuthor

    Marked later (owner decision, 2026-09-27). A good idea, but not worth the work right now. The triage's suggested first step, if it's picked up: a user-edited keyterm list passed to both Deepgram streaming and batch. Letting agents edit it over MCP is a separate, later decision.

  2. added
    laterValid, but not worth the work right now; revisit later
    on Sep 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    class:C7-new-featureBug cluster of a recently shipped featurelaterValid, but not worth the work right now; revisit latertype:featureNew capability (out of quality-loop fix scope)

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions