Skip to content

feat: dictation via Deepgram Flux with composer-local UX - #32

Merged
Juliusolsson05 merged 2 commits into
mainfrom
feat/dictation-flux
Apr 29, 2026
Merged

Juliusolsson05 merged 2 commits into
mainfrom
feat/dictation-flux

Conversation

@Juliusolsson05

Copy link
Copy Markdown
Owner

Summary

Holding a configured hotkey (default: bare Fn on macOS) records into the focused TileLeaf composer. Live transcript previews while you talk; on release the final, STT-tagged text replaces the preview as a normal editable draft. Multi-turn safe — pausing mid-recording does not lose earlier sentences.

What changed

Dictation feature (commit f8f3a9c)

  • Native Fn capture. native/macos-hotkey-helper is a tiny Swift CGEventTap. Compiled-on-first-run, hashed by source so repeat boots are instant. Chromium does not deliver bare-Fn to renderer keydown and Electron's globalShortcut rejects it as "not an accelerator", so a side-channel was the only path. Streams hotkey-down/up JSON over stdout to main.
  • Single dispatch registry. Per-hook IPC subscriptions silently dropped every press until a composer was focused. Now one shared registry owns the IPC, picks the focused composer (or most-recently-focused as fallback) at press time, and remembers the chosen target across press → release so a quick tap can never strand a recorder.
  • Recorder-led socket lifecycle. MediaRecorder starts immediately. The Deepgram WebSocket only opens after MIN_HOLD_TO_TRANSCRIBE_MS = 180ms so accidental Fn taps never open and immediately cancel a handshake. Audio queues locally; on WS open, the queue is drained before publishing recording.id so concurrent dataavailable handlers cannot interleave with the flush and corrupt WebM order.
  • Live preview chokepoint. Every interim/commit/restore write goes through one writeInput(next, source) so the composer's draft is always derivable from a single point.
  • Microphone pre-warm. macOS's first getUserMedia per app boot is a multi-second cold start. We fire a one-shot warm-up the moment any composer hook becomes enabled and immediately stop the tracks. Subsequent presses are instant. Pre-warm failure is non-fatal.
  • Build glue. electron.vite.config.ts aliases agent-voice-dictation to its dist and externalises ws (Vite otherwise inlines bufferUtil in a shape where mask is not a function and the first audio chunk crashes main). Matching path in tsconfig.node.json.

LSP destroyed-stream fix (commit e51c1cf)

Unrelated to dictation; the change was in the same working tree so it ships here. LSP notifications were calling connection.sendNotification directly and throwing ERR_STREAM_DESTROYED when a language server died mid-flight, leaving syntax highlighting stuck in an error state. Now tracks a closed flag per server, routes notifications through sendNotificationIfOpen that no-ops on a destroyed stdin, and the semantic-tokens read path returns null on stream-destroyed so the editor falls back to lexer highlighting cleanly.

Notes on the journey

This branch is the result of a long debug session that fixed several intertwined bugs:

  1. Spurious unmount cancel — the unmount effect's deps were unstable callbacks, so React fired the cleanup on every render and cancelled the WS within ~1 frame of opening. Fix: prop callbacks behind refs, true unmount effect with [] deps.
  2. Out-of-order chunks — recording.id was published before the queue drain, so late dataavailable handlers raced the flush. Fix: drain entirely BEFORE assigning recording.id.
  3. 30s "no Fn after launch" — every TileLeaf hook gated on its own focus, so on cold boot with no focused composer every press was a silent no-op. Fix: shared registry with most-recently-focused fallback.
  4. Cold mic = stuck "starting" — getUserMedia first-call took 4+ seconds on macOS; users released before recording began. Fix: pre-warm at boot.
  5. Multi-turn audio loss — fixed in agent-voice-dictation@cd0f87f (cumulative interim) and @06cb096 (combined final + interim).

Test plan

  • npm run dev boots cleanly; helper logs mac helper ready for "Fn" within ~1s.
  • First Fn press is instant; live transcript appears in the focused composer.
  • Speak, pause >1s, speak again, release: both sentences land in the commit (multi-turn safety from the package fix).
  • Quick Fn tap (<180ms) does nothing user-visible — no orphaned WS, no stuck "listening".
  • Press Fn, click another tile mid-record, release: recording stops cleanly in the original tile.
  • Disable dictation in Settings → next press is a no-op; re-enable → works again.
  • LSP regression: open a TS file, kill typescript-language-server externally — editor falls back to lexer highlighting instead of erroring.

Out of scope / follow-ups

  • Packaging (electron-builder) currently does not ship native/** under the asar, so a packaged build would fail to find the Swift source. Dictation in dev only until that's wired.
  • /tmp chunk dump is opt-in via CC_SHELL_DICTATION_DUMP=1; off by default.
  • Verbose per-chunk traces gated on CC_SHELL_DICTATION_VERBOSE=1; off by default.

🤖 Generated with Claude Code

Juliusolsson05 and others added 2 commits April 28, 2026 23:01
Wires the agent-voice-dictation streaming provider into the composer.
Holding the configured hotkey (default: bare Fn on macOS) records into
the focused TileLeaf's input, with live preview during recording and a
single STT-tagged commit at release. Settings own the binding; main
is a passive mirror.

Architecture:

  - Native CGEventTap helper (Swift, compiled and cached on first run
    per source hash) captures bare-Fn cleanly. Chromium does not deliver
    Fn to renderer keydown and Electron's globalShortcut treats it as
    "not an accelerator", so a native side-channel is the only reliable
    path. The helper streams hotkey-down/up JSON over stdout to main.

  - Single shared dispatch registry. Each useComposerDictation hook
    publishes a small handle; the registry owns ONE IPC subscription
    and picks the focused composer (or the most-recently-focused one as
    fallback) at press time. Earlier per-hook subscriptions all gated
    on their own focus, so a fresh launch with no focused composer
    silently dropped every press until the user clicked into a tile.

  - Recorder-led socket lifecycle. The MediaRecorder starts immediately;
    the Deepgram WebSocket opens only after MIN_HOLD_TO_TRANSCRIBE_MS
    (180 ms) so accidental Fn taps don't open and immediately cancel a
    handshake. Audio queues locally during the wait. When the WS opens
    we drain the queue BEFORE publishing recording.id so concurrent
    dataavailable handlers cannot interleave and corrupt the WebM stream
    order.

  - Live preview. Every interim transcript event from the package
    rewrites the composer's draft to "baseInput + ' ' + cumulative
    text"; on commit, the wrapped final text replaces the preview.
    The preview/commit/restore writes go through one chokepoint so the
    composer's actual draft state is always derivable from one place.

  - Microphone pre-warm. macOS's first getUserMedia per app boot is a
    multi-second cold start; we kick a one-shot warm-up the moment any
    composer hook becomes enabled and immediately stop the tracks. The
    next real press is then instant. Failure is non-fatal.

Build glue:

  - electron.vite.config.ts aliases the agent-voice-dictation package
    to its dist build and externalises `ws` (Vite's bundling otherwise
    inlines bufferUtil in a shape where mask is not a function and the
    first audio chunk crashes main).

  - tsconfig.node.json mirrors the alias so type-only imports resolve
    against the package's d.ts surface.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The semantic-tokens request and document didOpen/Change/Close
notifications were calling `connection.sendRequest` /
`sendNotification` directly, which throws ERR_STREAM_DESTROYED when
the language server's stdin has been closed (server crashed, was
killed, or its connection was disposed mid-flight). Those throws
propagated as uncaught rejections and surfaced in the renderer as
broken syntax highlighting that survived until the next manual reload.

Track a `closed` flag per ServerRecord, set it on `process.on('error')`
and `process.on('exit')`, and route every notification through a
`sendNotificationIfOpen` helper that no-ops once the server is closed
or the stdin stream is destroyed. The semantic-tokens read path
catches ERR_STREAM_DESTROYED and returns null so the editor falls
back to lexer highlighting instead of getting stuck in an error
state.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@Juliusolsson05
Juliusolsson05 merged commit 3acc4c1 into main Apr 29, 2026
@Juliusolsson05
Juliusolsson05 deleted the feat/dictation-flux branch July 7, 2026 21:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant