Repository navigation
feat: dictation via Deepgram Flux with composer-local UX - #32
Merged
Merged
Conversation
Wires the agent-voice-dictation streaming provider into the composer.
Holding the configured hotkey (default: bare Fn on macOS) records into
the focused TileLeaf's input, with live preview during recording and a
single STT-tagged commit at release. Settings own the binding; main
is a passive mirror.
Architecture:
- Native CGEventTap helper (Swift, compiled and cached on first run
per source hash) captures bare-Fn cleanly. Chromium does not deliver
Fn to renderer keydown and Electron's globalShortcut treats it as
"not an accelerator", so a native side-channel is the only reliable
path. The helper streams hotkey-down/up JSON over stdout to main.
- Single shared dispatch registry. Each useComposerDictation hook
publishes a small handle; the registry owns ONE IPC subscription
and picks the focused composer (or the most-recently-focused one as
fallback) at press time. Earlier per-hook subscriptions all gated
on their own focus, so a fresh launch with no focused composer
silently dropped every press until the user clicked into a tile.
- Recorder-led socket lifecycle. The MediaRecorder starts immediately;
the Deepgram WebSocket opens only after MIN_HOLD_TO_TRANSCRIBE_MS
(180 ms) so accidental Fn taps don't open and immediately cancel a
handshake. Audio queues locally during the wait. When the WS opens
we drain the queue BEFORE publishing recording.id so concurrent
dataavailable handlers cannot interleave and corrupt the WebM stream
order.
- Live preview. Every interim transcript event from the package
rewrites the composer's draft to "baseInput + ' ' + cumulative
text"; on commit, the wrapped final text replaces the preview.
The preview/commit/restore writes go through one chokepoint so the
composer's actual draft state is always derivable from one place.
- Microphone pre-warm. macOS's first getUserMedia per app boot is a
multi-second cold start; we kick a one-shot warm-up the moment any
composer hook becomes enabled and immediately stop the tracks. The
next real press is then instant. Failure is non-fatal.
Build glue:
- electron.vite.config.ts aliases the agent-voice-dictation package
to its dist build and externalises `ws` (Vite's bundling otherwise
inlines bufferUtil in a shape where mask is not a function and the
first audio chunk crashes main).
- tsconfig.node.json mirrors the alias so type-only imports resolve
against the package's d.ts surface.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The semantic-tokens request and document didOpen/Change/Close
notifications were calling `connection.sendRequest` /
`sendNotification` directly, which throws ERR_STREAM_DESTROYED when
the language server's stdin has been closed (server crashed, was
killed, or its connection was disposed mid-flight). Those throws
propagated as uncaught rejections and surfaced in the renderer as
broken syntax highlighting that survived until the next manual reload.
Track a `closed` flag per ServerRecord, set it on `process.on('error')`
and `process.on('exit')`, and route every notification through a
`sendNotificationIfOpen` helper that no-ops once the server is closed
or the stdin stream is destroyed. The semantic-tokens read path
catches ERR_STREAM_DESTROYED and returns null so the editor falls
back to lexer highlighting instead of getting stuck in an error
state.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Holding a configured hotkey (default: bare Fn on macOS) records into the focused TileLeaf composer. Live transcript previews while you talk; on release the final, STT-tagged text replaces the preview as a normal editable draft. Multi-turn safe — pausing mid-recording does not lose earlier sentences.
What changed
Dictation feature (commit
f8f3a9c)native/macos-hotkey-helperis a tiny Swift CGEventTap. Compiled-on-first-run, hashed by source so repeat boots are instant. Chromium does not deliver bare-Fn to renderer keydown and Electron'sglobalShortcutrejects it as "not an accelerator", so a side-channel was the only path. Streams hotkey-down/up JSON over stdout to main.MIN_HOLD_TO_TRANSCRIBE_MS = 180msso accidental Fn taps never open and immediately cancel a handshake. Audio queues locally; on WS open, the queue is drained before publishingrecording.idso concurrent dataavailable handlers cannot interleave with the flush and corrupt WebM order.writeInput(next, source)so the composer's draft is always derivable from a single point.getUserMediaper app boot is a multi-second cold start. We fire a one-shot warm-up the moment any composer hook becomes enabled and immediately stop the tracks. Subsequent presses are instant. Pre-warm failure is non-fatal.electron.vite.config.tsaliasesagent-voice-dictationto its dist and externalisesws(Vite otherwise inlines bufferUtil in a shape wheremask is not a functionand the first audio chunk crashes main). Matching path intsconfig.node.json.LSP destroyed-stream fix (commit
e51c1cf)Unrelated to dictation; the change was in the same working tree so it ships here. LSP notifications were calling
connection.sendNotificationdirectly and throwingERR_STREAM_DESTROYEDwhen a language server died mid-flight, leaving syntax highlighting stuck in an error state. Now tracks aclosedflag per server, routes notifications throughsendNotificationIfOpenthat no-ops on a destroyed stdin, and the semantic-tokens read path returns null on stream-destroyed so the editor falls back to lexer highlighting cleanly.Notes on the journey
This branch is the result of a long debug session that fixed several intertwined bugs:
[]deps.recording.idwas published before the queue drain, so late dataavailable handlers raced the flush. Fix: drain entirely BEFORE assigningrecording.id.getUserMediafirst-call took 4+ seconds on macOS; users released before recording began. Fix: pre-warm at boot.Test plan
npm run devboots cleanly; helper logsmac helper ready for "Fn"within ~1s.typescript-language-serverexternally — editor falls back to lexer highlighting instead of erroring.Out of scope / follow-ups
electron-builder) currently does not shipnative/**under the asar, so a packaged build would fail to find the Swift source. Dictation in dev only until that's wired./tmpchunk dump is opt-in viaCC_SHELL_DICTATION_DUMP=1; off by default.CC_SHELL_DICTATION_VERBOSE=1; off by default.🤖 Generated with Claude Code