Skip to content

Investigate and harden speech-to-text failure handling and diagnostics #243

Closed

Description

@Juliusolsson05

Status (quality-loop audit, 2026-09-25)

Landed: The cold-start audio-loss fix plus history, no-speech/missing-api-key outcomes (ipc/dictation.ts:280-287,477), and cancel on quit.

Remaining: Reason codes (other errors are free text, :672-676), and deadlines for connect, first-audio, final and insertion.

Acceptance: Every non-success outcome carries a reason code, and each of the four phases has a deadline (from recorded dictation-debug/*.dictation.jsonl).

Original report

Problem

Speech-to-text sometimes fails without enough visible or saved evidence to explain what happened. When dictation does not produce usable text, the user currently has too little information about whether the failure came from recording, permissions, local audio capture, network/API transport, Deepgram streaming, finalization, post-processing, or composer insertion.

This makes STT feel unreliable and hard to debug. We need to solidify the dictation pipeline so common edge cases are handled explicitly and every failure leaves actionable diagnostics.

Desired behavior

Dictation should either succeed or fail with a clear reason and enough debug context to investigate later.

Examples:

  • microphone permission denied or device missing
  • no audio / silence / too-short press
  • recorder start/stop races
  • pending audio chunks never flushed
  • Deepgram websocket open/close failures
  • Deepgram returns no transcript or partial-only transcript
  • network interruption / auth failure / timeout
  • final transcript exists but composer insertion fails
  • post-processing/polish fails after raw transcript succeeded
  • cancellation vs real failure is distinguishable

Investigation areas

Likely starting points:

  • packages/agent-voice-dictation/src/speech/deepgram/streaming.ts
  • src/main/ipc/dictation.ts
  • src/main/dictationJournal.ts
  • renderer dictation/composer integration and any saved dictation debug dumps

Check whether every phase emits structured debug events with correlation ids:

  • press/start requested
  • permission/device result
  • recorder started
  • audio chunks queued/sent
  • stream opened
  • transcript partial/final received
  • stop requested
  • flush/finalize completed
  • result inserted into composer
  • failure/cancel path completed

Acceptance criteria

  • Add a clear phase model for dictation/STT so UI and debug logs agree on what happened.
  • Every non-success path records a specific reason code and enough context to reproduce or categorize it.
  • Silent failures become visible in the UI and saved diagnostics.
  • Timeouts are explicit for websocket connect, first audio, final transcript, and composer insertion.
  • Cancellation/short-press discard is separated from error failures.
  • Add a focused regression or diagnostic harness for at least the highest-risk edge cases.
  • Saved debug logs include enough dictation/STT context to diagnose failures after the fact.

Notes

This is separate from improving transcription accuracy with Deepgram keyterms/dictation context (#240). This issue is about reliability, lifecycle handling, and diagnostics when STT fails or produces no usable output.

Activity

  1. Juliusolsson05 commented on Sep 27, 2026

    @Juliusolsson05
    OwnerAuthor

    PR #1340 delivers reason codes (one shared union; every recorded terminal shape now ends with exactly one coded OUTCOME row, and the user reads the code's sentence) plus deadlines for connect (10 s), first audio (2 s) and final (30 s), taken from the 148 recorded journals. Ruling on the insertion deadline: insertion is a synchronous draft write (recorded 3 / 11 / 21 ms), so a timer on it can never fire. Its non-delivery cases carry delivery.hidden-terminal / delivery.abandoned codes instead. The PR says Refs, not Fixes, so this issue stays open until that ruling is accepted, or an asynchronous insertion path appears that needs a real deadline.

  2. Juliusolsson05 commented on Sep 27, 2026

    @Juliusolsson05
    OwnerAuthor

    Closing (owner decision, 2026-09-27): delivered by #1340. Every dictation failure now carries a reason code, and the phase deadlines are in place: connect 10 s, first audio 2 s, final 30 s, terminal insertion 5 s (src/main/ipc/dictation.ts:509-517, dictationReasons.test.ts:144). Accepted ruling: composer insertion is synchronous (3–21 ms), so a deadline timer could never fire. Its failures are reported by reason code instead. Reopen when: a dictation failure shows no reason, or a phase hangs past its deadline.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    class:C3-silent-failureThe app knows it failed and does not saysev:P2Real bug with a workaroundtype:bugSomething works wrong

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions