Skip to content

Fix Cohere/MLX local decode loop on long recordings#1

Merged
Newarr merged 2 commits into
mainfrom
fix/cohere-mlx-decode-loop
May 11, 2026
Merged

Fix Cohere/MLX local decode loop on long recordings#1
Newarr merged 2 commits into
mainfrom
fix/cohere-mlx-decode-loop

Conversation

@Newarr

@Newarr Newarr commented May 11, 2026

Copy link
Copy Markdown
Owner

Summary

Local Cohere transcription collapsed into "I think that's what I'm hearing" ×12 on the first real meeting recording — 583 s of audio produced ~140 words while ElevenLabs cloud returned 1,790 words on the same file. Root cause was a four-part bug across the upstream mlx-audio-swift decode path and the Scribe wrapper forwarding unsafe defaults verbatim.

What changed

Upstream fork (Newarr/mlx-audio-swift@b8ec430, diverges from Blaizzy/mlx-audio-swift@7734cd1):

  1. Anti-loop backstop in CohereTranscribe.generateSingleChunk and generateStream — breaks the decode loop when the last 24 tokens contain ≤3 unique IDs.
  2. Sign-aware repetition penalty in the same two loops, ported from Qwen3ASR.swift.
  3. chunkedParameters now propagates repetitionPenalty and repetitionContextSize (it dropped them before).
  4. Removed the global remainingTokens accumulator in generate/generateStream multi-chunk paths — each chunk gets the full budget; effectiveMaxGenerationTokens already clamps.

Wrapper-side overrides (CohereMLXBackend):

  • inferenceChunkDurationSeconds=30 (model card documents 35 s training; upstream default 1200 s loops)
  • inferenceRepetitionPenalty=1.2, inferenceRepetitionContextSize=32
  • New NativeCohereMLXAdapter.makeGenerationParameters static helper, regression-tested

Safety net:

  • DegenerateOutputDetector — pure-Swift tri-gram density + unique-word fraction checks
  • Throws CohereMLXBackendError.degenerateOutput if the model still emits loop output; TranscriptionWorker.isTransient does not classify this, so it surfaces as a terminal failed transcript instead of a useless one

Verification

  • 365 unit tests pass (3 new regression guards added)
  • SCRIBE_RUN_MLX_INTEGRATION=1 integration test skipped by default; passes when manually run with model weights present
  • End-to-end check on the failing recording (~/Scribe/2026-05-11-1756/audio.m4a):
    • Before: 140 words, "I think that's what I'm hearing" ×12
    • After: 1,477 words in 35 s, matches ElevenLabs reference verbatim on the opening utterance ("Yeah, I think Ron spoke about this that the FA had come back and now. I just think we need to take it offline, Peter, and we'll just see what he'd covered on that.")

Test plan

  • swift test is clean (already verified)
  • scripts/dev-install.sh --build produces a signed app (already verified)
  • Open Scribe → Recordings → 2026-05-11-1756 → tap Retry → confirm new transcript has >1,000 words and no looped phrases
  • Smoke test a fresh recording on local engine to confirm no regression on short audio

Szymon Sypniewicz added 2 commits May 11, 2026 18:05
Fix Cohere/MLX local decode loop with chunkDuration override and dete…
969f1cb 
…ctor

Local Cohere transcription collapsed into "I think that's what I'm hearing"
×12 on the first real meeting recording (583s audio, 140 word output) while
ElevenLabs cloud returned 1790 words on the same file. Root cause was in the
upstream mlx-audio-swift (1200s default chunk, repetition penalty never
applied in the Cohere decoder, chunkedParameters dropped penalty config) plus
the Scribe wrapper forwarding every broken default verbatim.

This change:

- Pins mlx-audio-swift to Newarr/mlx-audio-swift@b8ec430 with four targeted
  patches: anti-loop backstop, sign-aware repetition penalty,
  chunkedParameters propagation, and per-chunk token budget. Patches are
  documented in the fork's README.

- Overrides upstream defaults at the wrapper layer
  (CohereMLXBackend.inferenceChunkDurationSeconds=30, repetitionPenalty=1.2,
  repetitionContextSize=32) via a new makeGenerationParameters static helper
  on NativeCohereMLXAdapter that is regression-tested.

- Adds DegenerateOutputDetector as a pure-Swift safety net (tri-gram density
  and unique-word fraction) that throws CohereMLXBackendError.degenerateOutput
  when the model still emits loop output. Worker classifies this as
  non-transient so the user sees a clear failed transcript rather than a
  useless one.

Tests cover the override regression guard, the detector, and the backend's
throw behavior. An env-gated integration test
(SCRIBE_RUN_MLX_INTEGRATION=1) runs the real MLX model on the failing
recording for end-to-end verification.

TranscriberApp/project.yml mirrors the Package.swift dependency swap so
xcodebuild + xcodegen produce a binary against the patched fork.
Sync Xcode project to Newarr/mlx-audio-swift fork
0773b53 
xcodegen regenerates project.pbxproj from project.yml, which now points
at Newarr/mlx-audio-swift@b8ec430 for the patched Cohere decode loop. Old
UUIDs for the package reference are reissued — no other behavioral change.
@Newarr Newarr merged commit 436c56b into main May 11, 2026
1 check failed
@Newarr Newarr deleted the fix/cohere-mlx-decode-loop branch May 11, 2026 17:39
@Newarr Newarr mentioned this pull request May 11, 2026
Merged
3 tasks
Newarr added a commit that referenced this pull request May 11, 2026
@Newarr
Simplify DegenerateOutputDetector and harden detector tests (#2)
ef178e8 
Follow-up cleanups on top of the Cohere/MLX decode-loop fix (PR #1):

- Hoist `.lowercased()` out of the tri-gram loop in
  `DegenerateOutputDetector.evaluate` (was lowercasing each word 3x per
  iteration plus once more for the unique-word check; now once up front).
- Drop the unreachable `if words.count >= 3` guard inside the `>= 30`
  branch.
- Trim bug-story narration from the type-level doc comments
  (`CohereMLXBackendError`, `DegenerateOutputDetector`) - the changelog
  belongs in the PR, not the source.
- Replace the `try?` chain on the fixture load in
  `testDegenerateOutputDetectorFlagsRepetitiveTranscripts` with `try` so
  a missing/broken fixture fails loudly instead of silently asserting on
  an empty string.
- Make the integration test audio path overridable via
  `SCRIBE_MLX_INTEGRATION_AUDIO` (the dated default path is kept for
  back-compat).

No behavior change. 365 tests still pass (1 integration test skipped).

Co-authored-by: Szymon Sypniewicz <szymonsypniewicz@szymons-mac.local>
@Newarr Newarr mentioned this pull request May 11, 2026
Merged
2 tasks
Newarr pushed a commit that referenced this pull request May 11, 2026
Address codex review of CohereMLXBackend degenerate-output detector
09257e9 
Two follow-ups to PR #1 raised by an independent codex review:

1. Privacy: drop `sample` from `CohereMLXBackendError.degenerateOutput`.
   `TranscriptionWorker` persists `String(describing: error)` into the
   failed `transcript.md` and `metadata.json` artifacts, so the 120-char
   transcript snippet that used to ride along on the error was landing in
   on-disk failure metadata. The sample now stays in the unified log at
   `.private` privacy level for live debugging only.

2. False-positive floor: require the dominant tri-gram to repeat at least
   5 times before firing, in addition to the >8% fraction check. At the
   30-word minimum, 3 occurrences of an ordinary tri-gram (e.g. "thank
   you very") cleared the fraction gate at 3/28 ≈ 11% and terminally
   failed otherwise-usable short transcripts. The observed real-world
   loop fires 12+ times so the new floor is still well within range.

New regression test guards the short-transcript-with-three-repeats case;
the existing throw test now also asserts that `String(describing:)` on
the error does NOT contain transcript content.
@Newarr Newarr mentioned this pull request May 11, 2026
Merged
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Newarr