Skip to content

test(app): add voice mode CDP audio check#1976

Merged
benjaminshafii merged 7 commits into
devfrom
voice-cdp-audio-test
May 27, 2026
Merged

test(app): add voice mode CDP audio check#1976
benjaminshafii merged 7 commits into
devfrom
voice-cdp-audio-test

Conversation

@benjaminshafii
Copy link
Copy Markdown
Member

Summary

  • add a CDP-based voice test runner with preflight, transcript, audio, and full modes
  • split Electron media permission handling out of main and add fake audio device support for tests
  • propagate OPENWORK_ELECTRON_FAKE_MEDIA through Daytona Electron startup scripts

Verification

  • node --check apps/app/scripts/voice-cdp.mjs
  • node --check apps/desktop/electron/main.mjs
  • node --check apps/desktop/electron/media-permissions.mjs
  • bash -n .devcontainer/start-daytona-electron.sh
  • bash -n .devcontainer/test-on-daytona.sh
  • pnpm --filter @openwork/app typecheck
  • OPENWORK_ELECTRON_FAKE_MEDIA=1 bash .devcontainer/test-on-daytona.sh voice-cdp-audio-test
  • daytona exec openwork-test-20260527-103606 -- "bash -lc 'cd /workspace && pnpm --filter @openwork/app test:voice-cdp -- --mode preflight --cdp-url http://127.0.0.1:9825 --require-audio-permission'"

Daytona preflight result: passed. Audio getUserMedia succeeded with fake media, video getUserMedia failed with NotAllowedError, Electron target and OpenWork control API were ready.

Notes

  • full audio mode still requires a session with Voice Mode actions registered and Realtime/OpenAI configuration
  • generated Linux audio mode needs ffmpeg and espeak-ng/espeak available in the sandbox

@vercel
Copy link
Copy Markdown
Contributor

vercel Bot commented May 27, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
openwork-app Ready Ready Preview, Comment May 27, 2026 6:11pm
openwork-den Ready Ready Preview, Comment May 27, 2026 6:11pm
openwork-den-worker-proxy Ready Ready Preview, Comment May 27, 2026 6:11pm
openwork-landing Ready Ready Preview, Comment, Open in v0 May 27, 2026 6:11pm

Copy link
Copy Markdown

@cubic-dev-ai cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 6 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="apps/app/scripts/voice-cdp.mjs">

<violation number="1" location="apps/app/scripts/voice-cdp.mjs:203">
P2: After connection, WebSocket error/close events don't reject pending requests, so the test runner hangs if the target drops. Add a `close` listener that rejects all outstanding promises.</violation>
</file>

Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic

Comment thread apps/app/scripts/voice-cdp.mjs Outdated
@benjaminshafii
Copy link
Copy Markdown
Member Author

End-to-end Daytona proof added and verified.

Latest verification command:
daytona exec openwork-test-20260527-103606 -- "bash -lc 'cd /workspace && pnpm --filter @openwork/app test:voice-cdp -- --mode audio --cdp-url http://127.0.0.1:9825 --text \"Open extension settings\" --expect-route settings/extensions'"

Result:

  • generated PCM16 speech audio via espeak-ng + ffmpeg
  • injected it through voice.inject_audio into OpenAI Realtime
  • Realtime understood the audio command and invoked OpenWork UI control
  • app navigated to #/workspace/ws_ade89c384ac9/settings/extensions
  • proof output included narration: Done: Open MCP and extension settings

Also added visible Voice Mode diagnostics in the panel for prod/mac failures:

  • Realtime connection state/error
  • microphone track state, enabled/disabled state, and system-muted state
  • status error surfaced directly in the panel

Screenshot proof from the resulting settings page: /var/folders/d9/xqhkvsp94rg0n0n523snqztm0000gn/T/browser-screenshot-1779904565922.png

Copy link
Copy Markdown

@cubic-dev-ai cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 2 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic

Comment thread apps/app/scripts/voice-cdp.mjs Outdated
@benjaminshafii benjaminshafii merged commit 72a7f73 into dev May 27, 2026
6 of 10 checks passed
benjaminshafii added a commit that referenced this pull request May 27, 2026
* test(app): add voice mode CDP audio check

* fix(app): wait for voice CDP target readiness

* fix(desktop): require explicit audio media permission

* fix(desktop): keep fake media behind permission handler

* test(app): capture voice CDP proof

* fix(desktop): request mac microphone access for voice

* fix(app): harden voice CDP runner
benjaminshafii added a commit that referenced this pull request May 27, 2026
* feat(app): combine browser and artifacts views and state

* ignore: update download stats 2026-05-27

* feat(app): use static grain art avatars (#1970)

* fix(den): restore org subscription TypeID prefix (#1974)

* fix(app): resolve grain art avatars in electron builds (#1975)

* test(app): add voice mode CDP audio check (#1976)

* test(app): add voice mode CDP audio check

* fix(app): wait for voice CDP target readiness

* fix(desktop): require explicit audio media permission

* fix(desktop): keep fake media behind permission handler

* test(app): capture voice CDP proof

* fix(desktop): request mac microphone access for voice

* fix(app): harden voice CDP runner

---------

Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: ben <ben@prologe.io>
Co-authored-by: Source Open <gh2@mcadam.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant