Skip to content

Updates for exporting video .mp4 type #438

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
nguyenducdongdev wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Updates for exporting video .mp4 type #438

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5a0d3c5
Updates for exporting video .mp4 type
nguyenducdongdev Apr 13, 2026
4f06cc8
Update README.md
nguyenducdongdev Apr 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
npm run build:win

D:\Development\openscreen\release


> [!WARNING]
> This is very much in beta and might be buggy here and there (but hope you have a good experience!).

Expand Down
52 changes: 23 additions & 29 deletions src/lib/exporter/audioEncoder.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,15 @@ const AUDIO_BITRATE = 128_000;
const DECODE_BACKPRESSURE_LIMIT = 20;
const MIN_SPEED_REGION_DELTA_MS = 0.0001;

export type TargetAudioCodec = "aac" | "opus";

export class AudioProcessor {
private cancelled = false;
private targetCodec: TargetAudioCodec;

constructor(targetCodec: TargetAudioCodec) {
this.targetCodec = targetCodec;
}

/**
* Audio export has two modes:
Expand Down Expand Up @@ -132,16 +139,24 @@ export class AudioProcessor {
const sampleRate = audioConfig.sampleRate || 48000;
const channels = audioConfig.numberOfChannels || 2;

const encodeConfig: AudioEncoderConfig = {
codec: "opus",
sampleRate,
numberOfChannels: channels,
bitrate: AUDIO_BITRATE,
};
const encodeConfig: AudioEncoderConfig =
this.targetCodec === "aac"
? {
codec: "mp4a.40.2",
sampleRate,
numberOfChannels: channels,
bitrate: AUDIO_BITRATE,
}
: {
codec: "opus",
sampleRate,
numberOfChannels: channels,
bitrate: AUDIO_BITRATE,
};

const encodeSupport = await AudioEncoder.isConfigSupported(encodeConfig);
if (!encodeSupport.supported) {
console.warn("[AudioProcessor] Opus encoding not supported, skipping audio");
console.warn("[AudioProcessor] Audio encoding not supported, skipping audio");
for (const frame of decodedFrames) frame.close();
return;
}
Expand Down Expand Up @@ -314,28 +329,7 @@ export class AudioProcessor {

try {
await demuxer.load(file);
const audioConfig = (await demuxer.getDecoderConfig("audio")) as AudioDecoderConfig;
const reader = (demuxer.read("audio") as ReadableStream<EncodedAudioChunk>).getReader();
let isFirstChunk = true;

try {
while (!this.cancelled) {
const { done, value: chunk } = await reader.read();
if (done || !chunk) break;
if (isFirstChunk) {
await muxer.addAudioChunk(chunk, { decoderConfig: audioConfig });
isFirstChunk = false;
} else {
await muxer.addAudioChunk(chunk);
}
}
} finally {
try {
await reader.cancel();
} catch {
/* reader already closed */
}
}
await this.processTrimOnlyAudio(demuxer, muxer, [], undefined);
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot Apr 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Avoid full-frame audio buffering in speed-edited exports

Routing muxRenderedAudioBlob through processTrimOnlyAudio changes this path from chunk streaming to full decode/re-encode buffering, because processTrimOnlyAudio accumulates all decoded AudioData frames before encoding/muxing. For long videos with speed regions, this can drive memory usage to uncompressed PCM scale and cause export stalls/OOMs that did not occur with the previous streamed chunk path.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot Apr 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

This makes speed-region exports much heavier in memory.

Line 332 now sends the rendered blob through processTrimOnlyAudio(), and that path buffers the whole track twice: once in decodedFrames and again in encodedChunks before muxing. for longer recordings that's a pretty easy way to balloon memory on top of the already-recorded blob. If re-encode is required here, it'd be safer to stream decode → encode → mux instead of materializing both arrays.

} finally {
try {
demuxer.destroy();
Expand Down
12 changes: 7 additions & 5 deletions src/lib/exporter/muxer.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,17 +8,19 @@ import {
} from "mediabunny";
import type { ExportConfig } from "./types";

export type Mp4AudioCodec = "aac" | "opus";

export class VideoMuxer {
private output: Output | null = null;
private videoSource: EncodedVideoPacketSource | null = null;
private audioSource: EncodedAudioPacketSource | null = null;
private hasAudio: boolean;
private audioCodec: Mp4AudioCodec | null;
private target: BufferTarget | null = null;
private config: ExportConfig;

constructor(config: ExportConfig, hasAudio = false) {
constructor(config: ExportConfig, audioCodec: Mp4AudioCodec | null = null) {
this.config = config;
this.hasAudio = hasAudio;
this.audioCodec = audioCodec;
}

async initialize(): Promise<void> {
Expand All @@ -39,8 +41,8 @@ export class VideoMuxer {
});

// Create audio source if needed
if (this.hasAudio) {
this.audioSource = new EncodedAudioPacketSource("opus");
if (this.audioCodec) {
this.audioSource = new EncodedAudioPacketSource(this.audioCodec);
this.output.addAudioTrack(this.audioSource);
}

Expand Down
51 changes: 47 additions & 4 deletions src/lib/exporter/videoExporter.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import type { WebDemuxer } from "web-demuxer";
import type {
AnnotationRegion,
CropRegion,
Expand All @@ -10,12 +11,13 @@ import type {
import { AsyncVideoFrameQueue } from "./asyncVideoFrameQueue";
import { AudioProcessor } from "./audioEncoder";
import { FrameRenderer } from "./frameRenderer";
import { VideoMuxer } from "./muxer";
import { type Mp4AudioCodec, VideoMuxer } from "./muxer";
import { StreamingVideoDecoder } from "./streamingDecoder";
import type { ExportConfig, ExportProgress, ExportResult } from "./types";

const ENCODER_STALL_TIMEOUT_MS = 15_000;
const ENCODER_FLUSH_TIMEOUT_MS = 20_000;
const AUDIO_BITRATE = 128_000;

interface VideoExporterConfig extends ExportConfig {
videoUrl: string;
Expand Down Expand Up @@ -66,6 +68,40 @@ export class VideoExporter {
this.config = config;
}

private async pickAudioCodec(demuxer: WebDemuxer | null): Promise<Mp4AudioCodec | null> {
if (!demuxer) return null;

let audioConfig: AudioDecoderConfig;
try {
audioConfig = (await demuxer.getDecoderConfig("audio")) as AudioDecoderConfig;
} catch {
return null;
}

const sampleRate = audioConfig.sampleRate || 48000;
const channels = audioConfig.numberOfChannels || 2;

const aacConfig: AudioEncoderConfig = {
codec: "mp4a.40.2",
sampleRate,
numberOfChannels: channels,
bitrate: AUDIO_BITRATE,
};
const aacSupport = await AudioEncoder.isConfigSupported(aacConfig);
if (aacSupport.supported) return "aac";

const opusConfig: AudioEncoderConfig = {
codec: "opus",
sampleRate,
numberOfChannels: channels,
bitrate: AUDIO_BITRATE,
};
const opusSupport = await AudioEncoder.isConfigSupported(opusConfig);
if (opusSupport.supported) return "opus";

return null;
}
Comment on lines +71 to +103
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot Apr 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
echo "video exporter codec probing:"
rg -n -C3 'pickAudioCodec|AudioEncoder\.isConfigSupported|return "aac"|return "opus"|return null' src/lib/exporter/videoExporter.ts

echo '---'
echo "audio processor decode/encode gates:"
rg -n -C3 'AudioDecoder\.isConfigSupported|Audio codec not supported|Audio encoding not supported' src/lib/exporter/audioEncoder.ts

echo '---'
echo "hard failure on missing audio codec:"
rg -n -C2 'Source has audio, but no supported audio encoder was found for MP4 export' src/lib/exporter/videoExporter.ts

Repository: siddharthvaddem/openscreen

Length of output: 2198

Add decoder gate to pickAudioCodec() and gracefully degrade on audio encode failures.

The codec picker at lines 71–103 only probes AudioEncoder.isConfigSupported() for AAC and Opus, but audioEncoder.ts separately checks AudioDecoder.isConfigSupported() at line 71. If source audio can be decoded (demuxer config parses fine) but can't be encoded to any supported format, pickAudioCodec() returns null at line 102. Then lines 193–196 throw a hard error instead of falling back to video-only export, which tanks the whole operation for files with oddball audio tracks.

Suggested fix:

  • Add AudioDecoder.isConfigSupported() check inside pickAudioCodec() before probing encoders
  • Change the throw at lines 193–196 to console.warn() so export continues without audio
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@src/lib/exporter/videoExporter.ts` around lines 71 - 103, pickAudioCodec
currently probes only AudioEncoder.isConfigSupported for AAC/Opus and returns
null if no encoder is supported; update pickAudioCodec to also call
AudioDecoder.isConfigSupported (using the decoded AudioDecoderConfig from
demuxer) before probing encoders so you gate encoding checks on decoder support,
and if neither decoder+encoder combos work return null; then change the hard
throw in the export flow that handles a null audio codec (the error raised when
audio cannot be encoded) to a console.warn and continue the export as video-only
so the operation degrades gracefully instead of failing completely.


async export(): Promise<ExportResult> {
const encoderPreferences = this.getEncoderPreferences();
let lastError: Error | null = null;
Expand Down Expand Up @@ -153,7 +189,14 @@ export class VideoExporter {
await this.initializeEncoder(encoderPreference);

const hasAudio = videoInfo.hasAudio;
const muxer = new VideoMuxer(this.config, hasAudio);
const audioCodec = hasAudio ? await this.pickAudioCodec(streamingDecoder.getDemuxer()) : null;
if (hasAudio && !audioCodec) {
throw new Error(
"Source has audio, but no supported audio encoder was found for MP4 export.",
);
Comment on lines +193 to +196
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot Apr 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Do not fail export when no audio encoder is available

This new hard failure blocks the entire MP4 export whenever the input has an audio stream but neither AAC nor Opus encoding is supported in the runtime. That is a functional regression for environments with limited WebCodecs audio support, where a video-only export is still possible and was previously handled by skipping audio in AudioProcessor.

Useful? React with 👍 / 👎.

}

const muxer = new VideoMuxer(this.config, audioCodec);
this.muxer = muxer;
await muxer.initialize();

Expand Down Expand Up @@ -335,11 +378,11 @@ export class VideoExporter {
phase: "finalizing",
});

if (hasAudio && !this.cancelled) {
if (audioCodec && !this.cancelled) {
const demuxer = streamingDecoder.getDemuxer();
if (demuxer) {
console.log("[VideoExporter] Processing audio track...");
this.audioProcessor = new AudioProcessor();
this.audioProcessor = new AudioProcessor(audioCodec);
await this.audioProcessor.process(
demuxer,
muxer,
Expand Down
1 change: 1 addition & 0 deletions vitest.config.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ export default defineConfig({
globals: true,
environment: "jsdom",
include: ["src/**/*.{test,spec}.{js,mjs,cjs,ts,mts,cts,jsx,tsx}"],
exclude: ["src/**/*.browser.test.{ts,tsx}"],
},
resolve: {
alias: {
Expand Down