Decoder hardening pack before production use on untrusted input (frame-size mismatch panics, silent short decodes, stereo-SILK channel loss)

[ekhodzitsky]

Decoder hardening pack before production use on untrusted input (frame-size mismatch panics, silent short decodes, stereo-SILK channel loss)

Constructive bundle of smaller robustness/API findings from differential + fuzz testing. Happy to split into separate issues if preferred. The most severe single bug (panic on valid SILK 40/60 ms packets) is filed separately; this pack covers everything else.

Summary

Fuzzing the decoder (4,631 packets — all 256 TOC bytes with minimal payloads, crafted 40/60 ms SILK, 3,000 random packets, real libopus packets with every truncation length; 46,310 decode calls total) plus differential testing against libopus 1.6.1 surfaced several robustness/API gaps. None of these require malformed Ogg containers — they are reachable through the public OpusDecoder::decode API alone.

1. Panics when the caller's frame_size doesn't match the packet — libopus returns an error code (OPUS_BUFFER_TOO_SMALL / OPUS_INVALID_PACKET) in these cases; it never crashes. Two confirmed panic sites: src/bands.rs:2346 (quant_all_bands) and src/bands.rs:2742 (denormalise_bands); a third site at src/quant_bands.rs:348 was observed in earlier fuzz iterations. With a correct frame size, the only panics observed in 9,262 calls were the SILK 40/60 ms workspace overflow filed as issue Hybrid mode (SILK+CELT) at 48kHz produces broken output - only 32% energy #2 (227 panics, 2.4% of calls — all the same root cause).
2. decode returns Ok(frame_size) unconditionally — not the number of samples actually written. In the SILK path a short decode leaves the tail of the caller's buffer untouched (stale data) while the caller is told the full frame was produced. There is no way to detect partially-decoded packets.
3. Stereo SILK decodes as mono-replicated — one channel of stereo content is silently lost. The code comments say "SILK only decodes channel 0 … replicate the mono samples to every channel" (src/lib.rs, SILK path), while libopus decodes SILK stereo (M/S) properly. Measured on a stereo file with different L/R speech: opus-rs output has L == R bit-for-bit, matches libopus's right channel only (corr 0.98), and is uncorrelated with the left channel (corr 0.10); whole-file SNR vs libopus: 2.9 dB.
4. Errors are &'static str — no error enum, so callers cannot distinguish "invalid packet" (skip frame) from "buffer too small" (caller bug) programmatically.

Environment

• opus-rs 0.1.23 (crates.io tarball, sha256 1cc978697f3ada0ad98b323b9d85e85c6b08d478830148636c19b6d5f2ac9b85, verified against the sparse index; used as a path dep with identical content)
• rustc 1.96.0, macOS aarch64 (Apple Silicon), release build
• ffmpeg 8.1.1 with libopus 1.6.1

Minimal reproductions

(a) frame_size = 0 on a valid hybrid packet → panic (any real 20 ms hybrid packet works; see "Reproduction assets" for how to extract one):

use opus_rs::OpusDecoder;
fn main() {
    let pkt = std::fs::read(std::env::args().nth(1).unwrap()).unwrap(); // repro05_hybrid_packet.bin (TOC 0x68)
    let mut dec = OpusDecoder::new(48000, 1).unwrap();
    let mut buf = [];
    let _ = dec.decode(&pkt, 0, &mut buf); // libopus: OPUS_BUFFER_TOO_SMALL
}

Backtrace (fresh, RUST_BACKTRACE=1):

thread 'main' panicked at deps/opus-rs-0.1.23/src/bands.rs:2346:29:
range start index 40 out of range for slice of length 0
   3: core::slice::index::SliceIndex<[T]>::index_mut
   5: opus_rs::bands::quant_all_bands          at src/bands.rs:2346:29
   6: opus_rs::celt::CeltDecoder::decode_impl_from_rc
   7: opus_rs::celt::CeltDecoder::decode_from_range_coder_with_band_range  at src/celt.rs:2441
   8: opus_rs::OpusDecoder::decode             at src/lib.rs:1182:34

(b) too-small frame_size on a valid multi-frame CELT packet → panic (the packet is a fuzz-found code-3 packet, TOC 0x97, hex-dumped inline below; its correct frame size is 2880, passing 240):

use opus_rs::OpusDecoder;
fn main() {
    let pkt = std::fs::read(std::env::args().nth(1).unwrap()).unwrap(); // repro05_bands2742_packet.bin
    let mut dec = OpusDecoder::new(48000, 2).unwrap();
    let mut buf = vec![0.0f32; 1920];
    let _ = dec.decode(&pkt, 240, &mut buf); // libopus: OPUS_BUFFER_TOO_SMALL
}

Backtrace (fresh):

thread 'main' panicked at deps/opus-rs-0.1.23/src/bands.rs:2742:32:
range start index 120 out of range for slice of length 80
   5: opus_rs::bands::denormalise_bands        at src/bands.rs:2742:32
   6: opus_rs::celt::CeltDecoder::decode_impl_from_rc
   7: opus_rs::celt::CeltDecoder::decode_from_range_coder_with_band_range  at src/celt.rs:2441
   8: opus_rs::OpusDecoder::decode             at src/lib.rs:1066:39

Note the CELT path does check output.len() in one place (Err("Output buffer too small")), but the check happens after several unchecked slices are already taken with the wrong geometry.

(c) stereo SILK channel loss: encode a stereo file with different content per channel at 16 kHz/24 kbps (forces SILK stereo, TOC 0x4c), decode with opus-rs and libopus, compare per channel:

ffmpeg -y -i left.wav -i right.wav -filter_complex "[0:a][1:a]join=inputs=2:channel_layout=stereo[a]" \
       -map "[a]" -c:a libopus -ar 16000 -b:a 24k silk_stereo.ogg
ffmpeg -y -c:a libopus -i silk_stereo.ogg -f f32le -acodec pcm_f32le -ar 48000 ref.f32

Measured (fresh): opus-rs output channels are bit-identical (L == R); corr vs libopus = 0.70 overall; the replicated channel matches only one of the two source channels.

Expected vs actual

• Expected: never panic on any input the API accepts — return Err with a distinguishable error kind, like opus_decode does. Return the actual number of decoded samples (or document the buffer contract and zero-fill unwritten regions). Decode stereo SILK per the spec (or reject stereo packets explicitly instead of silently dropping a channel).
• Actual: panics as above; Ok(frame_size) always; stereo SILK loses a channel; stringly-typed errors.

Impact

Servers decoding untrusted Ogg/Opus uploads must treat every panic as a remote DoS (a panic in safe Rust unwinds into the request/task and can take down the process depending on the runtime). The wrong-frame-size panics are caller-triggerable, but real callers compute frame sizes from untrusted TOCs and can easily get them wrong; libopus's contract (error codes for exactly these cases) exists precisely so untrusted input can't crash the host. The stereo-SILK behavior silently corrupts channel layout for low-bitrate stereo voice.

Reproduction assets

• Case (a): the hybrid packet comes from the issue-Higher complexity benchmark data #1-style encode (ffmpeg -c:a libopus -ar 48000 -ac 1 -b:a 32k); take the first audio packet with TOC 0x68 with any Ogg demuxer.

• Case (b): the fuzz-found 277-byte code-3 packet (TOC 0x97), inline as hex:

  9746a6d7500cc6dbd4f966b42d41bbe29db839b94cfed5d61b6f5701b554d24115172de968785e889524101cc6e782c3e20b6b9b4a7ab657f94a72b00339bd5e3430a6e188092f233d33b9e089886d1b2367f9983777b1fc5537af2632f4ff0e9afd9afdbefd98bbb0a9c47f31033569c3ca1dff4628e849856ef633751353f34e7260649837bd42a2afdcc3beb7347206b6472333db72b6e124a853e314427e6e2a58dc35f57c2e6820faa13c44b2e7bacdde274727f1b07d57876a440aaeb7bcc93e78ee9548426e2af48d36ca5d0f2fcb1c02f8dfe49de4eb468891ee6b5fdfc87751eb9269e90cead54ab45b57698bc2b1d69f38508c351b5fdd1ef8e13856d8d5fa15d18cad74ae7f616fc742baed4c329c43
  

• Case (c): the ffmpeg command in the text generates the stereo SILK file.

(All binary inputs are also kept as repro-05.zip; happy to attach or link on request.)