diff --git a/.github/workflows/fuzz.yml b/.github/workflows/fuzz.yml index 8ab87407..256be9f8 100644 --- a/.github/workflows/fuzz.yml +++ b/.github/workflows/fuzz.yml @@ -26,7 +26,11 @@ jobs: strategy: fail-fast: false matrix: - target: [asset_pack, trace_parse, wav, ui_document, ui_text, net_datagram, png_decode, json_parse, stl_read, ply_read] + # Every target `fuzz/Cargo.toml` declares, which a test in the + # stable workspace requires: a harness nothing runs is a claim + # nothing checks, and seven of these were declared, committed + # with corpora, and never once executed. + target: [asset_pack, trace_parse, wav, ui_document, ui_text, net_datagram, png_decode, json_parse, stl_read, ply_read, obj_read, mtl_read, blob_read, glb_read, accessor_view, gltf_read, data_uri_read] steps: - uses: actions/checkout@v4 - uses: dtolnay/rust-toolchain@nightly diff --git a/REFUSALS.md b/REFUSALS.md index f6fc0031..97348cfe 100644 --- a/REFUSALS.md +++ b/REFUSALS.md @@ -8,16 +8,17 @@ and answer it — because a parser tested only on the files it can already read is exactly the failure that fuzzing exists to catch, and it passes its own suite the whole time. -Thirteen readers in this tree already take bytes nobody here wrote. Their +Seventeen readers in this tree already take bytes nobody here wrote. Their refusals are the worked examples throughout, so what follows describes the house pattern rather than inventing one. **The second half was written before any reader here took geometry, and -four of the thirteen now do** — the STL, PLY, OBJ and blob readers in `crates/mesh`, built +seven of the seventeen now do** — the STL, PLY, OBJ and blob readers in `crates/mesh`, and the +three glTF layers (container, accessor, document) that stand behind one another, built against this list rather than against the handful of files that happened to be on somebody's disk, which is what the list was for. Where an entry below says there is no local precedent, check `crates/mesh` first: its -eleven `MeshError` variants answer "not this format at all", "too short +fifteen `MeshError` variants answer "not this format at all", "too short to hold its own header", "a declared count the bytes present cannot supply", "a value outside the format's own domain", "zero where zero has no meaning" (entry 12, which OBJ's one-based indices are exactly), @@ -57,17 +58,21 @@ is dead code that reads like safety. | JSON | `JsonErrorKind` | `crates/json/src/error.rs` | 29 | `json_parse` | 30 | | Deflate, inside PNG | `InflateError` | `crates/png/src/inflate.rs` | 7 | `png_decode` | — | | Mesh descriptor | `TargetError::Creation` | `crates/rhi/src/vk/mesh.rs` | 8, as strings | none | none | -| STL | `MeshError` | `crates/mesh/src/error.rs` | 11, shared | `stl_read` | 18 | -| PLY | `MeshError` | `crates/mesh/src/error.rs` | 11, shared | `ply_read` | 16 | -| OBJ | `MeshError` | `crates/mesh/src/error.rs` | 11, shared | `obj_read` | 16 | -| MTL | `MeshError` | `crates/mesh/src/error.rs` | 4 of the 11 | `mtl_read` | 13 | -| Mesh blob | `MeshError` | `crates/mesh/src/error.rs` | 8 of the 11 | `blob_read` | 16 | +| STL | `MeshError` | `crates/mesh/src/error.rs` | 6 of the 15 | `stl_read` | 18 | +| PLY | `MeshError` | `crates/mesh/src/error.rs` | 10 of the 15 | `ply_read` | 16 | +| OBJ | `MeshError` | `crates/mesh/src/error.rs` | 9 of the 15 | `obj_read` | 16 | +| MTL | `MeshError` | `crates/mesh/src/error.rs` | 4 of the 15 | `mtl_read` | 13 | +| Mesh blob | `MeshError` | `crates/mesh/src/error.rs` | 8 of the 15 | `blob_read` | 16 | +| glTF container | `GlbError` | `crates/mesh/src/glb.rs` | 11 | `glb_read` | 18 | +| Accessor | `AccessorError` | `crates/mesh/src/accessor.rs` | 14 | `accessor_view` | 26 | +| glTF document | `GltfError` | `crates/mesh/src/gltf.rs` | 9 | `gltf_read` | 18 | +| Data URI | `DataUriError` | `crates/mesh/src/data_uri.rs` | 7 | `data_uri_read` | 41 | **The MTL row is the shortest in this table, and that is the honest number rather than a gap.** A material library indexes nothing, declares no counts and multiplies nothing, so four refusals is the whole of what can go wrong in it, and its census writes a sentence for each of the -seven it cannot reach. **A reader with few answers needs a tighter +eleven it cannot reach. **A reader with few answers needs a tighter corpus floor, not the same one**, which is why its gate takes one seed of slack where the others take two. @@ -81,6 +86,32 @@ what deleted a refusal from the STL reader that no input could produce**: it demands a sentence per variant per reader, and a sentence that cannot be written truthfully is a finding. +**Three of the four newest rows read one format between them, and each +has its own error type rather than a share of one.** That is the opposite +of what the five mesh readers do, and the reason is that they are layers +rather than formats: a container fault, a document fault and an accessor +fault send a caller to three different places, and folding them together +would put "not a container" in every other reader's census. The document +reader's type wraps the other two — along with the JSON reader's and +the geometry vocabulary's — so a caller that wants one answer gets +the layer's name and the inner refusal's own numbers one call away. + +**The fourth is not part of that stack yet.** The data URI reader is a +decoder for payloads a document may embed, and nothing calls it: the +glTF reader still refuses every buffer but the container's own chunk, +so the row describes a reader that is complete, attacked and unwired. + +**The data URI row has the highest floor of any mesh reader and one of +the smallest error types**, which is not a contradiction: its seeds are +short, so a corpus that covers every rule is cheap, and a rule it stopped +covering would be invisible in a smaller one. (The highest floor in the +table is the input trace's hundred; seven refusals ties the deflate row.) + +**Three censuses here list nothing as unreachable** — the container, +the document and the data URI — and the reason is the same for all +three: each owns its error type outright, so there is no other reader's +refusal in it to be out of reach. + The mesh-descriptor row is the one to read before writing an importer that touches the GPU. Its refusals are formatted strings rather than variants — the shape the rest of this document argues against — and it @@ -718,6 +749,17 @@ effect. `UnsetPatchBits`, `TraceErrorKind::DuplicateHeaderKey`, `HeaderFieldOutOfOrder`, `HeaderAfterEvents` and `ByteOrderMark`. +**The base64 decoder is the newest instance, and it is the one that shows +the category is not about file formats.** `QQ==` and `QR==` decode to the +same single byte: the last four bits of the second character reach no +output byte, so an encoder writes them as zero and a lenient decoder +ignores whatever is there. `DataUriError::NonCanonical { at, bits }` +refuses the second spelling, and the fuzz target holds the line from the +other side — everything the decoder accepts is re-encoded and must come +back character for character. **A decoder that tolerated those bits would +pass every test that decodes a payload written by an encoder**, which is +exactly the shape of bug this entry exists for. + `PoolNotCanonical` is the most complete instance and worth reading in full: scanning every table entry in document order, each pooled index must first appear exactly when the count of already-seen entries equals it, and diff --git a/crates/mesh/Cargo.toml b/crates/mesh/Cargo.toml index de67e21f..47d4ec3e 100644 --- a/crates/mesh/Cargo.toml +++ b/crates/mesh/Cargo.toml @@ -12,7 +12,7 @@ readme = "README.md" homepage.workspace = true [package.metadata.renew] -purpose = "Turn the bytes of a mesh file this engine did not write into validated geometry, refusing by name everything it cannot vouch for, and write that geometry back out in the one format this repository owns." +purpose = "Turn the bytes of a mesh file this engine did not write into validated geometry, decoding the payloads such a file may embed, refusing by name everything it cannot vouch for, and write that geometry back out in the one format this repository owns." maturity = "bootstrap" core = false extension_points = [] diff --git a/crates/mesh/README.md b/crates/mesh/README.md index 9066ad5a..39531d6a 100644 --- a/crates/mesh/README.md +++ b/crates/mesh/README.md @@ -281,6 +281,60 @@ boundaries, so their element size is not the product of their parts, and they carry inverse bind matrices — which is skinning, which this reads nothing of. They are unrepresentable rather than accepted and mis-sized. +## The document, which is where the indices are + +`gltf::read` takes the bytes of a binary glTF and returns geometry. It is +the only layer here that knows the format has a document at all: the +container hands back two byte strings and parses neither, the accessor +layer is arithmetic over a range, and this is where the numbers that +drive both come from. + +**A document is a handful of parallel arrays and a great many indices +into them.** A primitive names an accessor by number, an accessor names a +buffer view by number, a view names a buffer by number. A number naming a +row that is not there is the commonest thing wrong with a hand-edited or +truncated document, so it has one refusal that carries the table, the +index, and how many rows there were. + +**Its refusals name the layer that failed**, not just the fault: a +`GltfError` says whether the container, the document, an accessor or the +geometry objected, and the inner refusal's own numbers are one call away +through the value. A caller that only wants geometry can go +through `format::detect` and then `Format::read`, which wraps the same +answer in `MeshError`. + +The scene walk is an explicit stack with a visited mark checked before +children are pushed, so a document whose nodes point at each other is a +refusal rather than a walk that never ends. + +## `data:` URIs, and why a decoder is strict about spelling + +`data_uri::read` turns a URI whose payload *is* the resource back into +bytes — RFC 2397, base64 — and does nothing else. It does not know what +the bytes are for and never touches the filesystem, so a URI naming a +second file is not its to refuse, because it is not its to fetch. + +**The rule it is built on is that a difference which cannot change an +output byte is forgiven, and one that can is refused.** The `;base64` +marker is read in any letter case. Whitespace, the URL-safe alphabet, a +payload that is not whole four-character groups, padding anywhere but the +end — all refused, and the three that are about one character name +its offset. + +**The one that is easy to miss is the last group's unused bits.** `QQ==` +and `QR==` would both decode to the single byte `A`: the last four bits +of the second character reach no output byte. A decoder that ignores them +lets two different texts name one resource, which is the same trade the +container refuses when it insists its declared length is exactly the +file's length rather than at most it. Refusing the second spelling keeps +text and bytes one to one — and the fuzz target holds that line from the +other side, re-encoding everything the decoder accepts and demanding the +input back character for character. + +The media type is reported and never judged. Which types are acceptable +is a fact about what the caller is reading, and belongs where that is +known. + ## Manifest `Cargo.toml` is authoritative for maturity, core status, dependencies and diff --git a/crates/mesh/examples/make_data_uri_corpus.rs b/crates/mesh/examples/make_data_uri_corpus.rs new file mode 100644 index 00000000..be617362 --- /dev/null +++ b/crates/mesh/examples/make_data_uri_corpus.rs @@ -0,0 +1,255 @@ +//! Write the `data:` URI seed corpus. +//! +//! **Every seed here is text this program wrote.** A URI carrying a real +//! resource would carry somebody's model or image with it; these carry +//! counting patterns and short words. +//! +//! A fuzzer finds `data:` quickly — fixed bytes at offset zero are what a +//! coverage-guided search is best at. What it does not find on its own is +//! **a payload that decodes**: four characters from a 64-character +//! alphabet, correctly padded, with the unused bits of the final group +//! zeroed. Random mutation produces a refusal essentially every time, so +//! without seeds that decode, the whole second half of the reader — the +//! bit assembly, the padding arithmetic, the canonical check — is reached +//! by luck rather than by search. +//! +//! The seeds below therefore come in pairs: for each rule, one text that +//! satisfies it and one that misses it by a single character. **A +//! mutation's starting point sits on both sides of every rule.** +//! +//! Run when the corpus needs regenerating: +//! +//! ```text +//! cargo run -p renew-mesh --example make_data_uri_corpus +//! ``` +//! +//! Existing files are left alone. The fuzzer adds its own finds to this +//! directory over time, and this program must never delete them. + +// The crate bans filesystem access because the library never touches a +// file -- a caller that reads one owns it, and owns the bound on reading +// it. This program is that caller: writing the corpus is its whole job. +#![allow(clippy::disallowed_methods)] +// And the path-type ban with it: the crate takes bytes, never a path. +#![allow(clippy::disallowed_types)] + +use std::path::PathBuf; +use std::process::ExitCode; + +/// The media type a buffer's payload is required to declare, which is +/// also the one a reader of these seeds will be checking for. +const BUFFER: &str = "application/octet-stream"; + +// The encoder is `crates/mesh/tests/shared/base64_encode.rs`, included +// here and by four other targets, so a seed can be described by the +// bytes it should decode to rather than by a string somebody typed. +#[path = "../tests/shared/base64_encode.rs"] +mod base64_encode; +use base64_encode::encode; + +/// A URI with the buffer media type and the given payload text. +fn buffer(payload: &str) -> String { + format!("data:{BUFFER};base64,{payload}") +} + +/// Seeds that decode, which is what the interesting half of the reader +/// needs in order to be reached at all. +fn readable_seeds() -> Vec<(String, String)> { + let mut seeds = vec![ + // **One seed per padding case**, because the three cases run + // different arithmetic and only one of them has no canonical + // check to make. + ("no-padding".to_owned(), buffer(&encode(b"foo"))), + ("one-pad".to_owned(), buffer(&encode(b"fo"))), + ("two-pads".to_owned(), buffer(&encode(b"f"))), + // Every alphabet entry, including `+` and `/`, which a payload + // built from words never reaches. + ( + "every-byte-value".to_owned(), + buffer(&encode(&(0..=255).collect::>())), + ), + // Nothing at all, which is legal here and a question for + // whoever wanted the resource. + ("empty-payload".to_owned(), buffer("")), + // The other media type a buffer may declare, so a reader + // checking for one name is not the only path exercised. + ( + "gltf-buffer-type".to_owned(), + format!("data:application/gltf-buffer;base64,{}", encode(b"foo")), + ), + // A type with a parameter after it, and a parameter with no + // type: the two shapes that make the split at the first `;` + // worth having. + ( + "type-with-parameter".to_owned(), + format!("data:text/plain;charset=UTF-8;base64,{}", encode(b"hello!")), + ), + ( + "parameter-without-type".to_owned(), + format!("data:;charset=UTF-8;base64,{}", encode(b"hello!")), + ), + ( + "no-type-at-all".to_owned(), + format!("data:;base64,{}", encode(b"abc")), + ), + // The marker in a case no exporter writes, which is the one + // leniency this reader allows and therefore the one that could + // be removed without a seed here. + ( + "marker-in-capitals".to_owned(), + format!("data:{BUFFER};BASE64,{}", encode(b"foo")), + ), + // Long enough that a mutation lands in the middle of a group + // rather than always at an edge. + ( + "long-payload".to_owned(), + buffer(&encode( + &(0..96_u32) + .map(|n| u8::try_from(n * 5 % 251).unwrap_or_default()) + .collect::>(), + )), + ), + ]; + + // A payload for each length modulo three, so the final group's + // arithmetic is entered from every state. + for len in 1_usize..=9 { + let bytes: Vec = (0..len) + .map(|n| u8::try_from(n * 37 % 256).unwrap_or_default()) + .collect(); + seeds.push((format!("length-{len}"), buffer(&encode(&bytes)))); + } + seeds +} + +/// Seeds that are refused, one per rule, each missing by one character. +fn refused_seeds() -> Vec<(String, String)> { + let good = encode(b"foo"); + vec![ + // Not a URI of this kind at all: the relative path a document + // uses to name a second file. + ("relative-path".to_owned(), "geometry.bin".to_owned()), + ( + "absolute-url".to_owned(), + "https://example.invalid/a.bin".to_owned(), + ), + // Everything before the payload, and no payload. + ("no-comma".to_owned(), format!("data:{BUFFER};base64")), + // The percent-encoded spelling, which RFC 2397 allows and this + // reader does not. + ( + "percent-encoded".to_owned(), + format!("data:{BUFFER},%00%01"), + ), + // `base64` as part of the type rather than as a parameter. + ( + "marker-without-semicolon".to_owned(), + "data:base64,Zm9v".to_owned(), + ), + // A character outside the alphabet, and the two that belong to + // the other alphabet. + ("space-in-payload".to_owned(), buffer("Zm v")), + ("newline-in-payload".to_owned(), buffer("Zm\nv")), + ("url-safe-dash".to_owned(), buffer("Zm9-")), + ("url-safe-underscore".to_owned(), buffer("Zm9_")), + // Lengths that are not whole groups, on both sides of one. + ("three-characters".to_owned(), buffer("Zm9")), + ("five-characters".to_owned(), buffer("Zm9vZ")), + // Padding where padding cannot be. + ("padding-mid-payload".to_owned(), buffer("Zg==Zg==")), + ("padding-at-the-front".to_owned(), buffer("=m9v")), + ("four-pads".to_owned(), buffer("Zm9v====")), + ("digit-after-padding".to_owned(), buffer("Zm=v")), + // **The rule a lenient decoder drops.** `QR==` would decode to + // the same single byte as `QQ==`; the bits that differ reach no + // output byte. + ("stray-bits-two-pads".to_owned(), buffer("QR==")), + ("stray-bits-one-pad".to_owned(), buffer("Zm9=")), + // A near miss on the scheme, which is what a truncated or + // reassembled document looks like. + ( + "scheme-misspelt".to_owned(), + format!("dat:{BUFFER};base64,{good}"), + ), + ("scheme-truncated".to_owned(), "data".to_owned()), + // A wide character where the scheme's fifth byte would be: no + // character boundary there, and a reader that sliced instead of + // comparing bytes would end the process rather than refuse. + ( + "wide-character-at-the-scheme".to_owned(), + "dat\u{e9}foo".to_owned(), + ), + ( + "wide-character-before-the-comma".to_owned(), + "data:text/\u{e9}\u{e9}\u{e9}\u{e9},Zm9v".to_owned(), + ), + ] +} + +/// Seeds that are not text, which is the shape a `String` cannot hold. +/// +/// **Every other seed here is valid UTF-8, and that was a gap.** A URI +/// arrives as bytes; whoever reads it has to turn those into text first, +/// and the byte strings that cannot be turned into text cleanly are +/// exactly the ones where that conversion does something. A corpus of +/// nothing but well-formed text never replays that step. +fn raw_seeds() -> Vec<(String, Vec)> { + let head = format!("data:{BUFFER};base64,").into_bytes(); + let with = |tail: &[u8]| { + let mut bytes = head.clone(); + bytes.extend_from_slice(tail); + bytes + }; + vec![ + // A continuation byte with nothing to continue, in the payload. + ( + "lone-continuation-byte".to_owned(), + with(&[0xFF, b'm', b'9', b'v']), + ), + // A truncated multi-byte sequence, which is what a file cut in + // half in the wrong place looks like. + ("truncated-sequence".to_owned(), with(&[0xE2, 0x82])), + // And one before the comma, where the media type is read. + ( + "invalid-bytes-in-the-type".to_owned(), + b"data:text/\xC3;base64,Zm9v".to_vec(), + ), + ] +} + +fn main() -> ExitCode { + let dir = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../../fuzz/corpus/data_uri_read"); + if let Err(error) = std::fs::create_dir_all(&dir) { + eprintln!("cannot create {}: {error}", dir.display()); + return ExitCode::FAILURE; + } + + let mut written = 0usize; + let mut kept = 0usize; + let seeds = readable_seeds() + .into_iter() + .chain(refused_seeds()) + .map(|(name, text)| (name, text.into_bytes())) + .chain(raw_seeds()); + for (name, text) in seeds { + let path = dir.join(format!("{name}.uri")); + if path.exists() { + kept += 1; + continue; + } + // **A generator that swallows a write failure and then reports + // success is worse than one that crashes**: the caller sees a + // count and believes the corpus is whole. + if let Err(error) = std::fs::write(&path, &text) { + eprintln!("cannot write {}: {error}", path.display()); + return ExitCode::FAILURE; + } + written += 1; + } + + println!( + "{written} written, {kept} already present, in {}", + dir.display() + ); + ExitCode::SUCCESS +} diff --git a/crates/mesh/src/data_uri.rs b/crates/mesh/src/data_uri.rs new file mode 100644 index 00000000..7d3318a3 --- /dev/null +++ b/crates/mesh/src/data_uri.rs @@ -0,0 +1,461 @@ +//! `data:` URIs, and the base64 their payloads are written in. +//! +//! A document that wants to be one file has to carry its binary +//! somewhere, and the way it does that is RFC 2397: a URI whose payload +//! *is* the resource, rather than a name for one somewhere else. **This +//! module turns such a URI back into bytes and does nothing else.** It +//! does not know what the bytes are for, does not know which media types +//! a caller will accept, and never touches the filesystem — a URI that +//! names a second file is not this module's to refuse, because it is not +//! this module's to fetch. +//! +//! # Strict about the payload, forgiving about its spelling +//! +//! The two are not in tension, they are the same rule applied twice: +//! **a difference that cannot change a single output byte is tolerated, +//! and a difference that can is refused.** +//! +//! Forgiven: the `;base64` marker in any letter case, because +//! `;BASE64` and `;base64` are the same instruction and neither one +//! decodes differently. +//! +//! Refused: an encoding that is not a whole number of four-character +//! groups; a character outside the alphabet, including whitespace, which +//! some decoders skip; padding anywhere but the end; and — the one that +//! is easy to miss — **a final group whose unused bits are not zero.** +//! +//! # Why the unused bits matter +//! +//! `QQ==` and `QR==` would both decode to the single byte `A`, and this +//! reader accepts only the first. The last four +//! bits of the second character are not part of any output byte, so an +//! encoder writes them as zero and a lenient decoder ignores whatever is +//! there. That makes the encoding **many-to-one: two documents that +//! differ in their bytes produce byte-identical resources.** A reader +//! that accepts both has quietly agreed that two different files are the +//! same file, which is the same trade the container refuses when it +//! insists its declared length is exactly the file's length rather than +//! at most it. Refusing the non-canonical spelling keeps text and bytes +//! one to one. +//! +//! # What the caller still has to decide +//! +//! [`DataUri::media_type`] is reported, never judged. Which types are +//! acceptable is a fact about what the caller is reading — a buffer, an +//! image, a font — and belongs where that is known. + +use core::fmt; + +/// The scheme, with its colon. +const SCHEME: &str = "data:"; + +/// The parameter that says the payload is base64 rather than +/// percent-encoded text. +const MARKER: &str = ";base64"; + +/// The padding character, which carries no bits. +const PAD: u8 = b'='; + +/// Characters per encoded group, and bytes per decoded one. +const GROUP: usize = 4; +const DECODED: usize = 3; + +/// Every way a `data:` URI can fail to yield bytes. +/// +/// **Closed on purpose**, as this crate's other refusals are: a caller +/// matching exhaustively should stop compiling when a refusal is added +/// rather than route a new one through a wildcard. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum DataUriError { + /// The text does not begin with `data:`. + /// + /// Usually not a fault at all — it is what a relative path to a + /// second file looks like from here. The caller knows whether it can + /// go and get that file; this module knows only that it cannot. + NotADataUri, + + /// No comma, so nothing marks where the payload begins. + NoPayload, + + /// The `;base64` marker is absent. + /// + /// **This reader's limit rather than the format's.** RFC 2397 also + /// permits a percent-encoded payload, and that spelling is legal in + /// every document that may use a `data:` URI at all. It is not + /// implemented here, and saying so is more useful than a complaint + /// that blames the document for a choice this reader made. + /// + /// **Named for the reader's gap rather than the payload's shape**, + /// which is why it is not called `NotBase64`: a percent-encoded + /// payload genuinely is not base64, and a program keying on that + /// name would read a conformant document as a malformed one. The + /// document reader one file over spells the same idea the same way. + Unsupported, + + /// A character that is not in the base64 alphabet. + /// + /// Whitespace lands here too. Some decoders skip it; skipping it + /// would mean two spellings of one payload, which is the thing this + /// module exists to avoid. + BadDigit { + /// The byte found. + byte: u8, + /// Its offset in the whole URI, so the caller can point at it. + at: usize, + }, + + /// The payload is not a whole number of four-character groups. + NotWholeGroups { + /// How many bytes the payload has. + /// + /// **Bytes, not characters.** This check runs before any + /// character is looked at, so a payload carrying anything wider + /// than ASCII is measured in bytes -- and in a module whose + /// whole argument turns on that distinction, the field had + /// better not blur it. + len: usize, + }, + + /// Padding somewhere padding cannot be. + /// + /// It is legal only as the last one or two characters of the last + /// group. Anywhere else it either pads a group that is not the end + /// or has a digit after it, and both mean the payload was assembled + /// from pieces rather than encoded. + BadPadding { + /// Where the `=` is, in the whole URI. + at: usize, + }, + + /// A padded final group whose unused bits are not zero. + /// + /// See this module's own documentation: the alternative is accepting + /// two different texts as one resource. + NonCanonical { + /// The character carrying the stray bits, in the whole URI. + at: usize, + /// What they were, so the message can show what was expected. + bits: u8, + }, +} + +impl DataUriError { + /// The variant's own name, for a caller acting on which refusal this + /// is rather than reading it. + /// + /// **A message is for a person and a name is for a program**, the + /// same trade this crate's other refusals make. + #[must_use] + pub const fn name(self) -> &'static str { + match self { + Self::NotADataUri => "NotADataUri", + Self::NoPayload => "NoPayload", + Self::Unsupported => "Unsupported", + Self::BadDigit { .. } => "BadDigit", + Self::NotWholeGroups { .. } => "NotWholeGroups", + Self::BadPadding { .. } => "BadPadding", + Self::NonCanonical { .. } => "NonCanonical", + } + } +} + +impl fmt::Display for DataUriError { + fn fmt(&self, out: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::NotADataUri => write!(out, "this does not begin with `{SCHEME}`"), + Self::NoPayload => write!(out, "there is no comma, so no payload begins"), + Self::Unsupported => write!( + out, + "the payload is not marked `{MARKER}`, and this reader decodes no other spelling" + ), + Self::BadDigit { byte, at } => write!( + out, + "byte {byte:#04x} at offset {at} is not a base64 character" + ), + Self::NotWholeGroups { len } => write!( + out, + "the payload is {len} bytes, which is not a whole number of {GROUP}" + ), + Self::BadPadding { at } => { + write!(out, "the padding at offset {at} is not at the end") + } + Self::NonCanonical { at, bits } => write!( + out, + "the character at offset {at} carries {bits:#04x} in bits no output byte uses, \ + and an encoder writes those as zero" + ), + } + } +} + +impl core::error::Error for DataUriError {} + +/// A decoded `data:` URI. +/// +/// The text fields borrow from the URI; only the payload is owned, +/// because decoding it is the one thing here that cannot be a view. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct DataUri<'a> { + /// The media type as written, or `""` when the URI omitted it. + /// + /// **Reported, never judged.** RFC 2397 says an omitted type means + /// `text/plain`, but saying so here would be inventing a claim the + /// document did not make; a caller that cares can apply the default + /// and a caller that requires an explicit type can see that there + /// was none. + pub media_type: &'a str, + + /// Whatever followed the media type, without the `;` that separated + /// it and without the `;base64` marker. `""` when there were none, + /// and separators *between* parameters are kept, so + /// `;charset=x;name=y` is reported as `charset=x;name=y`. + pub parameters: &'a str, + + /// The payload. + pub bytes: Vec, +} + +/// Whether this text is a `data:` URI at all. +/// +/// Cheap, and answers the question a caller actually has: whether the +/// resource is here or somewhere this crate will not go. +/// +/// **Compares bytes rather than characters**, which is not a +/// micro-optimisation: a URI beginning `data\u{e9}` has no character +/// boundary five bytes in -- the accented letter spans bytes four and +/// five -- and slicing a string there is a panic rather than a +/// mismatch. +#[must_use] +pub fn looks_like(uri: &str) -> bool { + uri.as_bytes() + .get(..SCHEME.len()) + .is_some_and(|head| head.eq_ignore_ascii_case(SCHEME.as_bytes())) +} + +/// Read a `data:` URI into its media type and its bytes. +/// +/// # Errors +/// +/// Every way the text can fail to be a base64 `data:` URI — see +/// [`DataUriError`]. The three that are about one character +/// ([`BadDigit`](DataUriError::BadDigit), +/// [`BadPadding`](DataUriError::BadPadding) and +/// [`NonCanonical`](DataUriError::NonCanonical)) carry its offset in the +/// whole URI; the rest are about the text as a whole and have no place +/// to point at. +pub fn read(uri: &str) -> Result, DataUriError> { + if !looks_like(uri) { + return Err(DataUriError::NotADataUri); + } + let comma = uri.find(',').ok_or(DataUriError::NoPayload)?; + + // Everything between the scheme and the comma describes the payload; + // everything after it is the payload. Both ends of this slice are + // character boundaries: the scheme was matched byte for byte above, + // and the comma came from a search of this very string. + let described = &uri[SCHEME.len()..comma]; + + // The marker is compared as bytes, and only then sliced off. Doing + // it the other way round would slice a string at whatever offset + // seven bytes from its end happens to be, which is a panic on any + // media type ending in a character wider than one byte. + let described = match described.len().checked_sub(MARKER.len()) { + Some(cut) if described.as_bytes()[cut..].eq_ignore_ascii_case(MARKER.as_bytes()) => { + &described[..cut] + } + _ => return Err(DataUriError::Unsupported), + }; + + // A type may be absent and parameters present -- `data:;charset=x` + // is a URI RFC 2397 spells out -- so the split is at the first `;` + // and either side may be empty. + let (media_type, parameters) = match described.find(';') { + Some(at) => (&described[..at], &described[at + 1..]), + // **The empty tail of this string, not an empty literal.** A + // `""` literal is a dangling `'static` pointer that borrows + // nothing, so a caller checking provenance -- and the fuzz + // target does, on every input -- would find a field the type + // says is a view into the URI and that is not one. Slicing at + // the string's own end is a character boundary by construction. + None => (described, &described[described.len()..]), + }; + + Ok(DataUri { + media_type, + parameters, + bytes: decode(uri.as_bytes(), comma + 1)?, + }) +} + +/// What [`TABLE`] holds for a byte that is not a base64 character. +/// +/// Outside the six-bit range every real entry occupies, so it cannot be +/// confused with a value. +const INVALID: u8 = 0xFF; + +/// The value of one base64 character, or [`INVALID`]. +const fn digit(byte: u8) -> u8 { + match byte { + b'A'..=b'Z' => byte - b'A', + b'a'..=b'z' => byte - b'a' + 26, + b'0'..=b'9' => byte - b'0' + 52, + b'+' => 62, + b'/' => 63, + _ => INVALID, + } +} + +/// Every byte's value, built at compile time from [`digit`]. +/// +/// **The alphabet is still written exactly once**, in the match above, +/// which is the reason to generate the table rather than type it out: a +/// hand-written 256-entry array is 256 chances to disagree with the +/// ranges it is meant to encode. +/// +/// The reason to have it at all is that the ranges cost a chain of +/// comparisons per character, and a payload is a whole mesh buffer -- +/// megabytes of text is ordinary here. A lookup is the textbook shape +/// for this, and measurement is what says so rather than instinct: the +/// match ran at roughly a quarter of the table's throughput on an eight +/// megabyte payload, with the error path ruled out as the cause by +/// measuring a sentinel-returning match at the same speed as the +/// `Option` one. +static TABLE: [u8; 256] = { + let mut table = [INVALID; 256]; + let mut byte = 0_usize; + while byte < 256 { + // `byte` is bounded by the loop, so the cast is exact. + #[expect( + clippy::cast_possible_truncation, + reason = "the loop bound is 256, which is every value of the type" + )] + { + table[byte] = digit(byte as u8); + } + byte += 1; + } + table +}; + +/// Decode the payload beginning at `start`. +/// +/// Offsets in every refusal are into the whole URI rather than into the +/// payload, because that is what a caller has in its hand. +fn decode(uri: &[u8], start: usize) -> Result, DataUriError> { + let payload = &uri[start..]; + if !payload.len().is_multiple_of(GROUP) { + return Err(DataUriError::NotWholeGroups { len: payload.len() }); + } + + let groups = payload.len() / GROUP; + let mut out = Vec::with_capacity(groups * DECODED); + for (group, chunk) in payload.as_chunks::().0.iter().enumerate() { + let last = group + 1 == groups; + let mut sextets = [0_u8; GROUP]; + let mut padding = 0_usize; + + for (offset, &byte) in chunk.iter().enumerate() { + let at = start + group * GROUP + offset; + if byte == PAD { + // Only the final group's last two places, and once a + // group is padding it stays padding to the end. + if !last || offset < GROUP - 2 { + return Err(DataUriError::BadPadding { at }); + } + padding += 1; + continue; + } + if padding != 0 { + return Err(DataUriError::BadPadding { at: at - padding }); + } + let value = TABLE[byte as usize]; + if value == INVALID { + return Err(DataUriError::BadDigit { byte, at }); + } + sextets[offset] = value; + } + + // Twenty-four bits in, three bytes out, and one fewer byte for + // each padding character. The bits the missing bytes would have + // used must be zero, which is what makes the encoding + // reversible. + let carrier = GROUP - 1 - padding; + let unused = match padding { + 1 => sextets[carrier] & 0b11, + 2 => sextets[carrier] & 0b1111, + _ => 0, + }; + if unused != 0 { + return Err(DataUriError::NonCanonical { + at: start + group * GROUP + carrier, + bits: unused, + }); + } + + let bits = (u32::from(sextets[0]) << 18) + | (u32::from(sextets[1]) << 12) + | (u32::from(sextets[2]) << 6) + | u32::from(sextets[3]); + for shift in 0..DECODED - padding { + #[expect( + clippy::cast_possible_truncation, + reason = "the shift leaves eight bits, which is what the cast keeps" + )] + out.push((bits >> (16 - shift * 8)) as u8); + } + } + Ok(out) +} + +#[cfg(test)] +mod tests { + use super::{INVALID, TABLE, digit}; + + /// **Every entry of the table is the match it was generated from.** + /// + /// The table exists so the decoder does one lookup per character + /// rather than a chain of comparisons, and it is built from `digit` + /// so the alphabet is written exactly once. This is what says the + /// generation actually worked -- and it is also the only thing that + /// runs `digit` outside const evaluation, which is worth knowing: + /// without it the function is compiled, used, and invisible to + /// anything that measures what ran. + #[test] + fn every_entry_agrees_with_the_match_it_came_from() { + for byte in 0..=u8::MAX { + assert_eq!( + TABLE[byte as usize], + digit(byte), + "the table and the match disagree about {byte:#04x}" + ); + } + } + + /// **The alphabet is a bijection onto the six-bit values.** + /// + /// Sixty-four characters, each with its own value, and every value + /// spoken for. That is what makes the encoding reversible, and it is + /// the property a typo in one of the five ranges would break -- + /// quietly, because a duplicated value still decodes and a missing + /// one is only reached by the payloads that happen to need it. + #[test] + fn sixty_four_characters_cover_every_six_bit_value_once() { + let mut seen = [false; 64]; + let mut characters = 0_usize; + for value in TABLE { + if value == INVALID { + continue; + } + characters += 1; + let index = value as usize; + assert!(index < 64, "{value} is not a six-bit value"); + assert!(!seen[index], "two characters both mean {value}"); + seen[index] = true; + } + assert_eq!(characters, 64, "base64 has sixty-four characters"); + assert!( + seen.iter().all(|&spoken_for| spoken_for), + "some six-bit value has no character to spell it" + ); + } +} diff --git a/crates/mesh/src/gltf.rs b/crates/mesh/src/gltf.rs index 30a96a7f..c432e6f3 100644 --- a/crates/mesh/src/gltf.rs +++ b/crates/mesh/src/gltf.rs @@ -20,10 +20,10 @@ //! A buffer may name a `uri`, which is a second file or an embedded //! payload. **This reader refuses both by name.** Reading a second file //! would mean opening one, which this crate does not do and says so -//! everywhere else; decoding an embedded one needs a decoder the tree -//! does not have. Neither is a silent limitation: a document that wants -//! either is told which, so the caller knows whether to convert the file -//! or to wait for a reader that can. +//! everywhere else; decoding an embedded one needs [`crate::data_uri`], +//! which this reader does not yet call. Neither is a silent limitation: +//! a document that wants either is refused, so the caller knows to +//! convert the file rather than wondering what it got. //! //! The one buffer this reads is the container's own binary chunk, which //! is how a self-contained binary glTF stores its geometry. diff --git a/crates/mesh/src/lib.rs b/crates/mesh/src/lib.rs index c46d752d..a19bbdf5 100644 --- a/crates/mesh/src/lib.rs +++ b/crates/mesh/src/lib.rs @@ -57,6 +57,7 @@ pub mod accessor; pub mod blob; +pub mod data_uri; mod error; pub mod format; pub mod glb; @@ -140,6 +141,7 @@ pub(crate) fn refuse_over_ceiling(have: usize, adding: usize) -> Result<(), Mesh } pub use accessor::{Accessor, AccessorError, BufferView, Component, Shape}; +pub use data_uri::{DataUri, DataUriError}; pub use error::MeshError; pub use glb::{Container, GlbError}; pub use primitive::{Mode, Primitive}; diff --git a/crates/mesh/tests/corpus_replay.rs b/crates/mesh/tests/corpus_replay.rs index 1901c7a3..61ad7622 100644 --- a/crates/mesh/tests/corpus_replay.rs +++ b/crates/mesh/tests/corpus_replay.rs @@ -27,7 +27,7 @@ use std::collections::BTreeSet; use std::path::PathBuf; use renew_mesh::accessor::Shape; -use renew_mesh::{blob, glb, gltf, mtl, obj, ply, stl}; +use renew_mesh::{blob, data_uri, glb, gltf, mtl, obj, ply, stl}; /// The committed corpus never shrinks below this many **distinct** /// inputs. @@ -1349,3 +1349,169 @@ fn gltf_census() { reached.len() ); } + +/// How many distinct `data:` URIs the corpus must still hold. +const DATA_URI_LOW_WATER: usize = 41; + +/// How many distinct outcomes those seeds must still reach. +/// +/// **Measured, not guessed** — `data_uri_census` below prints it. Eight +/// is every refusal this reader has plus success, which is the whole +/// vocabulary: unusually, nothing here is unreachable, because one reader +/// owns the enum and every variant has a seed. +const DATA_URI_DISTINCT_OUTCOMES: usize = 8; + +/// Every refusal, each of which a seed must provoke. +/// +/// The list is the enum. That is affordable here and not elsewhere: the +/// five geometry readers share one error type, so each of them names what +/// it cannot reach, while this reader can reach all of its own. +const DATA_URI_REQUIRED: [&str; 7] = [ + "NotADataUri", + "NoPayload", + "Unsupported", + "BadDigit", + "NotWholeGroups", + "BadPadding", + "NonCanonical", +]; + +// The encoder is the one the seeds were written with, shared rather than +// copied, so this gate cannot drift from the generator that fed it. +#[path = "shared/base64_encode.rs"] +mod base64_encode; + +fn data_uri_corpus_dir() -> PathBuf { + PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../../fuzz/corpus/data_uri_read") +} + +fn data_uri_corpus() -> Vec> { + let dir = data_uri_corpus_dir(); + let entries = std::fs::read_dir(&dir).unwrap_or_else(|error| { + panic!( + "the committed corpus at {} must exist: {error}", + dir.display() + ) + }); + entries + .map(|entry| { + let entry = entry.expect("corpus entries are readable"); + std::fs::read(entry.path()).expect("corpus files are readable") + }) + .collect() +} + +/// The answer a seed gets, as the name of the refusal or `Ok`. +/// +/// Bytes become text the way the fuzz target does it, so a seed replays +/// here exactly as it is mutated there. +fn data_uri_outcome(bytes: &[u8]) -> &'static str { + match data_uri::read(&String::from_utf8_lossy(bytes)) { + Ok(_) => "Ok", + Err(refusal) => refusal.name(), + } +} + +/// **Every committed URI answers, and everything it accepts re-encodes to +/// itself.** +/// +/// The second half is the claim that matters. A decoder that quietly +/// tolerated whitespace, a missing pad, or a stray bit in the final group +/// would still return bytes — and those bytes would re-encode to +/// something other than the text it was given, which is what this catches +/// and a "did it crash" replay would not. +#[test] +fn every_recorded_uri_answers_and_re_encodes_to_itself() { + for bytes in data_uri_corpus() { + let text = String::from_utf8_lossy(&bytes); + let Ok(read) = data_uri::read(&text) else { + continue; + }; + let payload = &text[text.find(',').expect("a URI that read has a comma") + 1..]; + assert_eq!( + base64_encode::encode(&read.bytes), + payload, + "{text} decoded to bytes that spell something else" + ); + } +} + +/// The URI corpus keeps its strength. +#[test] +fn the_data_uri_corpus_still_covers_what_it_was_recorded_to_cover() { + let inputs = data_uri_corpus(); + let distinct: BTreeSet> = inputs.iter().cloned().collect(); + assert!( + distinct.len() >= DATA_URI_LOW_WATER, + "the corpus holds {} distinct inputs and the floor is {DATA_URI_LOW_WATER}", + distinct.len() + ); + + let reached: BTreeSet<&'static str> = distinct + .iter() + .map(|bytes| data_uri_outcome(bytes)) + .collect(); + assert!( + reached.len() >= DATA_URI_DISTINCT_OUTCOMES, + "the corpus reaches {} distinct answers and the floor is {DATA_URI_DISTINCT_OUTCOMES}. \ + Reached: {reached:?}", + reached.len() + ); + for required in DATA_URI_REQUIRED { + assert!( + reached.contains(required), + "no committed seed reaches `{required}`, which is a refusal nothing is exercising. \ + Reached: {reached:?}" + ); + } + + // **Seeds that decode, in every padding state.** A corpus of nothing + // but refusals would leave the bit assembly and the canonical check + // reachable only by luck, and those are the parts where a wrong + // answer is silent rather than loud. + let readable: Vec<&Vec> = distinct + .iter() + .filter(|bytes| data_uri_outcome(bytes) == "Ok") + .collect(); + assert!( + readable.len() >= 15, + "the corpus needs URIs that decode, and holds {}", + readable.len() + ); + for pads in 0..=2_usize { + assert!( + readable.iter().any(|bytes| { + let text = String::from_utf8_lossy(bytes); + // **Counted in the payload, not in the whole URI.** + // A media type may carry an `=` of its own -- + // `charset=x` does -- and counting those satisfied + // this floor with seeds whose payloads hold no + // padding at all, so the floor could have been met + // by a corpus with no one-pad seed in it. + let Some(comma) = text.find(',') else { + return false; + }; + comma + 1 < text.len() + && text[comma + 1..] + .bytes() + .filter(|&byte| byte == b'=') + .count() + == pads + }), + "no seed that decodes has {pads} padding characters in its payload, and the three cases run \ + different arithmetic" + ); + } +} + +#[test] +#[ignore = "a census, not a gate: run it to update the numbers above"] +fn data_uri_census() { + let distinct: BTreeSet> = data_uri_corpus().into_iter().collect(); + let reached: BTreeSet<&'static str> = distinct.iter().map(|b| data_uri_outcome(b)).collect(); + println!( + "{} distinct inputs, {} outcomes: {reached:?}", + distinct.len(), + reached.len() + ); +} diff --git a/crates/mesh/tests/data_uri.rs b/crates/mesh/tests/data_uri.rs new file mode 100644 index 00000000..7428d7de --- /dev/null +++ b/crates/mesh/tests/data_uri.rs @@ -0,0 +1,526 @@ +//! `data:` URIs: what decodes, what does not, and where it says so. +//! +//! The refusals are the point. A decoder that only ever sees payloads +//! written by an encoder passes its own suite on the day it is written +//! and every day after, which is why most of what follows is input no +//! encoder would produce. + +// An integration test is its own crate, so the `allow-*-in-tests` +// settings in `clippy.toml` — which apply to `#[cfg(test)]` modules — do +// not reach it. A fixture this file built and then could not read back +// is a broken test rather than a condition to recover from. +#![allow(clippy::panic, clippy::expect_used, clippy::unwrap_used)] + +use renew_mesh::data_uri::{self, DataUriError}; + +// The encoder is `shared/base64_encode.rs`, included here and by four +// other targets. It is deliberately not the crate's: nothing in the +// engine writes a `data:` URI, and a round trip through one body of code +// proves only that the code agrees with itself. +#[path = "shared/base64_encode.rs"] +mod base64_encode; +use base64_encode::encode; + +/// The prefix every URI here shares, so offsets in the assertions below +/// are computed rather than counted by hand. +const HEAD: &str = "data:application/octet-stream;base64,"; + +fn bytes(uri: &str) -> Vec { + data_uri::read(uri) + .unwrap_or_else(|refused| panic!("{uri} was refused: {refused}")) + .bytes +} + +fn refused(uri: &str) -> DataUriError { + match data_uri::read(uri) { + Err(refusal) => refusal, + Ok(read) => panic!("{uri} decoded to {:?} and should not have", read.bytes), + } +} + +/// The vectors from RFC 4648, which cover every padding case there is. +#[test] +fn the_published_vectors_decode_to_their_bytes() { + for (encoded, plain) in [ + ("", ""), + ("Zg==", "f"), + ("Zm8=", "fo"), + ("Zm9v", "foo"), + ("Zm9vYg==", "foob"), + ("Zm9vYmE=", "fooba"), + ("Zm9vYmFy", "foobar"), + ] { + let uri = HEAD.to_owned() + encoded; + assert_eq!( + bytes(&uri), + plain.as_bytes(), + "`{encoded}` should have been `{plain}`" + ); + } +} + +/// Every byte value, in one payload, so no alphabet entry is untested. +#[test] +fn all_two_hundred_and_fifty_six_byte_values_survive_a_round_trip() { + let all: Vec = (0..=255).collect(); + let uri = HEAD.to_owned() + &encode(&all); + assert_eq!(bytes(&uri), all); +} + +#[test] +fn an_empty_payload_is_no_bytes_rather_than_a_refusal() { + // A URI may legitimately carry nothing. Whether nothing is + // acceptable is a question for whoever wanted the resource. + assert!(bytes(HEAD).is_empty()); +} + +#[test] +fn a_media_type_is_reported_and_its_parameters_kept_separate() { + let read = data_uri::read("data:image/png;quality=fine;base64,Zm9v").expect("a valid URI"); + assert_eq!(read.media_type, "image/png"); + assert_eq!(read.parameters, "quality=fine"); + assert_eq!(read.bytes, b"foo"); +} + +#[test] +fn a_type_may_be_absent_while_parameters_are_present() { + // RFC 2397 spells this one out, and it is the case a reader written + // against examples never sees. + let read = data_uri::read("data:;charset=UTF-8;base64,Zm9v").expect("a valid URI"); + assert_eq!(read.media_type, ""); + assert_eq!(read.parameters, "charset=UTF-8"); +} + +#[test] +fn a_type_may_be_absent_altogether() { + let read = data_uri::read("data:;base64,Zm9v").expect("a valid URI"); + assert_eq!(read.media_type, ""); + assert_eq!(read.parameters, ""); +} + +#[test] +fn the_marker_is_read_in_any_letter_case() { + // Forgiven because no spelling of the marker changes an output byte. + for uri in [ + "data:text/plain;base64,Zm9v", + "data:text/plain;BASE64,Zm9v", + "data:text/plain;Base64,Zm9v", + "DATA:text/plain;base64,Zm9v", + ] { + assert_eq!(bytes(uri), b"foo", "{uri}"); + } +} + +/// Whether `part` is a subslice of `whole`, by address rather than by +/// content. +/// +/// Content equality would pass for a copy, and a copy is exactly what +/// the two text fields promise not to be. +fn borrowed_from(part: &str, whole: &str) -> bool { + let base = whole.as_ptr() as usize; + let start = part.as_ptr() as usize; + start >= base && start + part.len() <= base + whole.len() +} + +/// **Both text fields borrow the URI, including when they are empty.** +/// +/// An empty `&str` literal is a `'static` dangling pointer, not a view +/// into anything, so a reader that reaches for `""` when a URI has no +/// parameters returns something that only looks borrowed. The type says +/// `&'a str` and the documentation says the text fields borrow; this is +/// what makes both true rather than usually true. +/// +/// The fuzz target asserts the same thing on every input it accepts, and +/// asserting it here as well is deliberate: a harness that aborts on its +/// own corpus stops being evidence, and this test is what says so in one +/// second rather than in a fuzzing run nobody watches. +#[test] +fn both_text_fields_borrow_the_uri_even_when_empty() { + // The ordinary shape, which has no parameters at all -- and is what + // a document embedding a buffer actually writes. + let uri = HEAD.to_owned() + "QQ=="; + let read = data_uri::read(&uri).expect("a valid URI"); + assert_eq!(read.parameters, ""); + assert!( + borrowed_from(read.media_type, &uri), + "the media type must be a view into the URI" + ); + assert!( + borrowed_from(read.parameters, &uri), + "empty parameters must still be a view into the URI, not a literal" + ); + + // And the shape that has them, so neither branch is left unchecked. + let with = "data:text/plain;charset=UTF-8;base64,QQ==".to_owned(); + let read = data_uri::read(&with).expect("a valid URI"); + assert_eq!(read.parameters, "charset=UTF-8"); + assert!(borrowed_from(read.media_type, &with)); + assert!(borrowed_from(read.parameters, &with)); + + // An absent media type is the same question again, one field over. + let bare = "data:;base64,QQ==".to_owned(); + let read = data_uri::read(&bare).expect("a valid URI"); + assert_eq!(read.media_type, ""); + assert!( + borrowed_from(read.media_type, &bare), + "an empty media type must be a view into the URI too" + ); + assert!(borrowed_from(read.parameters, &bare)); +} + +#[test] +fn two_reads_of_one_uri_are_the_same_value() { + // The struct is comparable, cloneable and printable because a caller + // holding one wants all three -- and because the fuzz target's + // "reading twice answers the same" check is worth nothing if the + // comparison it makes is not the one a caller would make. + let uri = HEAD.to_owned() + "Zm9v"; + let once = data_uri::read(&uri).expect("a valid URI"); + let again = data_uri::read(&uri).expect("and again"); + assert_eq!(once, again); + assert_eq!(once.clone(), again); + assert!( + format!("{once:?}").contains("DataUri"), + "a value that cannot be printed cannot be reported" + ); + + let different = HEAD.to_owned() + "Zm9u"; + let other = data_uri::read(&different).expect("a valid URI"); + assert_ne!(once, other, "different payloads are different values"); +} + +#[test] +fn a_relative_path_is_not_a_data_uri_and_says_so() { + // The commonest non-fault there is: a document naming a second file. + assert_eq!(refused("model.bin"), DataUriError::NotADataUri); + assert!(!data_uri::looks_like("model.bin")); + assert!(data_uri::looks_like(HEAD)); +} + +#[test] +fn a_multibyte_character_where_the_scheme_would_end_is_not_a_panic() { + // Five bytes into `dat\u{e9}...` is the middle of a character, and a + // reader comparing strings by slicing rather than by bytes ends the + // process here instead of returning a refusal. + assert_eq!(refused("dat\u{e9}foo"), DataUriError::NotADataUri); + assert!(!data_uri::looks_like("dat\u{e9}foo")); + + // The same hazard at the other end: seven bytes back from the comma + // lands mid-character in this one. + assert_eq!( + refused("data:text/\u{e9}\u{e9}\u{e9}\u{e9},Zm9v"), + DataUriError::Unsupported + ); +} + +#[test] +fn a_second_marker_is_a_parameter_rather_than_a_second_instruction() { + // Only the trailing marker is consumed, so a duplicate becomes an + // ordinary parameter. Worth pinning because it is the kind of input + // a generator produces and a reader is never shown. + let read = data_uri::read("data:text/plain;base64;base64,Zm9v").expect("a valid URI"); + assert_eq!(read.media_type, "text/plain"); + assert_eq!(read.parameters, "base64"); + assert_eq!(read.bytes, b"foo"); +} + +#[test] +fn a_parameter_carrying_a_comma_is_a_limit_this_reader_has() { + // The payload is taken from the FIRST comma, so a quoted parameter + // containing one is read as the end of the description. RFC 2397 + // permits it and this reader does not: the answer is honest -- a + // refusal, not a wrong decode -- but it is a limit rather than a + // fault in the document, and a test is where that gets remembered. + assert_eq!( + refused("data:text/plain;name=\"a,b\";base64,Zm9v"), + DataUriError::Unsupported + ); +} + +#[test] +fn the_smallest_uris_are_answered_rather_than_guessed_at() { + // `data:,` is the shortest legal RFC 2397 URI there is. It has no + // marker, so this reader declines it by name. + assert_eq!(refused("data:,"), DataUriError::Unsupported); + // And the shortest one this reader does accept carries nothing. + let read = data_uri::read("data:;base64,").expect("a valid URI"); + assert_eq!(read.media_type, ""); + assert_eq!(read.parameters, ""); + assert!(read.bytes.is_empty()); +} + +#[test] +fn a_media_type_wider_than_ascii_is_reported_as_written() { + // The marker matches, so the description is sliced -- and the byte + // it is sliced at is the `;`, which is why this is safe. The other + // multibyte test covers the mismatching side; this is the side that + // succeeds. + let read = data_uri::read("DaTa:\u{e9};base64,QQ==").expect("a valid URI"); + assert_eq!(read.media_type, "\u{e9}"); + assert_eq!(read.bytes, b"A"); +} + +#[test] +fn a_uri_with_no_comma_has_no_payload() { + assert_eq!( + refused("data:application/octet-stream;base64"), + DataUriError::NoPayload + ); +} + +#[test] +fn a_percent_encoded_payload_is_refused_as_this_readers_limit() { + // Legal RFC 2397, not implemented here, and the message says which + // of those two it is. + let refusal = refused("data:text/plain,hello"); + assert_eq!(refusal, DataUriError::Unsupported); + let said = refusal.to_string(); + assert!( + said.contains("this reader"), + "the message should own the limit rather than blame the document: {said}" + ); +} + +#[test] +fn the_marker_has_to_be_a_parameter_rather_than_a_suffix() { + // `base64` without its semicolon is part of the media type, not the + // instruction, and reading it as the instruction would decode a + // payload nobody said was encoded. + // + // **The media type here is longer than the marker**, deliberately. + // `data:base64,` is six characters before the comma against the + // marker's seven, so it never reaches the comparison at all -- it + // fails the length guard, which is a different rule with the same + // answer. A fixture that cannot reach the rule it is named for is + // the defect this file has now been caught by twice. + assert_eq!(refused("data:xbase64,Zm9v"), DataUriError::Unsupported); + assert_eq!( + refused("data:text/plain base64,Zm9v"), + DataUriError::Unsupported + ); +} + +#[test] +fn a_description_shorter_than_the_marker_cannot_carry_it() { + // The length guard, named and pinned separately now that the test + // above no longer reaches it. Six characters cannot end with seven. + assert_eq!(refused("data:base64,Zm9v"), DataUriError::Unsupported); + assert_eq!(refused("data:,Zm9v"), DataUriError::Unsupported); + assert_eq!(refused("data:x,Zm9v"), DataUriError::Unsupported); +} + +#[test] +fn whitespace_in_the_payload_is_a_bad_digit_rather_than_skipped() { + // Skipping it is common and gives one payload two spellings. + // + // **The whitespace takes a character's place rather than being added + // between characters**, because a payload with a space *inserted* is + // no longer a whole number of groups and the length check answers + // first -- correctly, and it is the next test that pins that order. + for filler in [" ", "\n", "\r", "\t"] { + let uri = HEAD.to_owned() + "Zm v".replace(' ', filler).as_str(); + let at = uri.find(filler).expect("the filler is in there"); + assert_eq!(at, HEAD.len() + 2, "{uri:?}"); + assert_eq!( + refused(&uri), + DataUriError::BadDigit { + byte: filler.as_bytes()[0], + at, + }, + "{uri:?}" + ); + } +} + +#[test] +fn a_payload_of_the_wrong_length_is_measured_before_it_is_read() { + // Two faults, one answer, and the cheap whole-payload one comes + // first: counting characters needs no alphabet, and a reader that + // reported the bad character instead would be describing group four + // of a payload that has no group four. + let uri = HEAD.to_owned() + "Zm 9v"; + assert_eq!(refused(&uri), DataUriError::NotWholeGroups { len: 5 }); +} + +#[test] +fn the_url_safe_alphabet_is_not_this_one() { + // `-` and `_` are base64url. A URI payload uses `+` and `/`, and + // accepting both alphabets would decode two texts to one resource. + for wrong in ['-', '_'] { + let uri = format!("{HEAD}Zm9{wrong}"); + assert_eq!( + refused(&uri), + DataUriError::BadDigit { + byte: wrong as u8, + at: HEAD.len() + 3, + }, + "{uri}" + ); + } + assert_eq!(bytes(&format!("{HEAD}Zm9+")), [0x66, 0x6f, 0x7e]); + assert_eq!(bytes(&format!("{HEAD}Zm9/")), [0x66, 0x6f, 0x7f]); +} + +#[test] +fn the_characters_just_outside_each_range_are_refused() { + // **A widened range would pass every other test in this file.** The + // round trip only ever emits characters that are in the alphabet, + // and the property that alters a character is satisfied by decoding + // to *different* bytes -- which a wrongly-accepted character does. + // So the neighbours are the only thing that pins the edges. + for (neighbour, of) in [ + ('@', "A-Z"), + ('[', "A-Z"), + ('`', "a-z"), + ('{', "a-z"), + (':', "0-9"), + ('/', "0-9 upwards, and this one IS in the alphabet"), + ] { + let uri = format!("{HEAD}Zm9{neighbour}"); + if neighbour == '/' { + assert!( + data_uri::read(&uri).is_ok(), + "`/` is the last alphabet entry and must decode" + ); + continue; + } + assert_eq!( + refused(&uri), + DataUriError::BadDigit { + byte: neighbour as u8, + at: HEAD.len() + 3, + }, + "`{neighbour}` sits just outside {of} and must not decode" + ); + } +} + +#[test] +fn a_payload_that_is_not_whole_groups_is_refused() { + for short in ["Z", "Zm", "Zm9", "Zm9vZ"] { + let uri = HEAD.to_owned() + short; + assert_eq!( + refused(&uri), + DataUriError::NotWholeGroups { len: short.len() }, + "{uri}" + ); + } +} + +#[test] +fn padding_before_the_last_group_is_refused() { + let uri = HEAD.to_owned() + "Zg==Zg=="; + assert_eq!( + refused(&uri), + DataUriError::BadPadding { at: HEAD.len() + 2 } + ); +} + +#[test] +fn padding_in_the_first_half_of_a_group_is_refused() { + // Two `=` are legal; four are not, and neither is a group that is + // nothing but padding. + for (payload, offset) in [("Zm9v====", 4), ("=m9v", 0), ("Z=9v", 1)] { + let uri = HEAD.to_owned() + payload; + assert_eq!( + refused(&uri), + DataUriError::BadPadding { + at: HEAD.len() + offset + }, + "{uri}" + ); + } +} + +#[test] +fn a_digit_after_padding_points_at_the_padding() { + // The digit is what was found; the padding is what was wrong. + let uri = HEAD.to_owned() + "Zm=v"; + assert_eq!( + refused(&uri), + DataUriError::BadPadding { at: HEAD.len() + 2 } + ); +} + +#[test] +fn a_final_group_with_stray_bits_is_refused_though_it_would_decode() { + // `QQ==` and `QR==` would both yield the single byte `A`: the last + // four bits of the second character reach no output byte. Accepting + // both means accepting two texts as one resource. + assert_eq!(bytes(&(HEAD.to_owned() + "QQ==")), b"A"); + + let uri = HEAD.to_owned() + "QR=="; + assert_eq!( + refused(&uri), + DataUriError::NonCanonical { + at: HEAD.len() + 1, + bits: 1, + } + ); + + // One padding character leaves two bits rather than four, and the + // carrier is the third character rather than the second. + assert_eq!(bytes(&(HEAD.to_owned() + "Zm8=")), b"fo"); + let uri = HEAD.to_owned() + "Zm9="; + assert_eq!( + refused(&uri), + DataUriError::NonCanonical { + at: HEAD.len() + 2, + bits: 0b01, + } + ); +} + +/// **Every variant, asked its name and its message.** +/// +/// Not only the ones a provocation above reaches: a refusal a caller +/// cannot key on is half a refusal, and the whole list is asked once so +/// that a variant added later is covered by a test that already exists. +#[test] +fn every_refusal_names_itself_and_says_something() { + let all = [ + DataUriError::NotADataUri, + DataUriError::NoPayload, + DataUriError::Unsupported, + DataUriError::BadDigit { byte: b' ', at: 7 }, + DataUriError::NotWholeGroups { len: 3 }, + DataUriError::BadPadding { at: 9 }, + DataUriError::NonCanonical { at: 9, bits: 3 }, + ]; + + let mut names = Vec::new(); + for refusal in all { + let name = refusal.name(); + assert!(!name.is_empty(), "{refusal:?} has no name"); + assert!(!refusal.to_string().is_empty(), "{refusal:?} says nothing"); + assert!( + !names.contains(&name), + "{name} is two different refusals, and a caller keying on it cannot tell which" + ); + names.push(name); + } + + // **No variant here is unreachable from this module**, unlike the + // geometry refusals five readers share. One reader, one enum, and + // every entry above is provoked by a test in this file -- which is a + // claim worth stating, because the day it stops being true is the day + // a refusal became decoration. + assert_eq!(names.len(), 7, "the count is part of the claim"); +} + +#[test] +fn the_fixture_encoder_agrees_with_the_published_vectors() { + // The round trip is only evidence if the encoder is right, so it is + // pinned against the same vectors rather than against the decoder. + for (plain, encoded) in [ + ("", ""), + ("f", "Zg=="), + ("fo", "Zm8="), + ("foo", "Zm9v"), + ("foob", "Zm9vYg=="), + ("fooba", "Zm9vYmE="), + ("foobar", "Zm9vYmFy"), + ] { + assert_eq!(encode(plain.as_bytes()), encoded, "{plain}"); + } +} diff --git a/crates/mesh/tests/format.rs b/crates/mesh/tests/format.rs index 5a6f4b27..e849b1f2 100644 --- a/crates/mesh/tests/format.rs +++ b/crates/mesh/tests/format.rs @@ -175,6 +175,28 @@ fn a_file_that_says_nothing_an_obj_says_is_declined() { /// **The name is what a machine keys on, and every format has a /// distinct one.** +/// +/// This census listed five formats while the type had six. Nothing +/// failed, because a list written by hand agrees with itself: the sixth +/// was simply never asked its name, and the assertion about which names +/// exist was true of the five that were. +/// +/// **So the list is no longer only a list.** The match below has no +/// wildcard, which means a seventh format stops this file compiling +/// until somebody says what it is called — the same trick the refusal +/// censuses use, and the same reason: a vocabulary check that can be +/// out of date is not a check. +/// +/// **What that does and does not guarantee, precisely.** A new variant +/// cannot be added without touching this file, and it cannot be given a +/// name here without saying which. It *can* still be left out of the +/// array below, in which case it is never asked its name at run time — +/// the compiler forces the arm, not the membership. Closing that last +/// gap needs a count derived from the type, which stable Rust does not +/// offer without a derive, and a derive is a dependency. The residual +/// hole is one line wide and is written down here rather than left for +/// somebody to find, which is the same bargain the MTL census makes +/// when it writes a sentence per unreachable variant. #[test] fn every_format_has_its_own_stable_name() { let all = [ @@ -183,13 +205,63 @@ fn every_format_has_its_own_stable_name() { Format::Stl, Format::Ply, Format::Blob, + Format::Glb, ]; + + for format in all { + // No wildcard, deliberately. A format added to the type and not + // to the array above still fails here, because this arm list is + // what the compiler checks for completeness. + let expected = match format { + Format::Obj => "obj", + Format::Mtl => "mtl", + Format::Stl => "stl", + Format::Ply => "ply", + Format::Blob => "blob", + Format::Glb => "glb", + }; + assert_eq!( + format.name(), + expected, + "{format:?} answers to a name this census does not know" + ); + } + let mut names: Vec<&str> = all.iter().map(|format| format.name()).collect(); names.sort_unstable(); let distinct = names.len(); names.dedup(); assert_eq!(names.len(), distinct, "two formats share a name: {names:?}"); - assert_eq!(names, vec!["blob", "mtl", "obj", "ply", "stl"]); + assert_eq!(names, vec!["blob", "glb", "mtl", "obj", "ply", "stl"]); +} + +/// **Whether a format carries geometry is asked of every one of them.** +/// +/// The other half of the same gap: a format added to the type inherits +/// an answer here from whichever arm its variant falls into, and nothing +/// says whether that answer was chosen or inherited. A material library +/// is the only one that carries none, and that is worth stating in a +/// place that breaks when it stops being true. +#[test] +fn only_a_material_library_carries_no_geometry() { + for format in [ + Format::Obj, + Format::Mtl, + Format::Stl, + Format::Ply, + Format::Blob, + Format::Glb, + ] { + let expected = match format { + Format::Mtl => false, + Format::Obj | Format::Stl | Format::Ply | Format::Blob | Format::Glb => true, + }; + assert_eq!( + format.carries_geometry(), + expected, + "{format:?} disagrees with what this census says it carries" + ); + } } /// **`detect` answers for every byte string, and what it names can be diff --git a/crates/mesh/tests/properties.rs b/crates/mesh/tests/properties.rs index 9d643f7e..0fe32909 100644 --- a/crates/mesh/tests/properties.rs +++ b/crates/mesh/tests/properties.rs @@ -15,8 +15,14 @@ use proptest::prelude::*; use renew_math::{Mat4, Quat, Vec3}; +use renew_mesh::data_uri::{self, DataUriError}; use renew_mesh::{Mesh, MeshError, blob, place, stl}; +// The encoder the seeds and the fuzz target use, shared rather than +// copied so the round trip below is the same round trip they make. +#[path = "shared/base64_encode.rs"] +mod base64_encode; + /// A binary STL over `triangles`, built the way an exporter would. fn binary(triangles: &[([f32; 3], [[f32; 3]; 3])]) -> Vec { let mut out = vec![0u8; 80]; @@ -411,3 +417,183 @@ proptest! { ); } } + +// ---- `data:` URIs ------------------------------------------------- + +/// A URI carrying `bytes`, written the way an exporter would. +fn embedded(bytes: &[u8]) -> String { + format!( + "data:application/octet-stream;base64,{}", + base64_encode::encode(bytes) + ) +} + +/// Text that reaches the decoder often enough to be worth generating. +/// +/// **A generator of arbitrary bytes does not work here, and the reason +/// is worth stating because it is not obvious.** For a format whose +/// reader is the fallback for everything, arbitrary bytes are exactly +/// the right shape -- the STL suite beside this one is built that way +/// and is right to be. A `data:` URI begins with five fixed bytes, so +/// arbitrary input reaches `looks_like` and stops. Two properties here +/// were once written over `vec(any::(), 0..400)` and passed while +/// proving only that a random byte string is not a URI. +/// +/// So the generator builds URIs and then damages them, and +/// `the_generator_reaches_the_decoder` below measures that it did. +fn uri_text() -> impl Strategy { + prop_oneof![ + // Something an encoder would have written, with a tail that may + // ruin the last group's length, padding or canonical bits. + ( + proptest::collection::vec(any::(), 0..40), + "[A-Za-z0-9+/=]{0,5}", + ) + .prop_map(|(bytes, tail)| embedded(&bytes) + &tail), + // A description and a payload drawn from the alphabet, its + // neighbours, the padding and whitespace, so the refusals about + // characters and groups are reachable. + "data:[a-z/;=]{0,12},[A-Za-z0-9+/=_@ -]{0,30}", + // A URI whose scheme is right and whose payload is arbitrary + // text, which is what a damaged document looks like. + "data:;base64,.{0,40}", + // And bytes that are not a URI at all, because the reader has to + // answer for those too. + proptest::collection::vec(any::(), 0..400) + .prop_map(|bytes| String::from_utf8_lossy(&bytes).into_owned()), + ] +} + +/// **The generator reaches the decoder, measured rather than assumed.** +/// +/// This test exists because its absence is what let two properties above +/// pass while exercising one branch. A strategy is code, and a strategy +/// nothing checks is code nothing checks -- so this samples it with a +/// fixed seed and asserts that the answers it produces are more than one +/// refusal. +#[test] +fn the_generator_reaches_the_decoder() { + use proptest::strategy::ValueTree as _; + use proptest::test_runner::TestRunner; + use std::collections::BTreeSet; + + let mut runner = TestRunner::deterministic(); + let mut reached: BTreeSet<&'static str> = BTreeSet::new(); + for _ in 0..600 { + let text = uri_text() + .new_tree(&mut runner) + .expect("the strategy produces a value") + .current(); + reached.insert(match data_uri::read(&text) { + Ok(_) => "Ok", + Err(refusal) => refusal.name(), + }); + } + + assert!( + reached.contains("Ok"), + "no generated text decodes, so every property over this strategy is about refusals \ + alone. Reached: {reached:?}" + ); + assert!( + reached.len() >= 5, + "the strategy reaches {} of the eight answers, which is too few to stand behind a \ + property about all of them. Reached: {reached:?}", + reached.len() + ); +} + +proptest! { + /// **Whatever an encoder writes, the reader gives back.** + /// + /// The named cases beside this cover the published vectors and the + /// three padding states; this says it for every byte string, which is + /// the half a named case cannot reach — and the payloads that matter + /// most are the ones nobody would type, because `+` and `/` appear + /// only when the bytes happen to land on them. + #[test] + fn what_an_encoder_wrote_the_reader_gives_back( + bytes in proptest::collection::vec(any::(), 0..300), + ) { + let uri = embedded(&bytes); + let read = data_uri::read(&uri).expect("what was encoded reads"); + prop_assert_eq!(read.bytes, bytes); + prop_assert_eq!(read.media_type, "application/octet-stream"); + prop_assert_eq!(read.parameters, ""); + } + + /// **Text and bytes are one to one, in the direction that is easy to + /// get wrong.** + /// + /// The property above says every encoding decodes. This says nothing + /// else does: change any single character of a payload and either the + /// reader refuses it, or it decodes to something *different*. A + /// decoder that ignored the unused bits of a final group would fail + /// this and pass everything else here, which is exactly how that bug + /// survives in the wild. + #[test] + fn no_other_text_decodes_to_the_same_bytes( + bytes in proptest::collection::vec(any::(), 1..60), + at in 0_usize..80, + replacement in any::(), + ) { + let uri = embedded(&bytes); + let payload_at = uri.find(',').expect("an encoded URI has a comma") + 1; + let index = payload_at + at % (uri.len() - payload_at); + let replacement = char::from(replacement); + prop_assume!(replacement.is_ascii() && uri.as_bytes()[index] != replacement as u8); + + let mut altered = uri.clone(); + altered.replace_range(index..=index, &replacement.to_string()); + + match data_uri::read(&altered) { + Err(_) => {} + Ok(other) => prop_assert_ne!( + other.bytes, + bytes, + "{} is a second spelling of the same resource", + altered + ), + } + } + + /// **Every text gets an answer**, which is the claim the whole + /// module rests on: a reader handed something no encoder wrote + /// returns rather than looping, panicking, or deciding for itself. + #[test] + fn every_text_gets_an_answer(text in uri_text()) { + match data_uri::read(&text) { + Ok(read) => prop_assert_eq!( + base64_encode::encode(&read.bytes), + &text[text.find(',').expect("a URI that read has a comma") + 1..], + ), + Err(refusal) => { + prop_assert!(!refusal.name().is_empty()); + prop_assert!(!refusal.to_string().is_empty()); + } + } + } + + /// **An offset a refusal reports is inside the text it is about.** + /// + /// A message pointing past the end of what the caller handed over is + /// worse than no message: it sends whoever is debugging to a byte + /// that is not there. + #[test] + fn a_reported_offset_is_inside_the_text(text in uri_text()) { + let Err( + DataUriError::BadDigit { at, .. } + | DataUriError::BadPadding { at } + | DataUriError::NonCanonical { at, .. }, + ) = data_uri::read(&text) + else { + return Ok(()); + }; + prop_assert!( + at < text.len(), + "offset {} is past the end of a {}-byte text", + at, + text.len() + ); + } +} diff --git a/crates/mesh/tests/shared/base64_encode.rs b/crates/mesh/tests/shared/base64_encode.rs new file mode 100644 index 00000000..e945b5a3 --- /dev/null +++ b/crates/mesh/tests/shared/base64_encode.rs @@ -0,0 +1,71 @@ +//! Canonical base64, written once for the four harnesses that need it. +//! +//! **This file is included by five targets in two crates**, with +//! `#[path]`, because none of them can reach the others' code any other +//! way: +//! +//! * `crates/mesh/examples/make_data_uri_corpus.rs` writes the seeds. +//! * `crates/mesh/tests/data_uri.rs` builds fixtures and round-trips. +//! * `crates/mesh/tests/properties.rs` generates URIs to attack. +//! * `crates/mesh/tests/corpus_replay.rs` replays the seeds at every +//! merge. +//! * `fuzz/fuzz_targets/data_uri_read.rs` mutates them. +//! +//! Cargo compiles a `tests/` subdirectory for nobody, which is what +//! makes this a shared file rather than a sixth target. +//! +//! # Why the engine does not ship this +//! +//! Nothing in the engine writes a `data:` URI. An encoder in the crate +//! would be code with no caller, and the crate is a reader of things +//! other tools produce. +//! +//! It also has to stay separate from the decoder to be worth anything. +//! **A round trip through one body of code proves only that the code +//! agrees with itself**; this encoder is written from the specification's +//! description rather than by reading the decoder backwards, so a shared +//! misunderstanding does not cancel out. +//! +//! # What "canonical" means here, and why it is the whole point +//! +//! Three bytes become four characters. When the input does not divide by +//! three, the final group is short, and the bits no input byte reached +//! are **written as zero** and the group is padded to four with `=`. +//! +//! Those zero bits are what make the encoding reversible. A decoder that +//! ignores them accepts `QQ==` and `QR==` as the same single byte `A`, +//! and at that moment two different texts name one resource. This +//! encoder never writes the second spelling, which is why "re-encoding +//! what decoded reproduces the input exactly" is a claim worth asserting. + +#![allow( + dead_code, + reason = "the alphabet is an implementation detail of the one function here" +)] + +/// The standard alphabet. Not base64url: `+` and `/`, never `-` and `_`. +const ALPHABET: &[u8; 64] = b"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"; + +/// Encode bytes as canonical, padded base64. +pub fn encode(bytes: &[u8]) -> String { + let mut out = String::with_capacity(bytes.len().div_ceil(3) * 4); + for group in bytes.chunks(3) { + // The group's bytes, left-aligned in twenty-four bits. Whatever + // is missing stays zero, which is exactly the rule above. + let mut bits = 0_u32; + for (index, &byte) in group.iter().enumerate() { + bits |= u32::from(byte) << (16 - index * 8); + } + // One character per six bits, and one more character than the + // group has bytes: three bytes give four, two give three, one + // gives two. + for index in 0..=group.len() { + let sextet = (bits >> (18 - index * 6)) & 0b11_1111; + out.push(char::from(ALPHABET[sextet as usize])); + } + for _ in group.len()..3 { + out.push('='); + } + } + out +} diff --git a/fuzz/Cargo.lock b/fuzz/Cargo.lock index 3c3b20a5..005974e0 100644 --- a/fuzz/Cargo.lock +++ b/fuzz/Cargo.lock @@ -120,9 +120,17 @@ dependencies = [ name = "renew-json" version = "0.1.1" +[[package]] +name = "renew-math" +version = "0.1.1" + [[package]] name = "renew-mesh" version = "0.1.1" +dependencies = [ + "renew-json", + "renew-math", +] [[package]] name = "renew-net" diff --git a/fuzz/Cargo.toml b/fuzz/Cargo.toml index a198a6cd..ba9fbe2a 100644 --- a/fuzz/Cargo.toml +++ b/fuzz/Cargo.toml @@ -147,3 +147,10 @@ path = "fuzz_targets/gltf_read.rs" test = false doc = false bench = false + +[[bin]] +name = "data_uri_read" +path = "fuzz_targets/data_uri_read.rs" +test = false +doc = false +bench = false diff --git a/fuzz/corpus/data_uri_read/absolute-url.uri b/fuzz/corpus/data_uri_read/absolute-url.uri new file mode 100644 index 00000000..f5987fd3 --- /dev/null +++ b/fuzz/corpus/data_uri_read/absolute-url.uri @@ -0,0 +1 @@ +https://example.invalid/a.bin \ No newline at end of file diff --git a/fuzz/corpus/data_uri_read/digit-after-padding.uri b/fuzz/corpus/data_uri_read/digit-after-padding.uri new file mode 100644 index 00000000..b70ca5a0 --- /dev/null +++ b/fuzz/corpus/data_uri_read/digit-after-padding.uri @@ -0,0 +1 @@ +data:application/octet-stream;base64,Zm=v \ No newline at end of file diff --git a/fuzz/corpus/data_uri_read/empty-payload.uri b/fuzz/corpus/data_uri_read/empty-payload.uri new file mode 100644 index 00000000..e85a6d19 --- /dev/null +++ b/fuzz/corpus/data_uri_read/empty-payload.uri @@ -0,0 +1 @@ +data:application/octet-stream;base64, \ No newline at end of file diff --git a/fuzz/corpus/data_uri_read/every-byte-value.uri b/fuzz/corpus/data_uri_read/every-byte-value.uri new file mode 100644 index 00000000..01048e2d --- /dev/null +++ b/fuzz/corpus/data_uri_read/every-byte-value.uri @@ -0,0 +1 @@ +data:application/octet-stream;base64,AAECAwQFBgcICQoLDA0ODxAREhMUFRYXGBkaGxwdHh8gISIjJCUmJygpKissLS4vMDEyMzQ1Njc4OTo7PD0+P0BBQkNERUZHSElKS0xNTk9QUVJTVFVWV1hZWltcXV5fYGFiY2RlZmdoaWprbG1ub3BxcnN0dXZ3eHl6e3x9fn+AgYKDhIWGh4iJiouMjY6PkJGSk5SVlpeYmZqbnJ2en6ChoqOkpaanqKmqq6ytrq+wsbKztLW2t7i5uru8vb6/wMHCw8TFxsfIycrLzM3Oz9DR0tPU1dbX2Nna29zd3t/g4eLj5OXm5+jp6uvs7e7v8PHy8/T19vf4+fr7/P3+/w== \ No newline at end of file diff --git a/fuzz/corpus/data_uri_read/five-characters.uri b/fuzz/corpus/data_uri_read/five-characters.uri new file mode 100644 index 00000000..e6011cd8 --- /dev/null +++ b/fuzz/corpus/data_uri_read/five-characters.uri @@ -0,0 +1 @@ +data:application/octet-stream;base64,Zm9vZ \ No newline at end of file diff --git a/fuzz/corpus/data_uri_read/four-pads.uri b/fuzz/corpus/data_uri_read/four-pads.uri new file mode 100644 index 00000000..23f5355e --- /dev/null +++ b/fuzz/corpus/data_uri_read/four-pads.uri @@ -0,0 +1 @@ +data:application/octet-stream;base64,Zm9v==== \ No newline at end of file diff --git a/fuzz/corpus/data_uri_read/gltf-buffer-type.uri b/fuzz/corpus/data_uri_read/gltf-buffer-type.uri new file mode 100644 index 00000000..6cd87ed4 --- /dev/null +++ b/fuzz/corpus/data_uri_read/gltf-buffer-type.uri @@ -0,0 +1 @@ +data:application/gltf-buffer;base64,Zm9v \ No newline at end of file diff --git a/fuzz/corpus/data_uri_read/invalid-bytes-in-the-type.uri b/fuzz/corpus/data_uri_read/invalid-bytes-in-the-type.uri new file mode 100644 index 00000000..92862b8b --- /dev/null +++ b/fuzz/corpus/data_uri_read/invalid-bytes-in-the-type.uri @@ -0,0 +1 @@ +data:text/Ã;base64,Zm9v \ No newline at end of file diff --git a/fuzz/corpus/data_uri_read/length-1.uri b/fuzz/corpus/data_uri_read/length-1.uri new file mode 100644 index 00000000..461f9998 --- /dev/null +++ b/fuzz/corpus/data_uri_read/length-1.uri @@ -0,0 +1 @@ +data:application/octet-stream;base64,AA== \ No newline at end of file diff --git a/fuzz/corpus/data_uri_read/length-2.uri b/fuzz/corpus/data_uri_read/length-2.uri new file mode 100644 index 00000000..c0e7a0f7 --- /dev/null +++ b/fuzz/corpus/data_uri_read/length-2.uri @@ -0,0 +1 @@ +data:application/octet-stream;base64,ACU= \ No newline at end of file diff --git a/fuzz/corpus/data_uri_read/length-3.uri b/fuzz/corpus/data_uri_read/length-3.uri new file mode 100644 index 00000000..df3611b0 --- /dev/null +++ b/fuzz/corpus/data_uri_read/length-3.uri @@ -0,0 +1 @@ +data:application/octet-stream;base64,ACVK \ No newline at end of file diff --git a/fuzz/corpus/data_uri_read/length-4.uri b/fuzz/corpus/data_uri_read/length-4.uri new file mode 100644 index 00000000..ac37099f --- /dev/null +++ b/fuzz/corpus/data_uri_read/length-4.uri @@ -0,0 +1 @@ +data:application/octet-stream;base64,ACVKbw== \ No newline at end of file diff --git a/fuzz/corpus/data_uri_read/length-5.uri b/fuzz/corpus/data_uri_read/length-5.uri new file mode 100644 index 00000000..410a7e0c --- /dev/null +++ b/fuzz/corpus/data_uri_read/length-5.uri @@ -0,0 +1 @@ +data:application/octet-stream;base64,ACVKb5Q= \ No newline at end of file diff --git a/fuzz/corpus/data_uri_read/length-6.uri b/fuzz/corpus/data_uri_read/length-6.uri new file mode 100644 index 00000000..1fc03d56 --- /dev/null +++ b/fuzz/corpus/data_uri_read/length-6.uri @@ -0,0 +1 @@ +data:application/octet-stream;base64,ACVKb5S5 \ No newline at end of file diff --git a/fuzz/corpus/data_uri_read/length-7.uri b/fuzz/corpus/data_uri_read/length-7.uri new file mode 100644 index 00000000..a854454f --- /dev/null +++ b/fuzz/corpus/data_uri_read/length-7.uri @@ -0,0 +1 @@ +data:application/octet-stream;base64,ACVKb5S53g== \ No newline at end of file diff --git a/fuzz/corpus/data_uri_read/length-8.uri b/fuzz/corpus/data_uri_read/length-8.uri new file mode 100644 index 00000000..a5c97154 --- /dev/null +++ b/fuzz/corpus/data_uri_read/length-8.uri @@ -0,0 +1 @@ +data:application/octet-stream;base64,ACVKb5S53gM= \ No newline at end of file diff --git a/fuzz/corpus/data_uri_read/length-9.uri b/fuzz/corpus/data_uri_read/length-9.uri new file mode 100644 index 00000000..725cbd1f --- /dev/null +++ b/fuzz/corpus/data_uri_read/length-9.uri @@ -0,0 +1 @@ +data:application/octet-stream;base64,ACVKb5S53gMo \ No newline at end of file diff --git a/fuzz/corpus/data_uri_read/lone-continuation-byte.uri b/fuzz/corpus/data_uri_read/lone-continuation-byte.uri new file mode 100644 index 00000000..9ae5161f --- /dev/null +++ b/fuzz/corpus/data_uri_read/lone-continuation-byte.uri @@ -0,0 +1 @@ +data:application/octet-stream;base64,ÿm9v \ No newline at end of file diff --git a/fuzz/corpus/data_uri_read/long-payload.uri b/fuzz/corpus/data_uri_read/long-payload.uri new file mode 100644 index 00000000..e543bbc9 --- /dev/null +++ b/fuzz/corpus/data_uri_read/long-payload.uri @@ -0,0 +1 @@ +data:application/octet-stream;base64,AAUKDxQZHiMoLTI3PEFGS1BVWl9kaW5zeH2Ch4yRlpugpaqvtLm+w8jN0tfc4ebr8PX6BAkOExgdIicsMTY7QEVKT1RZXmNobXJ3fIGGi5CVmp+kqa6zuL3Cx8zR1tvg \ No newline at end of file diff --git a/fuzz/corpus/data_uri_read/marker-in-capitals.uri b/fuzz/corpus/data_uri_read/marker-in-capitals.uri new file mode 100644 index 00000000..91110d7d --- /dev/null +++ b/fuzz/corpus/data_uri_read/marker-in-capitals.uri @@ -0,0 +1 @@ +data:application/octet-stream;BASE64,Zm9v \ No newline at end of file diff --git a/fuzz/corpus/data_uri_read/marker-without-semicolon.uri b/fuzz/corpus/data_uri_read/marker-without-semicolon.uri new file mode 100644 index 00000000..97727324 --- /dev/null +++ b/fuzz/corpus/data_uri_read/marker-without-semicolon.uri @@ -0,0 +1 @@ +data:base64,Zm9v \ No newline at end of file diff --git a/fuzz/corpus/data_uri_read/newline-in-payload.uri b/fuzz/corpus/data_uri_read/newline-in-payload.uri new file mode 100644 index 00000000..70c9eda0 --- /dev/null +++ b/fuzz/corpus/data_uri_read/newline-in-payload.uri @@ -0,0 +1,2 @@ +data:application/octet-stream;base64,Zm +v \ No newline at end of file diff --git a/fuzz/corpus/data_uri_read/no-comma.uri b/fuzz/corpus/data_uri_read/no-comma.uri new file mode 100644 index 00000000..803af201 --- /dev/null +++ b/fuzz/corpus/data_uri_read/no-comma.uri @@ -0,0 +1 @@ +data:application/octet-stream;base64 \ No newline at end of file diff --git a/fuzz/corpus/data_uri_read/no-padding.uri b/fuzz/corpus/data_uri_read/no-padding.uri new file mode 100644 index 00000000..263ab0b4 --- /dev/null +++ b/fuzz/corpus/data_uri_read/no-padding.uri @@ -0,0 +1 @@ +data:application/octet-stream;base64,Zm9v \ No newline at end of file diff --git a/fuzz/corpus/data_uri_read/no-type-at-all.uri b/fuzz/corpus/data_uri_read/no-type-at-all.uri new file mode 100644 index 00000000..921708cd --- /dev/null +++ b/fuzz/corpus/data_uri_read/no-type-at-all.uri @@ -0,0 +1 @@ +data:;base64,YWJj \ No newline at end of file diff --git a/fuzz/corpus/data_uri_read/one-pad.uri b/fuzz/corpus/data_uri_read/one-pad.uri new file mode 100644 index 00000000..baf7bec9 --- /dev/null +++ b/fuzz/corpus/data_uri_read/one-pad.uri @@ -0,0 +1 @@ +data:application/octet-stream;base64,Zm8= \ No newline at end of file diff --git a/fuzz/corpus/data_uri_read/padding-at-the-front.uri b/fuzz/corpus/data_uri_read/padding-at-the-front.uri new file mode 100644 index 00000000..71b0a1a2 --- /dev/null +++ b/fuzz/corpus/data_uri_read/padding-at-the-front.uri @@ -0,0 +1 @@ +data:application/octet-stream;base64,=m9v \ No newline at end of file diff --git a/fuzz/corpus/data_uri_read/padding-mid-payload.uri b/fuzz/corpus/data_uri_read/padding-mid-payload.uri new file mode 100644 index 00000000..1ded8052 --- /dev/null +++ b/fuzz/corpus/data_uri_read/padding-mid-payload.uri @@ -0,0 +1 @@ +data:application/octet-stream;base64,Zg==Zg== \ No newline at end of file diff --git a/fuzz/corpus/data_uri_read/parameter-without-type.uri b/fuzz/corpus/data_uri_read/parameter-without-type.uri new file mode 100644 index 00000000..25b28bf6 --- /dev/null +++ b/fuzz/corpus/data_uri_read/parameter-without-type.uri @@ -0,0 +1 @@ +data:;charset=UTF-8;base64,aGVsbG8h \ No newline at end of file diff --git a/fuzz/corpus/data_uri_read/percent-encoded.uri b/fuzz/corpus/data_uri_read/percent-encoded.uri new file mode 100644 index 00000000..51e380b9 --- /dev/null +++ b/fuzz/corpus/data_uri_read/percent-encoded.uri @@ -0,0 +1 @@ +data:application/octet-stream,%00%01 \ No newline at end of file diff --git a/fuzz/corpus/data_uri_read/relative-path.uri b/fuzz/corpus/data_uri_read/relative-path.uri new file mode 100644 index 00000000..21b39eba --- /dev/null +++ b/fuzz/corpus/data_uri_read/relative-path.uri @@ -0,0 +1 @@ +geometry.bin \ No newline at end of file diff --git a/fuzz/corpus/data_uri_read/scheme-misspelt.uri b/fuzz/corpus/data_uri_read/scheme-misspelt.uri new file mode 100644 index 00000000..2ce4a721 --- /dev/null +++ b/fuzz/corpus/data_uri_read/scheme-misspelt.uri @@ -0,0 +1 @@ +dat:application/octet-stream;base64,Zm9v \ No newline at end of file diff --git a/fuzz/corpus/data_uri_read/scheme-truncated.uri b/fuzz/corpus/data_uri_read/scheme-truncated.uri new file mode 100644 index 00000000..6320cd24 --- /dev/null +++ b/fuzz/corpus/data_uri_read/scheme-truncated.uri @@ -0,0 +1 @@ +data \ No newline at end of file diff --git a/fuzz/corpus/data_uri_read/space-in-payload.uri b/fuzz/corpus/data_uri_read/space-in-payload.uri new file mode 100644 index 00000000..7b5004dd --- /dev/null +++ b/fuzz/corpus/data_uri_read/space-in-payload.uri @@ -0,0 +1 @@ +data:application/octet-stream;base64,Zm v \ No newline at end of file diff --git a/fuzz/corpus/data_uri_read/stray-bits-one-pad.uri b/fuzz/corpus/data_uri_read/stray-bits-one-pad.uri new file mode 100644 index 00000000..a07ee2fe --- /dev/null +++ b/fuzz/corpus/data_uri_read/stray-bits-one-pad.uri @@ -0,0 +1 @@ +data:application/octet-stream;base64,Zm9= \ No newline at end of file diff --git a/fuzz/corpus/data_uri_read/stray-bits-two-pads.uri b/fuzz/corpus/data_uri_read/stray-bits-two-pads.uri new file mode 100644 index 00000000..a6c9c00d --- /dev/null +++ b/fuzz/corpus/data_uri_read/stray-bits-two-pads.uri @@ -0,0 +1 @@ +data:application/octet-stream;base64,QR== \ No newline at end of file diff --git a/fuzz/corpus/data_uri_read/three-characters.uri b/fuzz/corpus/data_uri_read/three-characters.uri new file mode 100644 index 00000000..cee241fe --- /dev/null +++ b/fuzz/corpus/data_uri_read/three-characters.uri @@ -0,0 +1 @@ +data:application/octet-stream;base64,Zm9 \ No newline at end of file diff --git a/fuzz/corpus/data_uri_read/truncated-sequence.uri b/fuzz/corpus/data_uri_read/truncated-sequence.uri new file mode 100644 index 00000000..9a854af5 --- /dev/null +++ b/fuzz/corpus/data_uri_read/truncated-sequence.uri @@ -0,0 +1 @@ +data:application/octet-stream;base64,â‚ \ No newline at end of file diff --git a/fuzz/corpus/data_uri_read/two-pads.uri b/fuzz/corpus/data_uri_read/two-pads.uri new file mode 100644 index 00000000..5951292e --- /dev/null +++ b/fuzz/corpus/data_uri_read/two-pads.uri @@ -0,0 +1 @@ +data:application/octet-stream;base64,Zg== \ No newline at end of file diff --git a/fuzz/corpus/data_uri_read/type-with-parameter.uri b/fuzz/corpus/data_uri_read/type-with-parameter.uri new file mode 100644 index 00000000..623ea4c9 --- /dev/null +++ b/fuzz/corpus/data_uri_read/type-with-parameter.uri @@ -0,0 +1 @@ +data:text/plain;charset=UTF-8;base64,aGVsbG8h \ No newline at end of file diff --git a/fuzz/corpus/data_uri_read/url-safe-dash.uri b/fuzz/corpus/data_uri_read/url-safe-dash.uri new file mode 100644 index 00000000..3ff0eb0c --- /dev/null +++ b/fuzz/corpus/data_uri_read/url-safe-dash.uri @@ -0,0 +1 @@ +data:application/octet-stream;base64,Zm9- \ No newline at end of file diff --git a/fuzz/corpus/data_uri_read/url-safe-underscore.uri b/fuzz/corpus/data_uri_read/url-safe-underscore.uri new file mode 100644 index 00000000..1d5efec9 --- /dev/null +++ b/fuzz/corpus/data_uri_read/url-safe-underscore.uri @@ -0,0 +1 @@ +data:application/octet-stream;base64,Zm9_ \ No newline at end of file diff --git a/fuzz/corpus/data_uri_read/wide-character-at-the-scheme.uri b/fuzz/corpus/data_uri_read/wide-character-at-the-scheme.uri new file mode 100644 index 00000000..2d9184de --- /dev/null +++ b/fuzz/corpus/data_uri_read/wide-character-at-the-scheme.uri @@ -0,0 +1 @@ +datéfoo \ No newline at end of file diff --git a/fuzz/corpus/data_uri_read/wide-character-before-the-comma.uri b/fuzz/corpus/data_uri_read/wide-character-before-the-comma.uri new file mode 100644 index 00000000..a4196611 --- /dev/null +++ b/fuzz/corpus/data_uri_read/wide-character-before-the-comma.uri @@ -0,0 +1 @@ +data:text/éééé,Zm9v \ No newline at end of file diff --git a/fuzz/fuzz_targets/data_uri_read.rs b/fuzz/fuzz_targets/data_uri_read.rs new file mode 100644 index 00000000..616e9cfc --- /dev/null +++ b/fuzz/fuzz_targets/data_uri_read.rs @@ -0,0 +1,97 @@ +//! `data:` URIs, against text nobody wrote on purpose. +//! +//! **The property under test is that text and bytes are one to one.** A +//! base64 decoder is easy to write leniently — skip the whitespace, +//! ignore the bits no output byte uses, accept a missing pad — and every +//! one of those leniencies makes two different texts decode to the same +//! resource. This target holds the decoder to the strict reading by +//! re-encoding whatever it accepted and demanding the original payload +//! back, character for character. +//! +//! That is a stronger claim than "it did not crash", and it is the claim +//! that a corpus can actually falsify: a decoder that quietly tolerated a +//! stray bit would return bytes whose re-encoding differs from the input +//! in exactly that character. +//! +//! **Every input is text here, and that costs something worth naming.** +//! A URI is text by definition, so the reader takes `&str` and these +//! bytes are converted lossily rather than discarded when they are not +//! UTF-8. That keeps every input useful, but it also means **a raw byte +//! above 0x7F never reaches the decoder as itself**: it arrives as the +//! replacement character, three bytes wide, sitting wherever the fuzzer +//! put a bad one. So what is under attack is the wide character landing +//! at an awkward offset — which is a real hazard for a reader that +//! slices strings — and not the byte the fuzzer chose. The alphabet's +//! rejection of `0x80..=0xFF` is reached by the suite beside the crate +//! rather than from here. + +#![no_main] + +use libfuzzer_sys::fuzz_target; +use renew_mesh::data_uri; + +// **The encoder is deliberately not the crate's**, because the crate has +// none: nothing in the engine writes a `data:` URI. Keeping it separate +// is also what makes the round trip evidence — a round trip through one +// body of code proves only that the code agrees with itself. +#[path = "../../crates/mesh/tests/shared/base64_encode.rs"] +mod base64_encode; +use base64_encode::encode; + +/// Whether `part` is a subslice of `whole`, by address rather than by +/// content. +fn borrowed_from(part: &[u8], whole: &[u8]) -> bool { + let base = whole.as_ptr() as usize; + let start = part.as_ptr() as usize; + let end = start.saturating_add(part.len()); + start >= base && end <= base.saturating_add(whole.len()) +} + +fuzz_target!(|data: &[u8]| { + let uri = String::from_utf8_lossy(data); + + let Ok(read) = data_uri::read(&uri) else { + // A refusal is an answer. Which refusal is the suite's business + // beside the crate; that the call returned at all is this + // target's. + return; + }; + + // The text came back borrowed, which is the whole reason those two + // fields are slices rather than owned strings. + assert!( + borrowed_from(read.media_type.as_bytes(), uri.as_bytes()), + "the media type must borrow the URI" + ); + assert!( + borrowed_from(read.parameters.as_bytes(), uri.as_bytes()), + "the parameters must borrow the URI" + ); + + let comma = uri.find(',').expect("a URI that read has a comma"); + let payload = &uri[comma + 1..]; + + // Three bytes out per four characters in, less one for each pad. + let pads = payload.bytes().filter(|&byte| byte == b'=').count(); + assert_eq!( + read.bytes.len(), + payload.len() / 4 * 3 - pads, + "{} bytes out of a {}-character payload with {pads} pads", + read.bytes.len(), + payload.len() + ); + + // **The claim this target exists for.** Anything the decoder accepts + // must be what an encoder would have written, or two texts decode to + // one resource and a document's bytes stop determining what it means. + assert_eq!( + encode(&read.bytes), + payload, + "what decoded does not re-encode to itself" + ); + + // Reading twice answers the same, which is what makes a corpus worth + // recording at all. + let again = data_uri::read(&uri).expect("what read once reads again"); + assert_eq!(again, read, "the same text read to the same bytes"); +}); diff --git a/tools/cli/tests/workspace_lists.rs b/tools/cli/tests/workspace_lists.rs index 40b5dcdd..3750a846 100644 --- a/tools/cli/tests/workspace_lists.rs +++ b/tools/cli/tests/workspace_lists.rs @@ -1343,6 +1343,114 @@ fn parenthesised_rule_citation(line: &str) -> Option { None } +/// Every fuzz target the workspace builds, in declaration order. +fn fuzz_targets(root: &Path) -> Result, String> { + let manifest = std::fs::read_to_string(root.join("fuzz/Cargo.toml")) + .map_err(|error| format!("fuzz/Cargo.toml is unreadable: {error}"))?; + + // A target is a `[[bin]]` section, and the package's own `name` is + // not one -- so sections are counted rather than every `name = ` + // line, which would pick up the package and call it a target. + let mut names = Vec::new(); + let mut in_bin = false; + for line in manifest.lines() { + let line = line.trim(); + if line.starts_with('[') { + in_bin = line == "[[bin]]"; + continue; + } + if !in_bin { + continue; + } + if let Some(rest) = line.strip_prefix("name = ") { + names.push(rest.trim_matches('"').to_owned()); + } + } + if names.is_empty() { + return Err("no fuzz targets found, which cannot be right".to_owned()); + } + Ok(names) +} + +/// **Every fuzz target has an entry in the refusal catalogue.** +/// +/// The catalogue is written to be implemented against: a reader is built +/// by going down its list and answering every entry. That only works +/// while the list describes the readers that exist — and a table of +/// readers is exactly the kind of document that drifts, because nothing +/// fails when a row is missing. +/// +/// Three readers went in with fuzz targets and corpora and no row here +/// before this check existed, which is the whole argument for it. A +/// target is the right thing to key on: it is declared once, in one file, +/// by the same change that adds the reader. +#[test] +fn every_fuzz_target_has_an_entry_in_the_refusal_catalogue() { + let root = workspace_root(); + let targets = fuzz_targets(&root).expect("the fuzz manifest lists its targets"); + let catalogue = std::fs::read_to_string(root.join("REFUSALS.md")) + .expect("the catalogue is part of the tree"); + + // **The table, not the whole document.** A substring search over the + // file is satisfied by any passing mention -- including one in a + // paragraph the same change added -- while the message below asks + // for a row with an error type, a refusal count and a corpus floor. + // A check that accepts less than its message demands teaches the + // next reader to write the mention and move on. + let rows: Vec<&str> = catalogue + .lines() + .map(str::trim) + .filter(|line| line.starts_with('|')) + .collect(); + + let missing: Vec<&String> = targets + .iter() + .filter(|target| !rows.iter().any(|row| row.contains(target.as_str()))) + .collect(); + + assert!( + missing.is_empty(), + "these fuzz targets have no row in REFUSALS.md's table of readers: {missing:?}. A reader \ + worth attacking is worth describing: add its row, with its error type, how many refusals \ + it has and its corpus floor." + ); +} + +/// **Every fuzz target is actually run by the fuzz workflow.** +/// +/// A harness that is declared, built by nothing and run by nothing is +/// not evidence of anything, and it fails silently: the corpus still +/// replays through the library at every merge, so the reader looks +/// covered while the assertions written into the harness itself have +/// never once executed. +/// +/// This is the half the catalogue check does not cover. That one asks +/// whether a reader is described; this asks whether it is attacked. +#[test] +fn every_fuzz_target_is_run_by_the_fuzz_workflow() { + let root = workspace_root(); + let targets = fuzz_targets(&root).expect("the fuzz manifest lists its targets"); + let workflow = std::fs::read_to_string(root.join(".github/workflows/fuzz.yml")) + .expect("the fuzz workflow is part of the tree"); + + let matrix = workflow + .lines() + .find(|line| line.trim_start().starts_with("target: [")) + .expect("the fuzz workflow names its targets in one matrix line"); + + let unrun: Vec<&String> = targets + .iter() + .filter(|target| !matrix.contains(target.as_str())) + .collect(); + + assert!( + unrun.is_empty(), + "these fuzz targets are declared but never run: {unrun:?}. Add them to the matrix in \ + .github/workflows/fuzz.yml, or delete the harness -- a target nothing runs is a claim \ + nothing checks." + ); +} + /// No source file may cite material this repository does not contain. /// /// **A comment naming a document a reader cannot open is worse than a