A data: URI carries its own bytes - #331
Merged
Merged
Conversation
A document that wants to be one file embeds its binary as an RFC 2397 `data:` URI, and nothing in this tree could decode one. This adds the decoder as its own layer: it turns a URI into bytes, reports the media type without judging it, and never touches the filesystem. The rule it is built on is that a difference which cannot change an output byte is forgiven and one that can is refused. The `;base64` marker is read in any letter case; whitespace, the URL-safe alphabet, a payload that is not whole four-character groups, and padding anywhere but the end are all refused with the offset that failed. The one that is easy to miss is the final group's unused bits. `QQ==` and `QR==` would both decode to the single byte `A`, because the last four bits of the second character reach no output byte. Accepting both lets two different texts name one resource, so the second spelling is refused - and the fuzz target holds that line from the other side, by re-encoding everything the decoder accepts and demanding the input back character for character. Forty-one seeds, one per rule on both sides of it, with the replay gate the other readers here have. The encoder those seeds are written with is shared by four harnesses and deliberately absent from the crate: nothing in the engine writes a URI, and a round trip through one body of code proves only that the code agrees with itself. The README gains a section for this layer and for the document reader, which had none.
Three readers went into this tree with fuzz targets, corpora and replay gates, and none of them reached the catalogue's table of readers: the glTF container, the accessor layer and the glTF document. The shared mesh error type also grew from eleven variants to fifteen while the table still said eleven, and the per-reader counts moved with it. The table is the kind of document that drifts, because nothing fails when a row is missing. So this adds the check that would have caught it: every fuzz target the workspace builds must be named somewhere in the catalogue. A target is the right thing to key on - it is declared once, in one file, by the same change that adds the reader. Measured rather than estimated: each reader's reach comes from the census test that already names what it cannot reach. Entry 17 gains the base64 decoder as a worked example, because it is the instance that shows the category is not about file formats: two texts that decode to one byte string are a second legal spelling of one fact, exactly as two orderings of a pool are.
The census listed five formats while the type had six: the container arm went in and was never asked its name, and the assertion about which names exist stayed true of the five that were. Nothing failed, because a list written by hand agrees with itself. So the list is no longer only a list. The match inside it has no wildcard, which means a seventh format stops the file compiling until somebody says what it is called - the same trick the refusal censuses use, for the same reason: a vocabulary check that can be out of date is not a check. A second census covers the other half of the same gap. Whether a format carries geometry is now asked of every one of them, so a format added to the type cannot inherit an answer from whichever arm it happens to fall into without anyone choosing it.
Four review passes against this reader rejected it, and the checks were green the whole time. The two worst findings are both invisible to a gate, because both pass. The fuzz target had never run and could not. Reading a URI with no parameters returned an empty string literal, which is a static pointer that borrows nothing, and the target asserts by address that both text fields are views into the URI it was given. So it aborts on eighteen of the twenty seeds it accepts, which is every ordinary embedded buffer. The empty tail of the description is a view; an empty literal is not. A test now pins provenance for both fields in both shapes. Nothing would have caught that, because the target is in no matrix. Seven targets were declared, given corpora and run by nothing: obj_read, mtl_read, blob_read, glb_read, accessor_view, gltf_read and this one. All seven are in the schedule now, and a check requires every declared target to be there. The catalogue check beside it asked for a table row in its message and accepted any mention in the file; it reads rows now. Two property tests asserted nothing. Both generated arbitrary bytes, which begin with the scheme essentially never, so both only ever reached the first refusal and one of them ran its assertion zero times. The suite beside this one generates arbitrary bytes and is right to, because that format is the fallback for everything; this one begins with five fixed bytes. There is a generator now, and a test that measures the generator, because a strategy is code and a strategy nothing checks is code nothing checks. The alphabet is a lookup table generated at compile time from the match that already spelled it, which measured about four times the throughput of the ranges on a payload of the size this reader is documented to expect. Two tests pin the generation: every entry against the match it came from, and the alphabet as a bijection onto the six-bit values. The refusal for a payload this reader cannot decode was named for the payload rather than for the reader. A percent-encoded data URI is legal and genuinely is not base64, so a program keying on that name would read a conformant document as a malformed one. It is Unsupported now, which is the word the document reader already uses. Also: a fixture that tripped the length guard rather than the rule it was named for; nothing pinning the characters just outside each alphabet range, so a widened range passed the whole suite; no seed that was not valid text, so the conversion every input goes through was never replayed; a padding floor that counted the separator in a media type; a doc comment the new helper had taken from another test; and per-reader refusal counts in the catalogue that were measured with a parser that could not see a match arm with a block body.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A document that wants to be one file embeds its binary as an RFC 2397
data:URI, and nothing in this tree could decode one. This adds the decoder as a layer of its own: a URI in, bytes out, the media type reported rather than judged, and no filesystem access.The rule it is built on is that a difference which cannot change an output byte is forgiven and one that can is refused. The
;base64marker is read in any letter case. Whitespace, the URL-safe alphabet, a payload that is not whole four-character groups, and padding anywhere but the end are refused, and the three refusals that are about one character carry its offset.The one that is easy to miss is the final group's unused bits:
QQ==andQR==would both decode to the single byteA, because the last four bits of the second character reach no output byte. Accepting both lets two different texts name one resource, so only the first is accepted.Seven refusals, all reachable, each named and each provoked. Forty-four seeds, one per rule on both sides of it, with the replay gate the other readers here have. Five property tests, including one that alters a single character of an encoded payload and requires the result to be refused or to decode to something different.
Four review passes rejected the first version of this, and the checks were green throughout. The last commit is what came of them, and two findings are worth naming here because neither is visible to a gate — both pass:
Nothing would have caught the first of those, because the target is in no matrix. Seven targets were declared, given corpora and run by nothing:
obj_read,mtl_read,blob_read,glb_read,accessor_view,gltf_readand this one. All seven are scheduled now, and a check requires every declared target to be there.The alphabet is a lookup table generated at compile time from the match that already spelled it, which measured about four times the throughput of the ranges on a payload of the size this reader is documented to expect. Two tests pin the generation: every entry against the match it came from, and the alphabet as a bijection onto the six-bit values.
REFUSALS.mdgains rows for three readers that landed with fuzz targets and never reached it, and its per-reader counts are re-measured — the first attempt used a parser that could not see a match arm with a block body.Local gates seen on the final commit:
cargo fmt --all --checkclean,cargo clippy --workspace --all-targets -D warningsclean,cargo test --workspace264 suites / 2,932 passed / 0 failed,renew checkreports the workspace healthy, and scoped coverage overrenew-meshputs the new module at 100% of lines and 100% of regions. The fuzz target itself cannot be run locally —cargo fuzzneeds nightly and this workspace is stable-only — which is exactly why it now runs on the schedule.