Skip to content

A data: URI carries its own bytes - #331

Merged
CagdasErturk merged 4 commits into
mainfrom
gltf-external
Sep 9, 2026
Merged

A data: URI carries its own bytes#331
CagdasErturk merged 4 commits into
mainfrom
gltf-external

Conversation

@CagdasErturk

@CagdasErturk CagdasErturk commented Sep 9, 2026

Copy link
Copy Markdown
Member

A document that wants to be one file embeds its binary as an RFC 2397 data: URI, and nothing in this tree could decode one. This adds the decoder as a layer of its own: a URI in, bytes out, the media type reported rather than judged, and no filesystem access.

The rule it is built on is that a difference which cannot change an output byte is forgiven and one that can is refused. The ;base64 marker is read in any letter case. Whitespace, the URL-safe alphabet, a payload that is not whole four-character groups, and padding anywhere but the end are refused, and the three refusals that are about one character carry its offset.

The one that is easy to miss is the final group's unused bits: QQ== and QR== would both decode to the single byte A, because the last four bits of the second character reach no output byte. Accepting both lets two different texts name one resource, so only the first is accepted.

Seven refusals, all reachable, each named and each provoked. Forty-four seeds, one per rule on both sides of it, with the replay gate the other readers here have. Five property tests, including one that alters a single character of an encoded payload and requires the result to be refused or to decode to something different.

Four review passes rejected the first version of this, and the checks were green throughout. The last commit is what came of them, and two findings are worth naming here because neither is visible to a gate — both pass:

  • The fuzz target had never run and could not. A URI with no parameters returned an empty string literal, which is a static pointer borrowing nothing, and the target asserts by address that both text fields are views into the URI. It aborts on eighteen of the twenty seeds it accepts — every ordinary embedded buffer. The empty tail of the description is a view; an empty literal is not. A test now pins provenance for both fields in both shapes.
  • Two property tests asserted nothing. Both generated arbitrary bytes, which begin with the scheme essentially never, so both only reached the first refusal and one ran its assertion zero times. There is a generator now, and a test that measures the generator.

Nothing would have caught the first of those, because the target is in no matrix. Seven targets were declared, given corpora and run by nothing: obj_read, mtl_read, blob_read, glb_read, accessor_view, gltf_read and this one. All seven are scheduled now, and a check requires every declared target to be there.

The alphabet is a lookup table generated at compile time from the match that already spelled it, which measured about four times the throughput of the ranges on a payload of the size this reader is documented to expect. Two tests pin the generation: every entry against the match it came from, and the alphabet as a bijection onto the six-bit values.

REFUSALS.md gains rows for three readers that landed with fuzz targets and never reached it, and its per-reader counts are re-measured — the first attempt used a parser that could not see a match arm with a block body.

Local gates seen on the final commit: cargo fmt --all --check clean, cargo clippy --workspace --all-targets -D warnings clean, cargo test --workspace 264 suites / 2,932 passed / 0 failed, renew check reports the workspace healthy, and scoped coverage over renew-mesh puts the new module at 100% of lines and 100% of regions. The fuzz target itself cannot be run locally — cargo fuzz needs nightly and this workspace is stable-only — which is exactly why it now runs on the schedule.

A document that wants to be one file embeds its binary as an RFC 2397
`data:` URI, and nothing in this tree could decode one. This adds the
decoder as its own layer: it turns a URI into bytes, reports the media
type without judging it, and never touches the filesystem.

The rule it is built on is that a difference which cannot change an
output byte is forgiven and one that can is refused. The `;base64`
marker is read in any letter case; whitespace, the URL-safe alphabet,
a payload that is not whole four-character groups, and padding anywhere
but the end are all refused with the offset that failed.

The one that is easy to miss is the final group's unused bits. `QQ==`
and `QR==` would both decode to the single byte `A`, because the last
four bits of the second character reach no output byte. Accepting both
lets two different texts name one resource, so the second spelling is
refused - and the fuzz target holds that line from the other side, by
re-encoding everything the decoder accepts and demanding the input back
character for character.

Forty-one seeds, one per rule on both sides of it, with the replay gate
the other readers here have. The encoder those seeds are written with
is shared by four harnesses and deliberately absent from the crate:
nothing in the engine writes a URI, and a round trip through one body
of code proves only that the code agrees with itself.

The README gains a section for this layer and for the document reader,
which had none.
Three readers went into this tree with fuzz targets, corpora and replay
gates, and none of them reached the catalogue's table of readers: the
glTF container, the accessor layer and the glTF document. The shared
mesh error type also grew from eleven variants to fifteen while the
table still said eleven, and the per-reader counts moved with it.

The table is the kind of document that drifts, because nothing fails
when a row is missing. So this adds the check that would have caught
it: every fuzz target the workspace builds must be named somewhere in
the catalogue. A target is the right thing to key on - it is declared
once, in one file, by the same change that adds the reader.

Measured rather than estimated: each reader's reach comes from the
census test that already names what it cannot reach.

Entry 17 gains the base64 decoder as a worked example, because it is
the instance that shows the category is not about file formats: two
texts that decode to one byte string are a second legal spelling of one
fact, exactly as two orderings of a pool are.
The census listed five formats while the type had six: the container
arm went in and was never asked its name, and the assertion about which
names exist stayed true of the five that were. Nothing failed, because
a list written by hand agrees with itself.

So the list is no longer only a list. The match inside it has no
wildcard, which means a seventh format stops the file compiling until
somebody says what it is called - the same trick the refusal censuses
use, for the same reason: a vocabulary check that can be out of date is
not a check.

A second census covers the other half of the same gap. Whether a format
carries geometry is now asked of every one of them, so a format added
to the type cannot inherit an answer from whichever arm it happens to
fall into without anyone choosing it.
Four review passes against this reader rejected it, and the checks were
green the whole time. The two worst findings are both invisible to a
gate, because both pass.

The fuzz target had never run and could not. Reading a URI with no
parameters returned an empty string literal, which is a static pointer
that borrows nothing, and the target asserts by address that both text
fields are views into the URI it was given. So it aborts on eighteen of
the twenty seeds it accepts, which is every ordinary embedded buffer.
The empty tail of the description is a view; an empty literal is not.
A test now pins provenance for both fields in both shapes.

Nothing would have caught that, because the target is in no matrix.
Seven targets were declared, given corpora and run by nothing: obj_read,
mtl_read, blob_read, glb_read, accessor_view, gltf_read and this one.
All seven are in the schedule now, and a check requires every declared
target to be there. The catalogue check beside it asked for a table row
in its message and accepted any mention in the file; it reads rows now.

Two property tests asserted nothing. Both generated arbitrary bytes,
which begin with the scheme essentially never, so both only ever reached
the first refusal and one of them ran its assertion zero times. The
suite beside this one generates arbitrary bytes and is right to, because
that format is the fallback for everything; this one begins with five
fixed bytes. There is a generator now, and a test that measures the
generator, because a strategy is code and a strategy nothing checks is
code nothing checks.

The alphabet is a lookup table generated at compile time from the match
that already spelled it, which measured about four times the throughput
of the ranges on a payload of the size this reader is documented to
expect. Two tests pin the generation: every entry against the match it
came from, and the alphabet as a bijection onto the six-bit values.

The refusal for a payload this reader cannot decode was named for the
payload rather than for the reader. A percent-encoded data URI is legal
and genuinely is not base64, so a program keying on that name would read
a conformant document as a malformed one. It is Unsupported now, which
is the word the document reader already uses.

Also: a fixture that tripped the length guard rather than the rule it
was named for; nothing pinning the characters just outside each alphabet
range, so a widened range passed the whole suite; no seed that was not
valid text, so the conversion every input goes through was never
replayed; a padding floor that counted the separator in a media type; a
doc comment the new helper had taken from another test; and per-reader
refusal counts in the catalogue that were measured with a parser that
could not see a match arm with a block body.
@CagdasErturk
CagdasErturk merged commit 9b5e26e into main Sep 9, 2026
32 checks passed
@CagdasErturk
CagdasErturk deleted the gltf-external branch September 9, 2026 17:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant