Skip to content

Store an INT128 column as sixteen bytes at a fixed stride - #777

Merged
tamnd merged 1 commit into
mainfrom
s2-int128-column
Aug 25, 2026
Merged

tamnd merged 1 commit into
mainfrom
s2-int128-column

Conversation

@tamnd

@tamnd tamnd commented Aug 25, 2026

Copy link
Copy Markdown
Owner

Every number a column has held so far rides the sixty four bit scalar lane. This adds the first one that does not, which is the S2 checklist row for the wide integers, and it lands one of the three rather than all of them.

The layout is the one BINARY(16) already had: sixteen little endian bytes a row at a fixed stride, on the blob side of the store, with no per row offsets because every row is the same width. Reaching it took one line. props::fixed_octets is the single hook that puts a column on that layout and write_columns is already generic over it, so answering Some(16) for IntBits::B128 gets the padding, the segment write and the reader for nothing.

The value carrier is Value::Decimal at scale nought. An exact numeric of scale nought is what a whole number is: it prints without a point, it compares equal to the integer of the same size, and it holds the whole i128 range, since MAX_DIGITS = 38 is a cap on what a declaration may ask for and not on what the carrier holds.

UINT128 and INT256 are refused, and the refusal lands at the declaration rather than at the first insert. Sixteen bytes would hold a UINT128 happily, but the exact numeric carries a signed hundred and twenty eight bit number, so the top half of the unsigned range is a range no value could name, and INT256 has no carrier here at all. A column whose declared range no value can reach would take a row in and refuse to give it back, which is worse than saying so where the user wrote it. That is where the wide decimal's refusal lands and for the same reason, so the two agree.

Three paths are closed on purpose rather than by accident. The vector layer has no lane for sixteen bytes and would have handed the column back as a string vector, a number dressed as text, so check_col refuses it the way it already refuses a zoned and a decimal column. sqlite's INTEGER is sixty four bits, so a staged INT128 column would have lost its top half silently, and the converter refuses instead of narrowing. ColBuf::for_type matches on the exact declared type and so has no arm for it, which refuses a bulk append when the appender opens rather than after a buffer is full.

The declared form and the column form stay one encoding, since the new extended code is written by extended_bytes and both forms go through it, so an INT128 a graph type may name is an INT128 a column can hold and there is no gap between them for a declaration to fall into.

Directory version 13. A file written before it has no column of this type to read, so nothing older changes shape.

Tests: a_declared_int128_column_holds_a_number_wider_than_a_lane writes thirty eight nines, minus thirty eight nines and 7 into one column and reads all three back in order, and pins that a value with a fraction is refused with 22003 rather than rounded. The frontier table in props.rs moves INT128 to the storable half and the unstorable count falls from nineteen to eighteen. graph_type.rs moves it to Answer::Declared and the declarable set goes from twenty six to twenty seven. Two conformance cases, one for the declaration standing and one for UINT128 being refused with 42000.

Local gate is green: fmt, clippy with -D warnings, terms, api-map, and eighty nine test result: ok lines across the six test runs.

Left open: DECIMAL(19..=38, s), which is now nearly free on this plane, being the same sixteen bytes plus the scale from the declared type.

Part of #700.

Every number a column has held so far rides the sixty four bit scalar
lane. This adds the first one that does not.

The layout is the one BINARY(16) already had: sixteen little endian
bytes a row at a fixed stride, on the blob side of the store, with no
per row offsets because every row is the same width. Reaching it took
one line, because props::fixed_octets is the single hook that puts a
column on that layout and write_columns is already generic over it.
Answering Some(16) for IntBits::B128 gets the padding, the segment
write and the reader for nothing.

The value carrier is Value::Decimal at scale nought. An exact numeric
of scale nought is what a whole number is, it prints without a point,
it compares equal to the integer of the same size, and it holds the
whole i128 range: MAX_DIGITS = 38 is a cap on what a declaration may
ask for and not on what the carrier holds.

UINT128 and INT256 are refused, and the refusal lands at the
declaration rather than at the first insert. Sixteen bytes would hold
a UINT128 happily, but the exact numeric carries a signed hundred and
twenty eight bit number, so the top half of the unsigned range is a
range no value could name; INT256 has no carrier here at all. A column
whose declared range no value can reach would take a row in and refuse
to give it back, which is worse than saying so where the user wrote
it. That is where the wide decimal's refusal lands and for the same
reason, so the two agree.

Three paths are closed on purpose rather than by accident. The vector
layer has no lane for sixteen bytes and would have handed the column
back as a string vector, a number dressed as text, so check_col
refuses it the way it already refuses a zoned and a decimal column.
sqlite's INTEGER is sixty four bits, so a staged INT128 column would
have lost its top half silently, and the converter refuses instead.
ColBuf::for_type matches on the exact declared type and so has no arm
for it, which refuses a bulk append when the appender opens.

The declared form and the column form stay one encoding, since the new
extended code is written by extended_bytes and both go through it, so
an INT128 a graph type may name is an INT128 a column can hold.

Directory version 13. A file written before it has no column of this
type to read, so nothing older changes shape.

Left open: DECIMAL(19..=38, s), which is now nearly free on this
plane, being the same sixteen bytes plus the scale from the declared
type.
@tamnd
tamnd merged commit 1e810b6 into main Aug 25, 2026
39 of 40 checks passed
@tamnd
tamnd deleted the s2-int128-column branch August 25, 2026 08:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant