S2: typesafety, the write path enforces the declaration and the encodings it buys

[tamnd]

Milestone S2 of the schema program (Spec/2064g/schema/00-overview.md). Specs: schema/05, schema/06 §6, schema/02 §5. Depends on S1. Unblocks S3 and S6.

This is the core of the program. Everything else is arranged around it, and its first item is deliberately the one S6 needs.

A closed graph type, feature GG02, is the only construct in GQL that licenses a storage optimisation, because it is the only promise that a property nobody declared will not turn up. zu claims GG02 and currently spends it on one check, promised at crates/zu/src/declare.rs:301. Neo4j's graph types are open only, so Neo4j cannot spend it at all. That is the whole advantage and S2 is where it gets collected.

Two halves. The stored encodings that a declaration should buy, and the enforcement that makes the declaration true.

The encodings

Ordered by value, which is the order they get built. The rule they all serve is in schema/06 §2: the catalog holds what the user declared, each block holds the narrowest encoding that losslessly represents its own values, and the reader promotes on the way out. Half of this exists already, in the split between TYPE_CODES at crates/zu-zu1/src/props.rs:76 and extended_bytes at props.rs:261. S2 generalises it.

1. FixedList(n) and BoundedList(n) over a fixed width element. LIST<T>(n) bounds the maximum, not the length, so a list of 700 is legal in a (768) column and a fixed stride is not guaranteed by the declaration alone. Two encodings, and the block directory says which. A batch of embeddings always lands in FixedList: no outer mask because of the outer NOT NULL, no child mask because of the element NOT NULL, no offsets because every list is exactly n, so the column is n times 3072 contiguous bytes that can be mmapped and handed to a SIMD dot product as a flat &[f32]. BoundedList keeps offsets but sizes them to ceil(log2(n)) bits instead of 32.
2. Fixed width Bytes(n). BINARY(16) NOT NULL is the standard GQL spelling of a UUID and it should be sixteen bytes at a fixed stride with no mask and no offsets. Also hashes, IPv6 addresses, and binary quantised vectors.
3. Fixed width Str(n,n), which is CHAR(n). Stride n bytes, UTF-8 length checked on write.
4. ZonedDatetime and ZonedTime column codes. Both are declarable today and neither has a column code, so the declarable/storable asymmetry is not only about exotic types.
5. Decimal(p,s). i64 mantissa where p is at most 18, i128 above, scale in the catalog rather than per value. This is the money story.
6. Int128, Int256, UInt128. Fixed stride, large ids and hashes as numbers.
7. Float16. Two bytes, quantised ML scalars.
8. Shredded Record. One child column per field, recursively. Structs, geospatial points, and JSON with a stable shape. The largest item here, and the one schema/06 §7 says may need a format bump. If child column addressing does not fit the existing table of columns shape in the directory, this item moves to S6 and S2 ships without it. That is a decision to make early, not late.

Items 1 through 7 ride the existing format. New column type codes are extensible by construction and a reader meeting an unknown code already refuses cleanly, which is a rule props.rs already states and this milestone does not weaken.

The enforcement

Plan time, before a row moves: an undeclared property name with a suggestion, a literal whose type is not assignable to the declared type, a null literal into a NOT NULL property, an edge pattern no element type can type. These are 42002, 22G03, 22004 and 42002 respectively, and none of them should cost a row read.

Write time, vectorised, once per batch and never once per value:

• null into NOT NULL: a popcount over the batch validity mask, 22004
• string longer than the declared bound: a max over the length array, 22001 for the error case and 01004 with a warning where truncation is the defined behaviour
• numeric out of the declared range: a min/max compare over the batch, 22003
• list longer than the declared bound: a max over the offset deltas, 22G0B list data right truncation, which is exactly the embedding dimension check
• null element in a LIST<T NOT NULL>: a popcount over the child mask, 22G0C
• array truncation: 2202F
• record shape mismatch: 22G0U, 22G0X, 22G0Y
• graph type violation on the element as a whole: G2000

Endpoint typing on insert: free in the single element type case because the plan already knows which type it is, and one vectorised pass in the GG24 case where more than one type could hold the labels.

Every condition above gets one corpus case, and the message shapes get golden files.

The gate

Enforcement cannot be turned off. There is no session flag, because a schema that is sometimes true is worse than no schema, and because every optimisation in schema/06 §3 is unsound the moment it can be. That means the cost has to be small enough that nobody wants it off, so the gate is ingest throughput with enforcement on against enforcement off on perf-path-100k, and the ceiling is 3%. Every check above is batch level for that reason. If one of them turns out not to be, it gets redesigned rather than made optional.

Checklist

• FixedList(n) over a fixed width element
• BoundedList(n) over a fixed width element, which is the narrow offsets
• Fixed width Bytes(n)
• Fixed width Str(n,n)
• ZonedDatetime and ZonedTime column codes
• Decimal(p,s), both planes: a lane word of unscaled units to p at most 18 (Store a decimal column as unscaled units on the scalar lane #765), sixteen bytes of them above that (Store the wide decimal on the sixteen byte plane #785), scale in the catalog rather than per value
• Int128, Int256, UInt128. Int128 landed in Store an INT128 column as sixteen bytes at a fixed stride #777; the other two are refused at the declaration because nothing here carries a value of them
• Float16
• Shredded Record, or the decision that it moves to S6
• The declared versus encoding split in the block directory, with the promote edge (the integer tower is done, the rest is not)
• Plan time checks
• Write time checks, a row at a time on the statement path
• Write time checks, vectorised, one pass per batch per column
• Endpoint typing on insert
• The condition mapping of schema/05 §3, one corpus case per row
• Message shapes of schema/05 §4, golden file tested
• The 3% ingest gate on perf-path-100k, published to docs/gql-performance.md
• Bytes per row published for the 768 dim FLOAT32 column and the BINARY(16) column against Neo4j and Ladybug
• The S0 round trip test un-ignored for every type above

Exit

A closed graph type is enforced on every write. The declarable/storable invariant holds for everything except the four types S6 owns. Ingest costs under 3%.

[tamnd]

#727 is merged. It is the declaration half of the first checklist item, and worth separating from the rest of that item because the two halves are two different promises.

LIST<FLOAT32 NOT NULL>[768] is the embedding column and zu used to answer a legal GQL declaration of one with a type this file cannot write. The catalog holds the type now. No column stores one yet, which is the encoding half and is next.

What made this more than a table entry is that the declared form and the column form were the same bytes. A column really does hold a value in every position, so dropping the element's nullability there is right and stays. The catalog is not storing values, it is remembering what somebody wrote, and LIST<FLOAT32> and LIST<FLOAT32 NOT NULL> are two types. The second is the one that asks for a column with no child validity mask, which is the whole difference between an embedding a SIMD kernel can be handed flat and one it cannot, so a catalog that could not tell them apart could not have asked for FixedList in the first place. A declared list goes through the extended form now and carries both the element's nullability and the maximum.

A list of lists fell out of the same change, since the element is written by the function that wrote the list. Both are declarable and neither is storable, which is the shape STRING(1,512), CHAR(2) and BINARY(16) have had all along.

Old files still read, because the two byte list code is still decoded and only new writes take the extended form.

The frontier probe in crates/zu/tests/graph_type.rs moved three rows from spelled and refused to declared, 23 to 25, and the props table that mirrors it moved the same three. Two corpus cases pin the declarations. Two tests that used a list of lists as their example of a type nothing can declare now use a decimal, which still is one.

Next is the encoding: FixedList when every list in the block is exactly n, BoundedList with offsets sized to ceil(log2 n) bits otherwise, and the block directory saying which.

[tamnd]

#747 is merged: the encoding half of item 1 has started, with the element width.

A list column stores one blob per row, and every element of a fixed width type took eight bytes inside it whatever the type was. That came from the scalar lane, which holds every such type in a 64 bit word because the integer cascade encodes words and gets its narrowing from the values it meets. A list row has no cascade under it, so the lane's width was being paid on every element of every row. A 768 dimension FLOAT32 embedding cost 6148 bytes a row against the 3072 the floats need; it now costs 3076, and an INT8 list is an eighth of what it was.

The reader takes both forms and needs no version passed to it, because a lane row is a count and a run of equal elements: the old length and the new one differ on every row that holds an element unless the natural width is already eight, and where they agree the two forms are the same bytes. The props directory version still goes to 7, so an older build refuses a newer file where it opens it rather than deeper in a read.

A word too wide for its element type is now refused at write time rather than cut. That is small but it is the first place a list column enforces its own element type, which is what the rest of this milestone is about.

Item 1 stays open. What is left is the layout: the surviving 4 bytes are a per-row count a FixedList(n) does not need, and the blob per row framing is an offset per row it does not need either. n * w * rows contiguous is the target and 3072 is the number. That wants a third structural layout beside MiniBlock and FullZip, which is the next PR.

[tamnd]

#749 is merged, and it is the layout half of item 1.

A list column is a column of byte rows, and it was going through FullZip, the layout for rows whose lengths differ: a len: u32 zipped with every row, a 12 byte index entry per chunk, and one FSST symbol table trained on a sample of the column and then applied to every chunk and compared against plain. On embeddings that last part is a full scan of the column for an answer known before it starts, because float bytes do not compress.

Stride is a third structural layout with nothing in it: a stride: u32, then value_count * stride bytes of rows end to end. Row i begins at 4 + i * stride, so there is no index to walk and nothing to decompress. A list column takes it when every row of it is the same length, and which layout a column took is in its meta, so no caller has to remember and a column that stops being uniform becomes FullZip at the next write with nothing to migrate. The id is 4 rather than 2 because the structural byte carries the sorted flag in bit 1. No props version bump: an unknown structural id is already refused by name.

The rule is lists only for now. Uniform length means Stride is right for numbers and for opaque bytes and wrong for text, since a column of equal length currency codes is uniform and compresses well under FSST. Bytes(n) and Str(n,n) are their own items further down this list and they arrive with the write time length checks that make them uniform by construction rather than by accident.

Where item 1 stands. A 768 dimension FLOAT32 row is 3076 bytes contiguous, from 6148 plus framing before #747. What is left of it is 4 bytes a row: the per row count, which a FixedList(n) does not need because n is in the catalog. That is not a layout change, it is an instance of the declared versus encoding split of schema/06 section 2, where the block drops what the declaration already says, so it lands with that item rather than with this one. BoundedList(n) and its narrow offsets are in the same place, since the bound that sizes the offsets is a declaration too. Item 1 stays open until both are done, and 3072 is the number it closes on.

[tamnd]

#752 is merged and it closes items 2 and 3, with one correction to what item 3 assumed.

A column's type came from the values handed to the writer, and values always spell the unbounded STRING or BYTES. So BINARY(16) was declarable in the catalog, where the extended type encoding has always been able to carry it, and unstorable in a column, where no entry could. Props version 8 lets a column entry carry the extended form, which is the whole of the bump, and it is the first place the declared versus encoding split of schema/06 section 2 is a fact at the storage layer rather than a plan. The writer takes the declaration alongside the values, refuses any row the declaration does not admit before a byte is written, and where the declaration fixes the octet count lays the column out with the Stride layout from #749. A BINARY(16) column is sixteen bytes a row plus the four the stride word costs once.

The correction. Item 3 above says CHAR(n) is "stride n bytes, UTF-8 length checked on write", and the first half of that is not right. STRING(5,5) is five Unicode scalar values, which is anywhere from five to twenty bytes, so a character bound fixes nothing about the row and cannot fix a stride. An octet bound is a layout and a character bound is a check. CHAR(n) is therefore enforced by counting characters and the column stays FullZip, which is where it wants to be anyway, since FSST is the right thing for text and a column of equal length currency codes compresses well. The two checks share a code path and differ only in what they count, and a test pins the difference on a string whose character count and octet count disagree.

A null row still keeps a position in a blob column, and what sits in it is a placeholder nothing reads, conventionally empty. In a fixed octet column an empty placeholder makes the rows ragged and drops the whole column back to FullZip, so a single null would undo the encoding for every row beside it. The writer pads the placeholder to the declared width and the fold does the same when a value becomes NULL in place. The mask still says the row is absent; the bytes under it are just the width the column claims. The fold also refuses a wrong length value on both the node and the edge path, whatever wrote it, because a column whose declared type does not describe its own rows is worse than a refused write: every reader after it takes the type at its word and reads the next row's bytes.

What is deliberately still open. The storage layer can hold these columns and nothing can yet ask it to. ColBuf::for_type matches only the exact unbounded Str and Bytes, so the append path refuses a bounded column. Ingest compares the arriving values' type against the stored one and so refuses one too, until it carries the declaration the same way. And PropertyDef exists only in the AST, so CREATE GRAPH TYPE does not create a typed column at all. Those belong to "plan time checks" and "write time checks, vectorised, one pass per batch per column" further down this list, and they are what turns a storable type into a reachable one. Until they land, the bytes per row item cannot be measured against Neo4j or Ladybug either, because tamnd/graph-bench has no statement to drive it with. The numbers the storage layer produces today are 3076 for the 768 dimension FLOAT32 row and 16 for the BINARY(16) id.

[tamnd]

#754 is merged and it closes FixedList(n), so item 1 is split: the fixed case is done and the narrow offsets of the genuinely variable bounded case are their own line now.

3072 bytes a row. That is the number this item opened with. The path to it was 6148 plus a len: u32 a row, a 12 byte chunk index entry, and an FSST symbol table trained on float bytes to learn they do not compress; then 3076 after #747 wrote the element at its own width and #749 laid the column out end to end; and now 3072, because a column declared LIST<FLOAT32>(768) has said the count once and the block does not have to say it again on every row. The column is the floats and nothing else, plus four bytes of stride word for the whole of it, and it can be handed to a SIMD dot product as a flat &[f32] with no gather.

Two things had to be settled, and they are the general shape of the declared versus encoding split rather than anything about lists.

The declaration alone does not license the drop. LIST<T>(n) bounds the maximum, so a column of 700 element lists is a legal column of that type and its rows still have to say how long they are. Whether a given block dropped the count is a property of the block and not of the type, decided per write from the values, which is the FixedList against BoundedList distinction schema/06 section 3.3 already drew.

And it cannot be recovered from the bytes. Version 7's element width can be, which is why that one needed no version passed to the reader: a lane row is a count and a run of equal elements, so the eight byte form and the natural one differ in length on every row that holds an element. That does not work here, because a bounded LIST<INT32>(768) row of 767 counted elements is the same 3072 bytes as one of 768 uncounted ones. So the count a block was written at is a field in the column's directory entry, beside the type it was written under, and the two together are one statement: what the column was declared, and what the block kept of it. That field is the encoding half of schema/06 section 2's table made real. A count that does not describe the payload beside it is refused at decode, before any row is named, because every row offset follows from it.

A null row is the bound in zero elements rather than an empty list, so one null does not push every row beside it back into the counted form, which is the rule #752 arrived at from the other side for a fixed octet column. The fold keeps the width on the way in for the same reason it keeps a fixed octet count: rows that carry no length of their own are rows nothing can read back if one of them is the wrong size.

What is left on the new line is BoundedList(n): lengths that differ but none past the bound, where the count stays in the row and wants ceil(log2(n)) bits rather than 32. It rides the existing format and it is worth much less than what has landed, since a batch of embeddings always lands in the fixed case.

Same caveat as #752. No GQL surface can declare a bounded list column yet, so this is reachable from the storage API and not from a statement, and the bytes per row item below cannot be measured against Neo4j or Ladybug until the plan time and write time items land.

[tamnd]

#755 is merged. It is the first item on this list that a user can see from a statement rather than from the storage API, and it turned out to be blocked on something none of us had looked at.

BINARY(16) and STRING(5,5) were storable after #752 and nothing could ask for them. The block was in declare.rs, the module that makes a table for a label no table carries yet. It refused outright in any graph whose type is closed, describing label or not, so a property type written in CREATE PROPERTY GRAPH TYPE reached the catalog and went no further. A described label now makes its table out of the declaration, which is the better answer than the pattern in three separate ways: it says BINARY(16) where a written value can only say a run of bytes was written, it says something for the properties whose values the plan has yet to work out, and it says it in the order the graph type wrote them. A graph that promised nothing is unchanged.

Three things fell out of doing it, and the first is why the plan time item was worth more than it looked.

GQL INT is INT32 in zu and a lane word is 64 bits wide. So until this PR no declared table could be created at all, whatever it declared, because the store compared the declaration against the arm the values arrived in and they disagreed. The fix is the promote edge for the integer tower, which is why that line now reads as half done. Every integer type rides the one lane, and the word is the same word whatever width reads it, so INT32 is not another encoding but a promise about the values: the catalog holds the declaration, the block holds the word, and what the store checks is that no row falls outside. Floats do not work this way, since an f32's IEEE bits are not a half of an f64's, so a narrow float declaration is admitted only for a column with no row in it yet, which is the column a table being made has. The remaining towers are the wide ones and Decimal, and they have no lane of their own, so they are a different job.

The declared bound was a label and not a check. insert::cell matched a bytes value against a bytes column with a wildcard and never read the width, so two octets went into a BINARY(16) column and would have been caught by the fold much later, if at all. That is the row at a time half of the write time item, now split out from the vectorised half, which is still open. The two ends raise two conditions and they are not the same condition: past a maximum is 22001, right truncation, because truncation is what would have to happen for the value to fit, and short of a minimum is 22G03 with the rest of the type refusals. An integer outside its declared width is 22003. A STRING bound counts characters and a BINARY bound counts octets, because five characters are up to twenty bytes.

And a declared type no column holds is now refused by name against the property with 42002, rather than by letting the store refuse a column halfway through building the table. A graph type may name a type no column holds, because naming one is a promise about values and holding one is a layout, so the statement that would need the layout is the right place to say so.

What is not here is the list write path, and there is a decision under it that should be made deliberately. write_columns picks the fixed count with rows.iter().all(...), and all() over zero rows is true, so a declared LIST<FLOAT32>[768] column created empty would take fixed_len = Some(768) and be locked to exactly 768 for the rest of its life. LIST<T>[n] bounds a maximum, so that is a meaning of zu's own for a standard type, which is the one thing we do not do. A test pins the current refusal so nobody arrives at the other answer by accident. The consequence for this issue is that bytes per row for the 768 dimension column still cannot be measured against Neo4j or Ladybug, since nothing can fill one from a statement.

[tamnd]

#758 is merged, so ZonedDatetime and ZonedTime column codes is ticked.

A zoned value is two numbers, an instant and the offset it was written with, and neither one alone is the value. The instants ride the lane, where a comparison and a sort already read them, and the offsets ride a second segment beside them, so two rows written in two zones for one instant are one word in the lane and still each print where they were written. Packing both into a word works for ZONED TIME and not for ZONED DATETIME, which is a full i64 of nanoseconds; a ten byte blob would have needed no new code but interleaves the offset between every timestamp and so takes apart the delta encoding on the most compressible column an analytic graph has.

Because the instant plane simply is the lane, is_lane() stays true for a zoned column and fold, zu check, query and snapshot keep working unchanged. That also admits zoned columns to every one of those paths, so each was checked and closed: the vector layer refuses one in check_col rather than reading the instants as plain integers, a cell write refuses in holds() so a word can never land without its offset, an ingest refuses because a seal is one segment per column and would drop the plane, and the free path, both zu check walks and props_segment_blocks learned about the second segment so its blocks are neither leaked nor reported as leaks.

Props version goes 9 to 10, and a version 9 directory naming a zoned type is refused as corrupt, because before this there was nowhere for the second segment to be written. Zoned stays out of TYPE_CODES on purpose: that table doubles as the list element table, and admitting zoned there would make LIST<ZONED DATETIME> look storable when a list element gets no plane of its own.

Two things stay open and neither is hidden. A statement cannot fill a zoned column: a cell is one word and this value is two numbers, so INSERT refuses with 22G03 naming what is missing, pinned by a_declared_zoned_column_is_not_yet_something_a_statement_can_fill. Making it work means a cell that carries the pair, rippling through the transaction path, the patch machinery, fold and write. The vector layer has no zoned lane either; ColType and TemporalLane in zu-query still have no arm for it.

Next on this list is Decimal(p,s).

[tamnd]

#761 is in. It is groundwork rather than a box on this list: it adds the exact decimal value, not the decimal column, and the Decimal(p,s) box stays open until the column PR lands.

The reason for the split is that there was nothing exact to read a decimal column back into. CAST('1.20' AS DECIMAL(5, 2)) handed back a binary64 and ID034 said so out loud: zu declares no decimal type, a value under that name is held as a float, there is no precision to publish. Adding the column first would have meant a second declarable type in a row that no statement can usefully fill, and the value would have had to land anyway to make it readable. So the value went first.

A decimal is an i128 of unscaled units and a u16 scale. An i128 holds thirty eight digits, which is exactly the largest precision DECIMAL(p, s) accepts, so every decimal a statement can declare fits the carrier and the carrier never has to refuse a value the type admits. The scale rides on the number rather than only on the declared type, because a decimal in a RETURN has no column to ask what scale to print at. Two decimals of one number at two scales are one value: they compare equal, they hash alike, and each prints the way it was written.

Add and subtract answer at the larger of the two scales, multiply at the sum of them, divide at the larger plus six guard digits capped at thirty eight with the last rounded half away from zero. An answer wider than thirty eight digits raises 22003. An integer beside a decimal is read as a decimal of scale nought so mixing them stays exact; a float beside one widens it, and that is where the exactness stops. ABS, SIGN, CEIL, FLOOR and ROUND are done on the digits, so ROUND of 1.005 to two places is 1.01 rather than the 1.00 a float answers.

What the PR forced open, and what the column PR now has to close: ColumnType::Decimal { scale } rides HeldData::Complex rather than a flat lane, because the crate's flat buffers are sixty four bits wide and this is a hundred and twenty eight; Arrow gets a proper Decimal128Array at the column's scale, which is the widest scale any row in it was written with. A decimal cannot reach the pipeline executor yet, because the only way to make one is a cast and a cast that is not a widening keeps its plan on the row engine. AVG over decimals answers a float, the same as it does over integers.

Deliberately still open and not decided in the margin of that PR: literal typing. An unsuffixed 1.5 and an M-suffixed literal are both still approximate numeric literals, which is what ID079 says. Whether ISO 39075 means those to be exact is a real conformance question and reversing it moves a lot of statements, so it wants its own change.

The C ABI gained tag 14 and zu_value_decimal, which splits the unscaled integer into a high and a low half because C has no portable hundred and twenty eight bit integer. Nothing that was numbered moved, so the revision went 0.14 to 0.15 and a binding built against 0.14 keeps working. ID034, ID057 and ID063 through ID067 are corrected and the register regenerated.

Next: the decimal column, which is the PR that ticks the box above.

[tamnd]

Decimal(p,s) is done, in #765, and the box is ticked. It took two pull requests because the value and the column are two different problems: #761 gave the engine an exact decimal to compute with, and #765 gives it a column to keep one in.

The column stores the unscaled integer on the scalar lane and the catalog stores the scale. A column of DECIMAL(12,2) is a column of pence, and the declared type is what turns a word of a hundred and twenty of them back into 1.20. That is the declared versus encoding split of schema/06 section 2 at its plainest, and it makes this the first column type whose declaration is needed to read a row back rather than only to check one. Every other lane column reads back the same value whether or not you know how it was declared. This one does not, which is why the props version goes to 11.

Where the frontier falls was not a number anyone chose in this change. LogicalType::physical already mapped a decimal by IntBits::for_digits, so lane_width already answered: four bytes to nine digits, eight to eighteen, nothing above. Eighteen digits of unscaled units ride a 64 bit word and nineteen do not, so DECIMAL(18,2) is storable and DECIMAL(38,2) stays declarable in the grammar and refused by the catalog. That is the same shape INT128 and FLOAT16 have on this list. Every test that used DECIMAL(12,2) as its example of a type the file cannot write now uses DECIMAL(38,2), and the frontier table in graph_type.rs has one type on both sides of it for the first time, which is what says the argument decides rather than the type.

The lane bound lives in extended_bytes, which both the declared form and the column form go through, so the two stay one encoding. A decimal a graph type may name is a decimal a column can hold, and the refusal lands at the declaration where the user wrote it rather than at the first insert.

A value the column cannot write exactly at its own scale is refused with 22003 rather than rounded. CAST('1.234' AS DECIMAL(6,3)) into a two place column raises. Rounding a price on the way into a ledger is the mistake this type exists to stop, and a caller who wants it rounded has ROUND to say so with. A value wider than the declared precision is refused the same way, on the statement path and again at the column writer where a bulk caller meets it. Going in, a plain integer is taken as well as a decimal, since 7 is a value of DECIMAL(12,2). Coming out, the value carries the column's scale, so a 0.5 written into a two place column reads back as 0.50.

Two hazards were closed deliberately rather than left for whichever path reached them first. bounded_list now refuses a decimal element: it asks lane_width, a decimal answers, and the element reader has no decimal arm, so a bounded list of prices would have gone in and come back as a list of integers of pence. check_col now refuses a decimal column on the vector read path for the same reason. Both would have been wrong answers rather than missing ones.

Left open and named: decimals as list elements, the bulk append path where ColBuf::for_type returns nothing for a decimal, the vector path, and precisions above eighteen digits, which want a plane of their own the way a zoned column has one, or a 128 bit lane.

The gate was green locally before the merge: fmt, clippy with -D warnings, xtask terms, xtask api-map, and 89 test result: ok lines across the six test targets.

Next on this list is BoundedList(n) over a fixed width element with narrow offsets, which is the other half of the embedding column.

[tamnd]

BoundedList(n) is done, merged in #770 as 31bb59b. That closes item 1 of the encodings: FixedList(n) landed earlier and this is the ragged half.

The count is sized to the declared bound. A column every row of which is at the bound already drops the count out of the rows entirely and keeps it in the directory, which is the case S6 wants and the reason a 768 dimension embedding row is 3072 bytes. A column whose rows are ragged has to keep a per-row count, because a row of 700 in a (768) column has to say it is 700. But the count is a number between nought and the bound, check_declared has already checked every row against that bound, and a number that cannot pass 255 does not need four bytes to say. So the head is one byte to 255, two to 65535, and four above, and ListRows replaced the Option<usize> the encoder and reader took: Fixed(n) is the directory holding the count, Counted(w) is the row holding it in w bytes.

The width is carried in the column entry rather than derived from the declared bound, and that is the decision in this change worth writing down. A fold rewrites the directory at the current version while leaving the row blobs where they are, so a column written at version 11 comes back through encode with a version 12 header over rows that still spend four bytes on their counts. Deriving the width from the bound would read those rows as a small count followed by an enormous number of elements, which is to say it would report damage to a file that has none. fold.rs copies the width off the old entry the way it already copies fixed_len, and a test pins the combination that makes the two answers differ: a bound of four with rows that say their counts in four bytes.

Where the saving is real: small elements with a small bound, which is where the count was the larger part of the row. A LIST<INT8>(4) row of four elements was four bytes of payload and four more to say so, and is now four and one. Where it is beneath noticing: an embedding column, where the same change would be two bytes in three thousand, and where in practice it does not arise at all because those rows carry no count. A bounded list of a variable width element keeps its four bytes too, since the element already carries its own length and the row is not fixed stride either way, and an unbounded list keeps them for want of a number to be smaller than.

Directory version 12. A version 11 directory decodes to a width of four and its rows read unchanged. A width byte that is not one, two or four is refused as corrupt, because that byte decides every element offset below it.

Still standing behind this: nothing writes a list from a statement, only a bulk load, so the ragged case is reachable through store_props_nullable and not through INSERT. That is the list write path, which is on the S2 tail and not this item.

Next on the checklist: Int128, Int256, UInt128.