diff --git a/table-spec/schema/README.md b/table-spec/schema/README.md new file mode 100644 index 0000000..f18908e --- /dev/null +++ b/table-spec/schema/README.md @@ -0,0 +1,142 @@ + + +# Schema decoding + +Reading a schema JSON should produce the same schema in every implementation. +This surface covers the schema document: the field tree and its id spaces, +column order, required flags, `doc`, `schema-id` and `identifier-field-ids`. +Type-string grammar, including the parsing and rejection of parameterized types +such as `decimal(P, S)` and `fixed[L]`, is covered by the types surface under +`table-spec/types/`. + +## Assertion + +``` +project(parse(input)) == decoded +``` + +Bytes are not compared. The spec fixes no key order in a schema JSON, and does +not say whether an absent optional field is written out, so two byte-different +schema JSONs can be the same schema. `decoded` is the comparable form. + +## Scope + +This surface reads the schema JSON object and nothing around it. Writing a schema +back out is out of scope for this surface. Whether a type is legal at a given +format version cannot be decided here, because a schema JSON carries no version. +Type to format-version conformance is out of scope and is best verified at a +higher level. + +## Inputs + +`core/` holds the shape cases: nesting, every id space, column order, `doc`, +`schema-id`, `identifier-field-ids`, and every v1 and v2 primitive type. Every +input in `core/` is valid writer output for format versions 1, 2 and 3. + +`unknown/` holds the cases for the `unknown` type, which must be optional. + +Consumers subscribe by subdirectory. An implementation with no `unknown` type +runs `core/` only and names `unknown/` as excluded in its own configuration. + +## Case format + +Each file holds a `cases` array. An example: + +```json +{ + "id": "map-of-primitive", + "valid": true, + "input": { + "type": "struct", "schema-id": 0, + "fields": [ + { "id": 1, "name": "id", "required": true, "type": "long" }, + { "id": 2, "name": "props", "required": true, "type": { + "type": "map", "key-id": 3, "key": "string", + "value-id": 4, "value-required": false, "value": "int" } } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "id", "parent": null, "required": true, "type": "long", "doc": null }, + { "id": 2, "name": "props", "parent": null, "required": true, "type": "map", "doc": null }, + { "id": 3, "name": "key", "parent": 2, "required": true, "type": "string", "doc": null }, + { "id": 4, "name": "value", "parent": 2, "required": false, "type": "int", "doc": null } + ] + } +} +``` + +- `id` is a stable name for the case, used to label failures. +- `valid` is whether parsing should succeed. A `valid: false` case has no + `decoded`. +- `input` is a schema JSON. +- `decoded` is the expected result of parsing it: `schema-id`, + `identifier-field-ids`, and `fields`, one row per field id in document order. + +Each row of `decoded.fields` is: + +- `id`, the field id. List elements, map keys and map values carry `element-id`, + `key-id` and `value-id`. +- `name`, the field name. Elements, keys and values are named `element`, `key` + and `value`. +- `parent`, the enclosing field's id, or `null` at the top level. +- `required`, the field's required flag. Map keys are always `true`. +- `type`. A nested type is `struct`, `list` or `map`, and its children are their + own rows. A parameterized type is an object holding the type name and its + decoded parameters: `decimal` carries `precision` and `scale`, and `fixed` + carries `length`. + Every other type is the type name from the Appendix C types table. +- `doc`, the field's doc string, or `null`. + +## Procedure + +The cases in each subscribed subdirectory are worked through one at a time. + +The schema in `input` is read through the same path the implementation uses to +read a schema out of table metadata. + +A case marked `"valid": false` holds a malformed schema, and the read is expected +to fail. The case passes if it does. If the read succeeds instead, the case +fails, and the parser is more lenient than the spec. + +Otherwise the read is expected to succeed. The resulting schema is turned into +the row list described above, with each value read through the implementation's +public accessors. Serializing the schema and reading values back out of the JSON +checks the writer rather than the reader, so it does not satisfy the case. The +row list is then compared against `decoded`. + +The comparison is exact. A value differing only in whitespace or letter case is +still a difference, and the case fails. Row order is preserved, because that +order is the column order the schema asserts, and sorting rows by id discards the +assertion. `identifier-field-ids` is the one exception, and is compared as a set, +which is what the spec calls it. + +Every difference in a case is reported, not only the first, so that one run shows +everything that diverged. A report names the subdirectory, the case and the +value, reading like `core/list-of-struct fields[2].parent`. + +## Notes + +Expected values are JSON. Field ids, `schema-id` and `parent` are written as +numbers, because the spec types field ids as `int` and caps them below +`Integer.MAX_VALUE - 200`. `parent` and `doc` are written as `null` where they +have no value, and `null` there is a decoded value, not a missing assertion. diff --git a/table-spec/schema/core/cases.json b/table-spec/schema/core/cases.json new file mode 100644 index 0000000..ad7d4eb --- /dev/null +++ b/table-spec/schema/core/cases.json @@ -0,0 +1,836 @@ +{ + "cases": [ + { + "id": "flat-required-and-optional", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "data", + "required": false, + "type": "string" + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "id", "parent": null, "required": true, "type": "long", "doc": null }, + { "id": 2, "name": "data", "parent": null, "required": false, "type": "string", "doc": null } + ] + } + }, + { + "id": "fields-not-in-id-order", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 3, + "name": "c", + "required": true, + "type": "int" + }, + { + "id": 1, + "name": "a", + "required": true, + "type": "int" + }, + { + "id": 2, + "name": "b", + "required": false, + "type": "int" + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 3, "name": "c", "parent": null, "required": true, "type": "int", "doc": null }, + { "id": 1, "name": "a", "parent": null, "required": true, "type": "int", "doc": null }, + { "id": 2, "name": "b", "parent": null, "required": false, "type": "int", "doc": null } + ] + } + }, + { + "id": "nested-struct", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "location", + "required": false, + "type": { + "type": "struct", + "fields": [ + { + "id": 3, + "name": "lat", + "required": true, + "type": "double" + }, + { + "id": 4, + "name": "lon", + "required": true, + "type": "double" + } + ] + } + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "id", "parent": null, "required": true, "type": "long", "doc": null }, + { "id": 2, "name": "location", "parent": null, "required": false, "type": "struct", "doc": null }, + { "id": 3, "name": "lat", "parent": 2, "required": true, "type": "double", "doc": null }, + { "id": 4, "name": "lon", "parent": 2, "required": true, "type": "double", "doc": null } + ] + } + }, + { + "id": "list-of-primitive", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "tags", + "required": true, + "type": { + "type": "list", + "element-id": 3, + "element-required": false, + "element": "string" + } + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "id", "parent": null, "required": true, "type": "long", "doc": null }, + { "id": 2, "name": "tags", "parent": null, "required": true, "type": "list", "doc": null }, + { "id": 3, "name": "element", "parent": 2, "required": false, "type": "string", "doc": null } + ] + } + }, + { + "id": "map-of-primitive", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "props", + "required": true, + "type": { + "type": "map", + "key-id": 3, + "key": "string", + "value-id": 4, + "value-required": false, + "value": "int" + } + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "id", "parent": null, "required": true, "type": "long", "doc": null }, + { "id": 2, "name": "props", "parent": null, "required": true, "type": "map", "doc": null }, + { "id": 3, "name": "key", "parent": 2, "required": true, "type": "string", "doc": null }, + { "id": 4, "name": "value", "parent": 2, "required": false, "type": "int", "doc": null } + ] + } + }, + { + "id": "list-of-struct", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "points", + "required": true, + "type": { + "type": "list", + "element-id": 3, + "element-required": true, + "element": { + "type": "struct", + "fields": [ + { + "id": 4, + "name": "x", + "required": true, + "type": "int" + }, + { + "id": 5, + "name": "y", + "required": false, + "type": "int" + } + ] + } + } + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "id", "parent": null, "required": true, "type": "long", "doc": null }, + { "id": 2, "name": "points", "parent": null, "required": true, "type": "list", "doc": null }, + { "id": 3, "name": "element", "parent": 2, "required": true, "type": "struct", "doc": null }, + { "id": 4, "name": "x", "parent": 3, "required": true, "type": "int", "doc": null }, + { "id": 5, "name": "y", "parent": 3, "required": false, "type": "int", "doc": null } + ] + } + }, + { + "id": "map-with-struct-value", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "by_name", + "required": false, + "type": { + "type": "map", + "key-id": 3, + "key": "string", + "value-id": 4, + "value-required": false, + "value": { + "type": "struct", + "fields": [ + { + "id": 5, + "name": "count", + "required": true, + "type": "long" + } + ] + } + } + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "id", "parent": null, "required": true, "type": "long", "doc": null }, + { "id": 2, "name": "by_name", "parent": null, "required": false, "type": "map", "doc": null }, + { "id": 3, "name": "key", "parent": 2, "required": true, "type": "string", "doc": null }, + { "id": 4, "name": "value", "parent": 2, "required": false, "type": "struct", "doc": null }, + { "id": 5, "name": "count", "parent": 4, "required": true, "type": "long", "doc": null } + ] + } + }, + { + "id": "schema-id-and-identifier-fields", + "valid": true, + "input": { + "type": "struct", + "schema-id": 7, + "identifier-field-ids": [ + 1, + 2 + ], + "fields": [ + { + "id": 1, + "name": "tenant", + "required": true, + "type": "string" + }, + { + "id": 2, + "name": "key", + "required": true, + "type": "long" + }, + { + "id": 3, + "name": "payload", + "required": false, + "type": "string" + } + ] + }, + "decoded": { + "schema-id": 7, + "identifier-field-ids": [1, 2], + "fields": [ + { "id": 1, "name": "tenant", "parent": null, "required": true, "type": "string", "doc": null }, + { "id": 2, "name": "key", "parent": null, "required": true, "type": "long", "doc": null }, + { "id": 3, "name": "payload", "parent": null, "required": false, "type": "string", "doc": null } + ] + } + }, + { + "id": "field-doc", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long", + "doc": "primary key" + }, + { + "id": 2, + "name": "data", + "required": false, + "type": "string" + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "id", "parent": null, "required": true, "type": "long", "doc": "primary key" }, + { "id": 2, "name": "data", "parent": null, "required": false, "type": "string", "doc": null } + ] + } + }, + { + "id": "all-primitive-types", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "f_boolean", + "required": false, + "type": "boolean" + }, + { + "id": 2, + "name": "f_int", + "required": false, + "type": "int" + }, + { + "id": 3, + "name": "f_long", + "required": false, + "type": "long" + }, + { + "id": 4, + "name": "f_float", + "required": false, + "type": "float" + }, + { + "id": 5, + "name": "f_double", + "required": false, + "type": "double" + }, + { + "id": 6, + "name": "f_date", + "required": false, + "type": "date" + }, + { + "id": 7, + "name": "f_time", + "required": false, + "type": "time" + }, + { + "id": 8, + "name": "f_timestamp", + "required": false, + "type": "timestamp" + }, + { + "id": 9, + "name": "f_timestamptz", + "required": false, + "type": "timestamptz" + }, + { + "id": 10, + "name": "f_string", + "required": false, + "type": "string" + }, + { + "id": 11, + "name": "f_uuid", + "required": false, + "type": "uuid" + }, + { + "id": 12, + "name": "f_fixed", + "required": false, + "type": "fixed[16]" + }, + { + "id": 13, + "name": "f_binary", + "required": false, + "type": "binary" + }, + { + "id": 14, + "name": "f_decimal", + "required": false, + "type": "decimal(9, 2)" + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "f_boolean", "parent": null, "required": false, "type": "boolean", "doc": null }, + { "id": 2, "name": "f_int", "parent": null, "required": false, "type": "int", "doc": null }, + { "id": 3, "name": "f_long", "parent": null, "required": false, "type": "long", "doc": null }, + { "id": 4, "name": "f_float", "parent": null, "required": false, "type": "float", "doc": null }, + { "id": 5, "name": "f_double", "parent": null, "required": false, "type": "double", "doc": null }, + { "id": 6, "name": "f_date", "parent": null, "required": false, "type": "date", "doc": null }, + { "id": 7, "name": "f_time", "parent": null, "required": false, "type": "time", "doc": null }, + { "id": 8, "name": "f_timestamp", "parent": null, "required": false, "type": "timestamp", "doc": null }, + { "id": 9, "name": "f_timestamptz", "parent": null, "required": false, "type": "timestamptz", "doc": null }, + { "id": 10, "name": "f_string", "parent": null, "required": false, "type": "string", "doc": null }, + { "id": 11, "name": "f_uuid", "parent": null, "required": false, "type": "uuid", "doc": null }, + { "id": 12, "name": "f_fixed", "parent": null, "required": false, "type": { "type": "fixed", "length": 16 }, "doc": null }, + { "id": 13, "name": "f_binary", "parent": null, "required": false, "type": "binary", "doc": null }, + { "id": 14, "name": "f_decimal", "parent": null, "required": false, "type": { "type": "decimal", "precision": 9, "scale": 2 }, "doc": null } + ] + } + }, + { + "id": "decimal-max-precision", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "amount", + "required": false, + "type": "decimal(38, 10)" + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "id", "parent": null, "required": true, "type": "long", "doc": null }, + { "id": 2, "name": "amount", "parent": null, "required": false, "type": { "type": "decimal", "precision": 38, "scale": 10 }, "doc": null } + ] + } + }, + { + "id": "reject-identifier-field-does-not-exist", + "valid": false, + "input": { + "type": "struct", + "schema-id": 0, + "identifier-field-ids": [ + 999 + ], + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + } + ] + } + }, + { + "id": "reject-identifier-optional-field", + "valid": false, + "input": { + "type": "struct", + "schema-id": 0, + "identifier-field-ids": [ + 2 + ], + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "note", + "required": false, + "type": "string" + } + ] + } + }, + { + "id": "reject-identifier-double-field", + "valid": false, + "input": { + "type": "struct", + "schema-id": 0, + "identifier-field-ids": [ + 2 + ], + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "score", + "required": true, + "type": "double" + } + ] + } + }, + { + "id": "reject-identifier-float-field", + "valid": false, + "input": { + "type": "struct", + "schema-id": 0, + "identifier-field-ids": [ + 2 + ], + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "score", + "required": true, + "type": "float" + } + ] + } + }, + { + "id": "reject-identifier-non-primitive-field", + "valid": false, + "input": { + "type": "struct", + "schema-id": 0, + "identifier-field-ids": [ + 2 + ], + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "loc", + "required": true, + "type": { + "type": "struct", + "fields": [ + { + "id": 3, + "name": "x", + "required": true, + "type": "long" + } + ] + } + } + ] + } + }, + { + "id": "reject-identifier-nested-in-list", + "valid": false, + "input": { + "type": "struct", + "schema-id": 0, + "identifier-field-ids": [ + 4 + ], + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "rows", + "required": true, + "type": { + "type": "list", + "element-id": 3, + "element-required": true, + "element": { + "type": "struct", + "fields": [ + { + "id": 4, + "name": "k", + "required": true, + "type": "string" + } + ] + } + } + } + ] + } + }, + { + "id": "reject-identifier-nested-in-map-value", + "valid": false, + "input": { + "type": "struct", + "schema-id": 0, + "identifier-field-ids": [ + 5 + ], + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "rows", + "required": true, + "type": { + "type": "map", + "key-id": 3, + "key": "string", + "value-id": 4, + "value-required": true, + "value": { + "type": "struct", + "fields": [ + { + "id": 5, + "name": "k", + "required": true, + "type": "string" + } + ] + } + } + } + ] + } + }, + { + "id": "reject-identifier-map-key", + "valid": false, + "input": { + "type": "struct", + "schema-id": 0, + "identifier-field-ids": [ + 3 + ], + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "props", + "required": true, + "type": { + "type": "map", + "key-id": 3, + "key": "string", + "value-id": 4, + "value-required": true, + "value": "string" + } + } + ] + } + }, + { + "id": "identifier-nested-in-required-struct", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "identifier-field-ids": [ + 3 + ], + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "key", + "required": true, + "type": { + "type": "struct", + "fields": [ + { + "id": 3, + "name": "part", + "required": true, + "type": "string" + } + ] + } + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [3], + "fields": [ + { "id": 1, "name": "id", "parent": null, "required": true, "type": "long", "doc": null }, + { "id": 2, "name": "key", "parent": null, "required": true, "type": "struct", "doc": null }, + { "id": 3, "name": "part", "parent": 2, "required": true, "type": "string", "doc": null } + ] + } + }, + { + "id": "reject-identifier-nested-in-optional-struct", + "valid": false, + "input": { + "type": "struct", + "schema-id": 0, + "identifier-field-ids": [ + 3 + ], + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "loc", + "required": false, + "type": { + "type": "struct", + "fields": [ + { + "id": 3, + "name": "x", + "required": true, + "type": "long" + } + ] + } + } + ] + } + } + ] +} diff --git a/table-spec/schema/unknown/cases.json b/table-spec/schema/unknown/cases.json new file mode 100644 index 0000000..31c4d87 --- /dev/null +++ b/table-spec/schema/unknown/cases.json @@ -0,0 +1,56 @@ +{ + "cases": [ + { + "id": "unknown-optional", + "valid": true, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "future", + "required": false, + "type": "unknown" + } + ] + }, + "decoded": { + "schema-id": 0, + "identifier-field-ids": [], + "fields": [ + { "id": 1, "name": "id", "parent": null, "required": true, "type": "long", "doc": null }, + { "id": 2, "name": "future", "parent": null, "required": false, "type": "unknown", "doc": null } + ] + } + }, + { + "id": "reject-unknown-required", + "valid": false, + "input": { + "type": "struct", + "schema-id": 0, + "fields": [ + { + "id": 1, + "name": "id", + "required": true, + "type": "long" + }, + { + "id": 2, + "name": "future", + "required": true, + "type": "unknown" + } + ] + } + } + ] +}