diff --git a/CHANGELOG.md b/CHANGELOG.md index 0273682..d8a46f9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,15 @@ # Changelog +## 0.4.0 - Unreleased + +- Add typed data profile v2 and `NifBytes`, an opaque bounded byte sequence for + images, media, encrypted payloads, and other binary data without Base64 + expansion. +- Preserve UTF-8 validation for `string` and keep typed profile v1 decoding for + existing values. +- Document the boundary between in-memory binary values and application-level + streaming multipart transfers for large attachments. + ## 0.3.1 - 2026-08-10 - Make omitted `CodecLimits` unbounded at the application-policy level; diff --git a/README.md b/README.md index d02e960..a4c1dba 100644 --- a/README.md +++ b/README.md @@ -144,10 +144,10 @@ budget, and tighten pool, string, and container limits when the data model permits it. `CodecLimits` is the enforcement mechanism; choosing these values remains the embedding application's responsibility. -### Typed serializer (v0.3) +### Typed serializer (v0.4) NIFKit can encode and decode supported Nim values using the typed data profile -v1. The BIF APIs construct and read BIF directly, so application code need not +v2. The BIF APIs construct and read BIF directly, so application code need not allocate intermediate NIF text. ```nim @@ -167,6 +167,20 @@ accepts `TypedCodecOptions`. Unknown object fields and type-name mismatches are rejected by default. See [the typed serializer design](docs/typed-serializer-design.md) for the profile, supported types, canonicalization, and compatibility rules. +`NifBytes` represents bounded arbitrary bytes such as a small image, encrypted +payload, or document without Base64 expansion. It is intentionally distinct +from UTF-8 `string`. + +```nim +let thumbnail = initNifBytes(readFile("thumbnail.png")) +let payload = toBif(thumbnail) +``` + +Typed conversion materializes the complete `NifBytes` value. For large images, +videos, or other attachments, keep BIF for structured metadata and use the +application's streaming multipart or equivalent transport facility for the raw +file. Apply separate fixed limits to the metadata and streamed attachment. + Nim applications should call the Nim API directly. Applications may store BIF however they want; semantic interpretation belongs to the embedding application or another NIF/BIF implementation. diff --git a/docs/typed-serializer-design.md b/docs/typed-serializer-design.md index 6418081..4e23023 100644 --- a/docs/typed-serializer-design.md +++ b/docs/typed-serializer-design.md @@ -1,13 +1,14 @@ # Typed serializer design This document defines the general data-exchange profile implemented by the -typed NIF serializer in NIFKit v0.3. It is deliberately separate from compiler NIF ASTs: +typed NIF serializer in NIFKit. It is deliberately separate from compiler NIF ASTs: compiler-specific tags, line information, and symbol indexes are not part of this profile. ## Status and API -The profile is implemented in the v0.3.0 release. Its public API is: +Profile v1 was implemented in v0.3.0. Profile v2 is introduced in v0.4.0. Its +public API is: ```nim proc toNif*[T](value: T; limits = defaultCodecLimits()): string @@ -34,11 +35,15 @@ use supplied limits just as raw NIF/BIF calls do. an intermediate NIF text string. `toBif` constructs BIF directly while preserving the same canonical profile representation as `toNif`. -## Canonical profile +## Canonical profiles -The root is `(nifkit\2Ddata 1 value)`. The escaped hyphen is required by NIF -tag grammar; its decoded tag name is `nifkit-data`. Version `1` identifies -these mapping rules. Writers emit UTF-8 byte strings in canonical NIF escaping, fields in +Profile v1 uses the root `(nifkit\2Ddata 1 value)`. Profile v2 uses +`(nifkit\2Ddata 2 value)` and adds `NifBytes`. The escaped hyphen is required +by NIF tag grammar; its decoded tag name is `nifkit-data`. v0.4 writers emit +profile v2. Readers continue to accept v1 values that do not use v2-only +mappings. + +Writers emit UTF-8 `string` values in canonical NIF escaping, fields in declaration order, and `Table` entries sorted by their canonical encoded key. | Nim value | NIF representation | @@ -48,6 +53,7 @@ declaration order, and `Table` entries sorted by their canonical encoded key. | unsigned integer | decimal integer with `u` suffix | | `float32`, `float64` | canonical finite decimal float; non-finite values are rejected initially | | `string` | NIF string | +| `NifBytes` (profile v2) | `(bytes raw-octets)` | | `char` | NIF character | | `enum` | `(enum "TypeName" "MemberName")` | | `Option[T]` | `(some value)` or `none` | @@ -76,16 +82,28 @@ canonical order rather than its original insertion order. Decoders reject unknown fields by default; set `TypedCodecOptions.allowUnknownFields` to permit them for forward compatibility. -Missing object fields are errors in profile v1. Enum decoding uses member +Missing object fields are errors in profiles v1 and v2. Enum decoding uses member names, never ordinal values, to avoid silently changing meaning when source order changes. Type names are required by default; set `TypedCodecOptions.requireTypeNames` to `false` only for an explicitly managed compatibility boundary. Schema changes require a new root version or an -explicitly declared migration. +explicitly declared migration. `NifBytes` is the v2 mapping; a v1 root that +contains `(bytes ...)` is rejected. + +`NifBytes` is an opaque, bounded octet sequence. It is not UTF-8 validated and +is encoded into BIF string storage without Base64 expansion. When rendered as +NIF text, ASCII control bytes use standard NIF escapes. It is suitable for +image, media, encrypted, or other binary data, but typed conversion fully +materializes it and is not a streaming file-transfer API. + +Applications should use their HTTP stack's streaming multipart or equivalent +facility for large attachments. Keep the BIF metadata small and apply a +separate fixed byte budget while streaming the attachment. Do not increase the +typed BIF metadata limits merely to accommodate a large file. `Option.none`, `nil`, and an absent object field are distinct states. A non-`ref` value cannot decode from `nil`. Reference identity and cycles are out -of scope for profile version 1; ref values are tree-shaped. +of scope for the current profiles; ref values are tree-shaped. ## Error model diff --git a/nifkit.nimble b/nifkit.nimble index c7d9ee4..7b958e2 100644 --- a/nifkit.nimble +++ b/nifkit.nimble @@ -1,4 +1,4 @@ -version = "0.3.1" +version = "0.4.0" author = "puffball1567" description = "Spec-based NIF/BIF toolkit for multiple languages" license = "MIT" diff --git a/src/nifkit/typed_serializer.nim b/src/nifkit/typed_serializer.nim index 28109b8..f855a25 100644 --- a/src/nifkit/typed_serializer.nim +++ b/src/nifkit/typed_serializer.nim @@ -1,4 +1,4 @@ -## Typed NIF data profile v1. This module is intentionally Nim-only. +## Typed NIF data profiles v1 and v2. This module is intentionally Nim-only. import std/[options, strutils, typetraits, math, tables, sets, algorithm, unicode, macros] import ./[codec_limits, nif_encoder, bif_decoder] @@ -6,6 +6,7 @@ import ./[codec_limits, nif_encoder, bif_decoder] const DataRootTag = "nifkit\\2Ddata" DataRootName = "nifkit-data" + CurrentDataProfile = 2 BifKindChar = 1'u32 BifKindString = 2'u32 BifKindInt = 3'u32 @@ -16,11 +17,18 @@ const BifKindExtended = 10'u32 type + ## An opaque sequence of bytes for the typed NIF data profile. + ## + ## Unlike `string`, this value is not required to be UTF-8. It is suitable + ## for images, media, encrypted data, and other binary payloads. + NifBytes* = object + data*: string + TypedCodecOptions* = object allowUnknownFields*: bool requireTypeNames*: bool - DataKind = enum dkAtom, dkString, dkChar, dkCompound + DataKind = enum dkAtom, dkString, dkBytes, dkChar, dkCompound DataNode = ref object kind: DataKind text: string @@ -85,9 +93,33 @@ macro variantDiscriminantName(T: typedesc): untyped = proc defaultTypedCodecOptions*(): TypedCodecOptions = TypedCodecOptions(requireTypeNames: true) +proc initNifBytes*(data: string): NifBytes = + ## Creates an opaque binary value from raw bytes stored in a Nim string. + NifBytes(data: data) + +proc initNifBytes*(data: openArray[byte]): NifBytes = + ## Creates an opaque binary value from a byte sequence. + result.data = newString(data.len) + for index, value in data: + result.data[index] = char(value) + +proc toSeq*(value: NifBytes): seq[byte] = + ## Returns the opaque payload as bytes. + result = newSeq[byte](value.data.len) + for index, ch in value.data: + result[index] = byte(ord(ch)) + proc typedFail(kind: NifKitErrorKind; message, path: string; offset = -1) {.noreturn.} = raiseCodecError(kind, message, offset, path) +proc parseDataProfile(values: seq[DataNode]; offset: int): int = + if values.len != 3 or values[1].kind != dkAtom: + typedFail(nkeUnsupportedDataProfile, "unsupported NIFKit data profile", "$", offset) + case values[1].text + of "1": 1 + of $CurrentDataProfile: CurrentDataProfile + else: typedFail(nkeUnsupportedDataProfile, "unsupported NIFKit data profile", "$", offset) + proc require(node: DataNode; tag, path: string): seq[DataNode] = if node.kind != dkCompound or node.children.len == 0 or node.children[0].kind != dkAtom or @@ -96,10 +128,10 @@ proc require(node: DataNode; tag, path: string): seq[DataNode] = typedFail(nkeTypeMismatch, "expected " & tag, path, node.offset) result = node.children -proc quote(value: string; limits: CodecLimits; path: string): string = +proc quote(value: string; limits: CodecLimits; path: string; requireUtf8 = true): string = if value.len > limits.maxStringBytes: typedFail(nkeStringLimit, "string exceeds configured limit", path) - if validateUtf8(value) >= 0: + if requireUtf8 and validateUtf8(value) >= 0: typedFail(nkeInvalidUtf8, "string is not valid UTF-8", path) result = "\"" for c in value: @@ -118,6 +150,7 @@ proc render(node: DataNode; destination: var string; limits: CodecLimits; path: case node.kind of dkAtom: destination.boundedAdd(node.text, limits) of dkString: destination.boundedAdd(quote(node.text, limits, path), limits) + of dkBytes: destination.boundedAdd(quote(node.text, limits, path, false), limits) of dkChar: destination.boundedAdd('\'', limits) destination.boundedAdd(quote(node.text, limits, path)[1 .. ^2], limits) @@ -271,6 +304,7 @@ proc parseBifNode(document: BifDocument; pos: var int; limit, depth: int; proc nodeAtom(value: string): DataNode = DataNode(kind: dkAtom, text: value) proc nodeString(value: string): DataNode = DataNode(kind: dkString, text: value) +proc nodeBytes(value: string): DataNode = DataNode(kind: dkBytes, text: value) proc compound(tag: string; values: varargs[DataNode]): DataNode = DataNode(kind: dkCompound, children: @[nodeAtom(tag)] & @values) @@ -324,6 +358,8 @@ proc emitTypedNode(builder: var TypedBifBuilder; node: DataNode; limits: CodecLi of dkString: if validateUtf8(node.text) >= 0: typedFail(nkeInvalidUtf8, "string is not valid UTF-8", "$") builder.emitTypedText(BifKindString, node.text, limits) + of dkBytes: + builder.emitTypedText(BifKindString, node.text, limits) of dkChar: if node.text.len != 1: typedFail(nkeTypeMismatch, "character must contain one byte", "$") builder.emitTyped(BifKindChar, uint64(ord(node.text[0])), limits) @@ -396,6 +432,10 @@ proc encodeSet[T](value: T; limits: CodecLimits; path: string): DataNode = proc encodeValue[T](value: T; limits: CodecLimits; path: string): DataNode = when T is cstring: typedFail(nkeUnsupportedType, "cstring is unsupported by the typed data profile", path) + elif T is NifBytes: + if value.data.len > limits.maxStringBytes: + typedFail(nkeStringLimit, "byte payload exceeds configured limit", path) + compound("bytes", nodeBytes(value.data)) elif T is distinct: type Base = distinctBase(T) compound("distinct", nodeString(name(T)), encodeValue(Base(value), limits, path)) @@ -469,7 +509,7 @@ proc toNif*[T](value: T; limits = defaultCodecLimits()): string = validLimits(limits) if activeReferences.len != 0: activeReferences.setLen(0) - let root = compound(DataRootTag, nodeAtom("1"), encodeValue(value, limits, "$")) + let root = compound(DataRootTag, nodeAtom($CurrentDataProfile), encodeValue(value, limits, "$")) root.render(result, limits, "$") proc toBif*[T](value: T; limits = defaultCodecLimits()): string = @@ -477,52 +517,52 @@ proc toBif*[T](value: T; limits = defaultCodecLimits()): string = if activeReferences.len != 0: activeReferences.setLen(0) var builder: TypedBifBuilder - builder.emitTypedNode(compound(DataRootTag, nodeAtom("1"), encodeValue(value, limits, "$")), limits, 0) + builder.emitTypedNode(compound(DataRootTag, nodeAtom($CurrentDataProfile), encodeValue(value, limits, "$")), limits, 0) result = encodeBifDocument(builder.document, limits) if result.len > limits.maxOutputBytes: typedFail(nkeOutputTooLarge, "typed BIF output exceeds configured limit", "$") proc decodeValue[T](node: DataNode; limits: CodecLimits; options: TypedCodecOptions; - path: string): T + profile: int; path: string): T proc decodeTableEntry[K, V](target: var Table[K, V]; entry: DataNode; limits: CodecLimits; options: TypedCodecOptions; - path: string) = + profile: int; path: string) = if entry.kind != dkCompound or entry.children.len != 3 or entry.children[0].kind != dkAtom or entry.children[0].text != "entry": typedFail(nkeTypeMismatch, "invalid table entry", path, entry.offset) - let key = decodeValue[K](entry.children[1], limits, options, path & ".") + let key = decodeValue[K](entry.children[1], limits, options, profile, path & ".") if target.hasKey(key): typedFail(nkeTypeMismatch, "duplicate table key", path, entry.offset) - target[key] = decodeValue[V](entry.children[2], limits, options, path & "[key]") + target[key] = decodeValue[V](entry.children[2], limits, options, profile, path & "[key]") proc decodeOrderedTableEntry[K, V](target: var OrderedTable[K, V]; entry: DataNode; limits: CodecLimits; options: TypedCodecOptions; - path: string) = + profile: int; path: string) = if entry.kind != dkCompound or entry.children.len != 3 or entry.children[0].kind != dkAtom or entry.children[0].text != "entry": typedFail(nkeTypeMismatch, "invalid table entry", path, entry.offset) - let key = decodeValue[K](entry.children[1], limits, options, path & ".") + let key = decodeValue[K](entry.children[1], limits, options, profile, path & ".") if target.hasKey(key): typedFail(nkeTypeMismatch, "duplicate table key", path, entry.offset) - target[key] = decodeValue[V](entry.children[2], limits, options, path & "[key]") + target[key] = decodeValue[V](entry.children[2], limits, options, profile, path & "[key]") proc decodeSetItem[E](target: var HashSet[E]; node: DataNode; - limits: CodecLimits; options: TypedCodecOptions; path: string) = - let item = decodeValue[E](node, limits, options, path) + limits: CodecLimits; options: TypedCodecOptions; profile: int; path: string) = + let item = decodeValue[E](node, limits, options, profile, path) if item in target: typedFail(nkeTypeMismatch, "duplicate set item", path, node.offset) target.incl item proc decodeOrderedSetItem[E](target: var OrderedSet[E]; node: DataNode; - limits: CodecLimits; options: TypedCodecOptions; path: string) = - let item = decodeValue[E](node, limits, options, path) + limits: CodecLimits; options: TypedCodecOptions; profile: int; path: string) = + let item = decodeValue[E](node, limits, options, profile, path) if item in target: typedFail(nkeTypeMismatch, "duplicate set item", path, node.offset) target.incl item proc decodeValue[T](node: DataNode; limits: CodecLimits; options: TypedCodecOptions; - path: string): T = + profile: int; path: string): T = when T is distinct: let values = require(node, "distinct", path) if values.len != 3 or values[1].kind != dkString: @@ -530,10 +570,10 @@ proc decodeValue[T](node: DataNode; limits: CodecLimits; options: TypedCodecOpti if options.requireTypeNames and values[1].text != name(T): typedFail(nkeTypeMismatch, "distinct type name mismatch", path, values[1].offset) type Base = distinctBase(T) - result = T(decodeValue[Base](values[2], limits, options, path)) + result = T(decodeValue[Base](values[2], limits, options, profile, path)) elif T is range: type Base = typeof(low(T) + 0) - let value = decodeValue[Base](node, limits, options, path) + let value = decodeValue[Base](node, limits, options, profile, path) if value < low(T) or value > high(T): typedFail(nkeTypeMismatch, "value is outside the target range", path, node.offset) result = T(value) @@ -567,6 +607,15 @@ proc decodeValue[T](node: DataNode; limits: CodecLimits; options: TypedCodecOpti if classify(result) in {fcNan, fcInf, fcNegInf}: typedFail(nkeNonFiniteFloat, "non-finite float is unsupported", path, node.offset) except ValueError: typedFail(nkeTypeMismatch, "invalid float", path, node.offset) + elif T is NifBytes: + if profile < 2: + typedFail(nkeUnsupportedDataProfile, "byte payloads require typed data profile 2", path, node.offset) + let values = require(node, "bytes", path) + if values.len != 2 or values[1].kind != dkString: + typedFail(nkeTypeMismatch, "expected bytes", path, node.offset) + if values[1].text.len > limits.maxStringBytes: + typedFail(nkeStringLimit, "byte payload exceeds configured limit", path, values[1].offset) + result.data = values[1].text elif T is string: if node.kind != dkString: typedFail(nkeTypeMismatch, "expected string", path, node.offset) if node.text.len > limits.maxStringBytes: typedFail(nkeStringLimit, "string exceeds configured limit", path, node.offset) @@ -588,48 +637,48 @@ proc decodeValue[T](node: DataNode; limits: CodecLimits; options: TypedCodecOpti if node.kind == dkAtom and node.text == "none": return none(typeof(default(T).get)) let values = require(node, "some", path) if values.len != 2: typedFail(nkeTypeMismatch, "invalid Option", path, node.offset) - result = some(decodeValue[typeof(default(T).get)](values[1], limits, options, path)) + result = some(decodeValue[typeof(default(T).get)](values[1], limits, options, profile, path)) elif T is Table: let values = require(node, "table", path) if values.len - 1 > limits.maxContainerItems: typedFail(nkeTokenLimit, "table exceeds configured limit", path, node.offset) for i in 1 ..< values.len: - decodeTableEntry(result, values[i], limits, options, path & "[" & $(i - 1) & "]") + decodeTableEntry(result, values[i], limits, options, profile, path & "[" & $(i - 1) & "]") elif T is OrderedTable: let values = require(node, "table", path) if values.len - 1 > limits.maxContainerItems: typedFail(nkeTokenLimit, "table exceeds configured limit", path, node.offset) for i in 1 ..< values.len: - decodeOrderedTableEntry(result, values[i], limits, options, path & "[" & $(i - 1) & "]") + decodeOrderedTableEntry(result, values[i], limits, options, profile, path & "[" & $(i - 1) & "]") elif T is HashSet: let values = require(node, "set", path) if values.len - 1 > limits.maxContainerItems: typedFail(nkeTokenLimit, "set exceeds configured limit", path, node.offset) for i in 1 ..< values.len: - decodeSetItem(result, values[i], limits, options, path & "[" & $(i - 1) & "]") + decodeSetItem(result, values[i], limits, options, profile, path & "[" & $(i - 1) & "]") elif T is OrderedSet: let values = require(node, "set", path) if values.len - 1 > limits.maxContainerItems: typedFail(nkeTokenLimit, "set exceeds configured limit", path, node.offset) for i in 1 ..< values.len: - decodeOrderedSetItem(result, values[i], limits, options, path & "[" & $(i - 1) & "]") + decodeOrderedSetItem(result, values[i], limits, options, profile, path & "[" & $(i - 1) & "]") elif T is seq: let values = require(node, "seq", path) if values.len - 1 > limits.maxContainerItems: typedFail(nkeTokenLimit, "sequence exceeds configured limit", path, node.offset) type Elem = typeof(default(T)[0]) result = newSeqOfCap[Elem](values.len - 1) - for i in 1 ..< values.len: result.add decodeValue[Elem](values[i], limits, options, path & "[" & $(i - 1) & "]") + for i in 1 ..< values.len: result.add decodeValue[Elem](values[i], limits, options, profile, path & "[" & $(i - 1) & "]") elif T is array: let values = require(node, "array", path) if values.len - 1 != result.len: typedFail(nkeArrayLengthMismatch, "array length mismatch", path, node.offset) for i in 0 ..< result.len: - result[i] = decodeValue[typeof(result[i])](values[i + 1], limits, options, path & "[" & $i & "]") + result[i] = decodeValue[typeof(result[i])](values[i + 1], limits, options, profile, path & "[" & $i & "]") elif T is tuple: let values = require(node, "tuple", path) var index = 1 for fieldName, field in fieldPairs(result): if index >= values.len: typedFail(nkeArrayLengthMismatch, "tuple length mismatch", path, node.offset) - field = decodeValue[typeof(field)](values[index], limits, options, path & "." & fieldName) + field = decodeValue[typeof(field)](values[index], limits, options, profile, path & "." & fieldName) inc index if index != values.len: typedFail(nkeArrayLengthMismatch, "tuple length mismatch", path, node.offset) elif T is ref: @@ -638,7 +687,7 @@ proc decodeValue[T](node: DataNode; limits: CodecLimits; options: TypedCodecOpti let values = require(node, "ref", path) if values.len != 2: typedFail(nkeTypeMismatch, "invalid ref object", path, node.offset) new(result) - result[] = decodeValue[typeof(result[])](values[1], limits, options, path) + result[] = decodeValue[typeof(result[])](values[1], limits, options, profile, path) elif T is object: let values = require(node, "object", path) if values.len < 2 or values[1].kind != dkString: typedFail(nkeTypeMismatch, "invalid object", path, node.offset) @@ -658,7 +707,7 @@ proc decodeValue[T](node: DataNode; limits: CodecLimits; options: TypedCodecOpti if match >= 0: typedFail(nkeUnknownField, "duplicate object field", path & "." & fieldName, entry.offset) match = i if match < 0: typedFail(nkeMissingField, "missing object field", path & "." & fieldName, node.offset) - decodeValue[fieldType](values[match].children[2], limits, options, path & "." & fieldName) + decodeValue[fieldType](values[match].children[2], limits, options, profile, path & "." & fieldName) result = initTypedCaseObject(T, decodeDiscriminant) var consumed = newSeq[bool](values.len) # The discriminant was assigned in initTypedCaseObject above; fieldPairs @@ -677,7 +726,7 @@ proc decodeValue[T](node: DataNode; limits: CodecLimits; options: TypedCodecOpti match = i if match < 0: typedFail(nkeMissingField, "missing object field", path & "." & fieldName, node.offset) consumed[match] = true - field = decodeValue[typeof(field)](values[match].children[2], limits, options, path & "." & fieldName) + field = decodeValue[typeof(field)](values[match].children[2], limits, options, profile, path & "." & fieldName) when isVariantObject(T): if fieldName == variantDiscriminantName(T): for i in 2 ..< values.len: @@ -708,9 +757,8 @@ proc fromNif*[T](source: string; _: typedesc[T]; limits = defaultCodecLimits(); skipSpace(source, pos) if pos != source.len: typedFail(nkeMalformedInput, "trailing typed NIF data", "$", pos) let values = require(root, DataRootTag, "$") - if values.len != 3 or values[1].kind != dkAtom or values[1].text != "1": - typedFail(nkeUnsupportedDataProfile, "unsupported NIFKit data profile", "$", root.offset) - result = decodeValue[T](values[2], limits, options, "$") + let profile = parseDataProfile(values, root.offset) + result = decodeValue[T](values[2], limits, options, profile, "$") proc fromBif*[T](source: string; _: typedesc[T]; limits = defaultCodecLimits(); options = defaultTypedCodecOptions()): T = @@ -727,6 +775,5 @@ proc fromBif*[T](source: string; _: typedesc[T]; limits = defaultCodecLimits(); if pos != document.tokens.len: typedFail(nkeMalformedInput, "trailing typed BIF data", "$", pos) let values = require(root, DataRootTag, "$") - if values.len != 3 or values[1].kind != dkAtom or values[1].text != "1": - typedFail(nkeUnsupportedDataProfile, "unsupported NIFKit data profile", "$", root.offset) - result = decodeValue[T](values[2], limits, options, "$") + let profile = parseDataProfile(values, root.offset) + result = decodeValue[T](values[2], limits, options, profile, "$") diff --git a/tests/test_typed_serializer.nim b/tests/test_typed_serializer.nim index f517a36..f17b584 100644 --- a/tests/test_typed_serializer.nim +++ b/tests/test_typed_serializer.nim @@ -26,8 +26,11 @@ type count: int RefChain = ref object child: RefChain + BinaryRecord = object + name: string + content: NifBytes -suite "typed serializer v1": +suite "typed serializer profiles": test "round-trips primitive boundaries and canonical BIF": check fromNif(toNif(false), bool) == false check fromBif(toBif(true), bool) == true @@ -85,7 +88,7 @@ suite "typed serializer v1": let value = Record(title: "NIF\n\0", count: -12, enabled: true, state: stOpen, note: some("hello"), items: @[1, 2, 3]) let nif = toNif(value) - check nif.startsWith("(nifkit\\2Ddata 1 ") + check nif.startsWith("(nifkit\\2Ddata 2 ") check fromNif(nif, Record) == value let bif = toBif(value) check fromBif(bif, Record) == value @@ -100,9 +103,10 @@ suite "typed serializer v1": let decoded = fromNif(source, Record, options = TypedCodecOptions(allowUnknownFields: true, requireTypeNames: true)) check decoded.title == "x" - test "rejects incompatible data profile versions": + test "accepts v1 data and rejects incompatible profile versions": + check fromNif("(nifkit\\2Ddata 1 true)", bool) try: - discard fromNif("(nifkit\\2Ddata 2 true)", bool) + discard fromNif("(nifkit\\2Ddata 3 true)", bool) fail() except NifKitError as error: check error.kind == nkeUnsupportedDataProfile @@ -127,7 +131,7 @@ suite "typed serializer v1": test "round-trips distinct values with their declared type name": let value = UserId(42) let nif = toNif(value) - check nif == "(nifkit\\2Ddata 1 (distinct \"UserId\" 42u))" + check nif == "(nifkit\\2Ddata 2 (distinct \"UserId\" 42u))" check uint64(fromNif(nif, UserId)) == uint64(value) check uint64(fromBif(toBif(value), UserId)) == uint64(value) expect NifKitError: @@ -170,6 +174,41 @@ suite "typed serializer v1": except NifKitError as error: check error.kind == nkeUnsupportedType + test "round-trips arbitrary byte payloads without UTF-8 or base64": + let bytes = initNifBytes("\x89PNG\r\n\x1a\n\0\xff") + let nif = toNif(bytes) + check nif.startsWith("(nifkit\\2Ddata 2 (bytes ") + check fromNif(nif, NifBytes) == bytes + let bif = toBif(bytes) + check fromBif(bif, NifBytes) == bytes + check bifToNif(bif) == nif + check bytes.toSeq == @[0x89'u8, 0x50'u8, 0x4e'u8, 0x47'u8, 0x0d'u8, + 0x0a'u8, 0x1a'u8, 0x0a'u8, 0x00'u8, 0xff'u8] + + test "rejects byte payloads in profile v1 and enforces byte limits": + expect NifKitError: + discard fromNif("(nifkit\\2Ddata 1 (bytes \"x\"))", NifBytes) + var limits = defaultCodecLimits() + limits.maxStringBytes = 2 + expect NifKitError: + discard toBif(initNifBytes("abc"), limits) + var poolLimits = defaultCodecLimits() + poolLimits.maxPoolBytes = 2 + expect NifKitError: + discard toBif(initNifBytes("abc"), poolLimits) + + test "round-trips binary fields inside typed objects": + let value = BinaryRecord( + name: "preview.png", + content: initNifBytes("\x89PNG\r\n\x1a\n\0\xff") + ) + let nif = toNif(value) + check fromNif(nif, BinaryRecord) == value + check fromBif(toBif(value), BinaryRecord) == value + let v1 = "(nifkit\\2Ddata 1 (object \"BinaryRecord\" (field \"name\" \"preview.png\") (field \"content\" (bytes \"x\"))))" + expect NifKitError: + discard fromNif(v1, BinaryRecord) + test "round trips every active branch of a variant object": let textValue = VariantRecord(id: "a", kind: vkText, text: "hello") let countValue = VariantRecord(id: "b", kind: vkCount, count: 12)