diff --git a/docker/README.md b/docker/README.md index 0adb203..f654ad2 100644 --- a/docker/README.md +++ b/docker/README.md @@ -233,14 +233,62 @@ as little-endian `<5dq>` records (`open`, `high`, `low`, `close`, `volume`, slicing, so slice bounds remain a separate runtime concern. Paths, CSV newline style, header order, and equivalent numeric spellings do not affect the hash. -Decode the token to recover the provenance: +### Canonical fingerprint JSON + +`token` is base64 of the **canonical provenance JSON bytes**; `digest` is +`sha256:` plus the hex SHA-256 of those same bytes. Encoding is a direct +RFC 8785 / JCS-style canonical writer over the accepted value tree (not plain +`JSON.stringify`, which does not sort keys or implement full JCS). Over values +parsed under a JavaScript / IEEE-754 binary64 number model, the token bytes +match a JCS direct encoder for the types we accept: + +| Rule | Behavior | +| --- | --- | +| Objects | Keys sorted by **UTF-16 code unit** order (JCS / ECMAScript), not Unicode code point; no whitespace. Each key is normalized once via base `str.__str__` to a plain built-in `str`; **duplicate normalized names fail closed** (`ValueError`) so distinct str-subclass keys that coexist only via custom `__eq__`/`__hash__` cannot emit the same JSON name twice (JS would silently drop one). Non-str keys raise `TypeError`. Own keys such as `"10"`, `"2"`, and `"__proto__"` are retained in lexical order (a naive object-rebuild + `JSON.stringify` reorders array-index keys and can drop `__proto__`). | +| Arrays | Element order preserved; no whitespace | +| Floats | ECMAScript `NumberToString` form for every **finite** IEEE-754 value (no trailing `.0` on integral floats, `±0` → `0`, scientific notation at ES thresholds: exponent < -6 or ≥ 21). Float subclasses are normalized via base `float.__float__` first so hostile `__float__`/`__abs__`/comparison/`__repr__` hooks cannot alter math or emission. | +| Integers | Decimal digits only for exact integers inside the product safe-integer domain `[-(2⁵³-1), 2⁵³-1]` (= `Number.MIN/MAX_SAFE_INTEGER`). This is a **strict accepted-type policy** guaranteeing unique lossless integer identity across generic ECMAScript consumers — not a claim that every larger individual value must round as binary64. Exact integers outside the domain (e.g. int `2⁵³`) are **rejected** (`ValueError`); isolated binary64 values such as float `2⁵³` remain legal on the float path. Int subclasses (e.g. `IntEnum`) are normalized via base `int.__index__` to a plain built-in int before domain checks and digit emission, so hostile/`Enum.NAME` `__index__`/`__int__`/comparison/`__str__`/`__repr__`/`__format__` hooks cannot change the value or bypass rejection. Booleans are not integers. | +| Non-finite numbers | Rejected (`ValueError`) — not encoded as `null` | +| Strings | JSON control escapes + `"`/`\`; **raw valid Unicode** (no `ensure_ascii` `\uXXXX` for non-ASCII); U+2028/U+2029 and non-BMP stay as UTF-8 code points. Unpaired UTF-16 surrogates: **rejected**. Str subclasses are normalized via base `str.__str__` first so hostile `__str__`/`__iter__`/`encode` hooks cannot change emission, key order, or bypass surrogate rejection. | +| Bools / null | `true` / `false` / `null` | + +Semantic provenance values and the `{token,digest,provenance}` object shape +are unchanged; only the hashed byte form is language-stable. + +**Verification (authoritative path):** base64-decode `token` to the canonical +UTF-8 JSON bytes and hash those bytes. `digest` is exactly `sha256:` plus the +hex SHA-256 of that same byte string — no re-serialization is required. +Decoded canonical token bytes are authoritative; the inlined +`fingerprint.provenance` is a convenience view of the same data. + +**Re-canonicalization is not free-form `json.loads` / plain +`JSON.stringify`:** if a verifier rebuilds canonical JSON from a decoded +value tree, it must use an RFC 8785 / JCS **direct** encoder whose JSON +number model is IEEE-754 binary64 (ECMAScript `Number`). Default Python +`json.loads` is **not** sufficient for that path: canonical tokens such as +float `1e20` serialize as `100000000000000000000`, and plain `json.loads` +yields a Python `int` outside the product integer domain, which this helper +correctly **rejects** rather than rounding. Project tests that exercise +verifier reconstruction use `json.loads(text, parse_int=float)` so +integral-looking tokens re-enter as binary64 floats before re-encoding. + +**Out-of-domain provenance yields no fingerprint** under existing callers: +`build_fingerprint` raises, `docker/run_json.py` sets `fingerprint` to +`null`, and `scripts/run_strategy.py` skips writing a fingerprint file. + +**Residual intentional fail-closed cases:** non-finite numbers, exact +integers outside the product safe-integer domain, unpaired surrogates +(invalid I-JSON / UTF-8), and object keys that collide after `str.__str__` +normalization (duplicate normalized JSON names). + +Decode the token to inspect the canonical provenance JSON: ```bash jq -r '.fingerprint.token' report.json | base64 -d | jq . ``` -The provenance is also inlined under `fingerprint.provenance`, so decoding is -only needed to verify the token round-trips. +To verify `digest` without rebuilding JSON, hash the decoded token bytes +directly (for example `base64 -d | shasum -a 256`). ## Exit codes diff --git a/docker/run_json.py b/docker/run_json.py index 6aba802..ad92d2b 100755 --- a/docker/run_json.py +++ b/docker/run_json.py @@ -58,8 +58,8 @@ ... ], "fingerprint": { # decode-able backtest provenance - "token": "", # b64decode -> JSON - "digest": "sha256:", # stable run id over canonical JSON + "token": "", # b64decode -> authoritative UTF-8 bytes + "digest": "sha256:", # hash token bytes; do not assume json.loads round-trip "provenance": { "engine": { version_string, major, minor, patch, commit_sha }, "feed": { canonicalization, source_values_sha256 }, @@ -319,8 +319,225 @@ def build_provenance(engine: dict, cpp_path, transpiled: bool, } +# Product accepted-type domain for exact Python integers: JavaScript +# Number.MAX/MIN_SAFE_INTEGER (= ±(2**53 - 1)). Rejecting larger exact ints +# guarantees unique lossless integer identity across generic ECMAScript +# consumers. This is not a claim that every such magnitude must round as +# binary64 — e.g. float(2**53) is exactly representable and remains legal +# on the float path, while exact int(2**53) is deliberately outside the +# product integer domain. Out-of-domain provenance yields no fingerprint +# under existing callers (exception → fingerprint None / skipped). +# Booleans are handled separately (bool subclasses int). +_JS_MAX_SAFE_INTEGER = 9007199254740991 +_JS_MIN_SAFE_INTEGER = -9007199254740991 + + +def _canonical_json_number(num: float) -> str: + """Serialize a finite IEEE-754 float via ECMAScript NumberToString. + + Matches ECMAScript NumberToString (RFC 8785 / JCS numeric form): + integral values have no trailing ``.0``, ``±0`` is ``0``, and + scientific notation uses ES exponent thresholds (``e`` when the + exponent is < -6 or >= 21). Non-finite values raise ValueError. + Float subclasses are normalized via base ``float.__float__`` first so + hooks such as ``__float__``/``__abs__``/comparisons/``__repr__`` cannot + alter the underlying binary64 value used for math and emission. + """ + # Plain built-in float: ignore subclass __float__/__abs__/__lt__/... + num = float.__float__(num) + if not math.isfinite(num): + raise ValueError( + "non-finite numbers are not permitted in fingerprint JSON") + if num == 0.0: + return "0" + negative = num < 0 + r = repr(abs(num)) + if "e" in r or "E" in r: + mant, exp_s = r.lower().split("e") + exp = int(exp_s) + if "." in mant: + whole, frac = mant.split(".") + digits_raw = whole + frac + n = exp + len(whole) + else: + digits_raw = mant + n = exp + len(digits_raw) + else: + if "." in r: + whole, frac = r.split(".") + digits_raw = whole + frac + n = len(whole) + else: + digits_raw = r + n = len(digits_raw) + # Leading-zero strip adjusts n so value = int(digits)*10^(n-k) holds. + lead = len(digits_raw) - len(digits_raw.lstrip("0")) + digits = digits_raw.lstrip("0") or "0" + if digits != "0": + n -= lead + while len(digits) > 1 and digits[-1] == "0": + digits = digits[:-1] + k = len(digits) + sign = "-" if negative else "" + if 0 < n <= 21: + if k <= n: + return sign + digits + ("0" * (n - k)) + return sign + digits[:n] + "." + digits[n:] + if -6 < n <= 0: + return sign + "0." + ("0" * (-n)) + digits + exp = n - 1 + exp_s = f"+{exp}" if exp >= 0 else str(exp) + if k == 1: + return sign + digits + "e" + exp_s + return sign + digits[0] + "." + digits[1:] + "e" + exp_s + + +def _reject_unpaired_surrogates(s: str, *, what: str) -> None: + """Fail closed on unpaired UTF-16 surrogates (invalid I-JSON / UTF-8).""" + for ch in s: + cp = ord(ch) + if 0xD800 <= cp <= 0xDFFF: + raise ValueError( + f"unpaired UTF-16 surrogate in fingerprint JSON {what}") + + +def _canonical_json_string(s: str) -> str: + """Serialize a string in RFC 8785 / JCS string form. + + JSON control characters, quotes, and backslashes are escaped; valid + Unicode (including non-ASCII, emoji, and U+2028/U+2029) is emitted as + raw code points (not ``ensure_ascii`` ``\\uXXXX`` escapes). Unpaired + surrogates raise ValueError rather than producing invalid I-JSON. + Str subclasses are normalized via base ``str.__str__`` first so + ``__str__``/``__iter__`` hooks cannot redirect iteration or emission. + """ + # Plain built-in str: ignore subclass __str__/__iter__/... + s = str.__str__(s) + _reject_unpaired_surrogates(s, what="string") + return json.dumps(s, ensure_ascii=False) + + +def _utf16_code_unit_key(s: str) -> bytes: + """Object-key sort key matching JCS / ECMAScript UTF-16 code unit order. + + Str subclasses are normalized via base ``str.__str__`` first so + ``__str__``/``__iter__``/``encode`` hooks cannot corrupt key order or + hide unpaired surrogates. + """ + # Plain built-in str: ignore subclass __str__/__iter__/encode hooks. + s = str.__str__(s) + _reject_unpaired_surrogates(s, what="object key") + return s.encode("utf-16-be") + + +def _canonical_fingerprint_json(value) -> str: + """Canonical JSON text for fingerprint token/digest bytes. + + Direct RFC 8785 / JCS-style canonical writer over the accepted Python + value tree. The resulting UTF-8 token bytes are authoritative for + ``digest``; verifiers should hash those bytes rather than assuming plain + ``JSON.stringify`` is itself JCS (it does not sort keys or implement + full JCS). Over values parsed under a JavaScript / IEEE-754 binary64 + number model, this matches a JCS direct encoder for the accepted types: + - objects: each key is normalized once via base ``str.__str__`` to a + plain built-in str; keys sorted by UTF-16 code unit order on those + normalized names; no whitespace. Duplicate normalized names raise + ValueError (fail closed — distinct str-subclass keys can override + ``__eq__``/``__hash__`` so both coexist in a Python dict while + ``str.__str__`` yields the same text; emitting both would produce + duplicate JSON names that JS silently drops). Non-str keys raise + TypeError. The retained original key is used only for value lookup. + - arrays: element order preserved; no whitespace + - numbers (float): ECMAScript NumberToString for every finite IEEE-754 + value after base ``float.__float__`` normalization (subclass hooks + ignored); non-finite numbers raise ValueError + - numbers (int): decimal digits only for exact integers inside the + product safe-integer domain [-(2**53-1), 2**53-1]. This is a strict + accepted-type policy guaranteeing unique lossless integer identity + across generic ECMAScript consumers — not a claim that every larger + individual value is unrepresentable as binary64. Exact integers + outside the domain raise ValueError (e.g. int 2**53); isolated + binary64 floats such as float(2**53) remain legal via the float path. + Int subclasses (e.g. IntEnum) are normalized via base ``int.__index__`` + to a plain built-in int before domain checks and digit emission, so + ``__index__``/``__int__``/comparison/``__str__``/``__repr__``/ + ``__format__`` hooks cannot change the value or bypass rejection. + Booleans are not integers. + - strings: JCS form — control/quote/backslash escapes, raw valid + Unicode; unpaired surrogates raise ValueError. Str subclasses are + normalized via base ``str.__str__`` so ``__str__``/``__iter__``/ + ``encode`` hooks cannot change emission, key order, or bypass + surrogate rejection. + - bools/null: ``true`` / ``false`` / ``null`` + """ + if value is None: + return "null" + if value is True: + return "true" + if value is False: + return "false" + if isinstance(value, str): + return _canonical_json_string(value) + if isinstance(value, int) and not isinstance(value, bool): + # Plain built-in int via base slot: subclass __index__/__int__/ + # comparison hooks must not bypass domain checks or alter digits + # (Python 3.9 IntEnum str was "Enum.NAME"). + value = int.__index__(value) + if value < _JS_MIN_SAFE_INTEGER or value > _JS_MAX_SAFE_INTEGER: + raise ValueError( + "integers outside the JavaScript safe-integer range " + f"[{_JS_MIN_SAFE_INTEGER}, {_JS_MAX_SAFE_INTEGER}] " + "are not permitted in fingerprint JSON") + return int.__repr__(value) + if isinstance(value, float): + return _canonical_json_number(value) + if isinstance(value, list): + return "[" + ",".join( + _canonical_fingerprint_json(v) for v in value) + "]" + if isinstance(value, dict): + # Normalize each original key exactly once to a plain built-in str. + # Distinct str subclasses can override __eq__/__hash__ so two keys + # coexist while str.__str__ yields the same text; emit the + # normalized name, look up values via the retained original key, + # and fail closed on duplicate normalized names. + items = [] # (normalized_key, original_key) + seen_normalized = set() + for k in value.keys(): + if not isinstance(k, str): + raise TypeError( + "fingerprint JSON object keys must be str, " + f"got {type(k).__name__}") + nk = str.__str__(k) + if nk in seen_normalized: + raise ValueError( + "duplicate fingerprint JSON object key after " + f"str normalization: {nk!r}") + seen_normalized.add(nk) + items.append((nk, k)) + parts = [] + for nk, orig in sorted(items, key=lambda ik: _utf16_code_unit_key(ik[0])): + parts.append( + _canonical_json_string(nk) + ":" + + _canonical_fingerprint_json(value[orig])) + return "{" + ",".join(parts) + "}" + raise TypeError( + f"fingerprint JSON cannot encode {type(value).__name__}") + + def build_fingerprint(provenance: dict) -> dict: - canonical = json.dumps(provenance, sort_keys=True, separators=(",", ":")) + """Build ``{token, digest, provenance}`` for a provenance dict. + + ``token`` is base64 of the canonical UTF-8 JSON bytes; ``digest`` is + ``sha256:`` + hex of those same bytes. Verifiers should treat the decoded + token bytes as authoritative and hash them directly. Re-canonicalizing a + decoded value tree requires an RFC 8785/JCS direct encoder with an + IEEE-754 binary64 number model — default Python ``json.loads`` may yield + ints outside the product integer domain for tokens such as ``1e20`` + (``100000000000000000000``). Project tests that rebuild from token text + use ``json.loads(text, parse_int=float)``. Out-of-domain inputs raise + here; existing callers yield no fingerprint (``None`` / skip). + """ + canonical = _canonical_fingerprint_json(provenance) raw = canonical.encode("utf-8") return { "token": base64.b64encode(raw).decode("ascii"), diff --git a/scripts/fingerprint_self_test.py b/scripts/fingerprint_self_test.py index 850dd6e..2f43f1c 100644 --- a/scripts/fingerprint_self_test.py +++ b/scripts/fingerprint_self_test.py @@ -15,10 +15,14 @@ import ast import base64 +import enum import hashlib import importlib.util import json +import math +import shutil import struct +import subprocess import sys import tempfile from pathlib import Path @@ -27,6 +31,234 @@ RUN_JSON = REPO / "docker" / "run_json.py" RUN_STRATEGY = REPO / "scripts" / "run_strategy.py" +# Deterministic numeric vectors: (python_value, expected_canonical_text). +# Expected forms match ECMAScript NumberToString (RFC 8785 numeric form). +CANONICAL_NUMBER_VECTORS = [ + (12345.0, "12345"), + (0.0, "0"), + (-0.0, "0"), + (1.5, "1.5"), + (-1.5, "-1.5"), + (0.1, "0.1"), + (1e-7, "1e-7"), + (1e-6, "0.000001"), + (1e20, "100000000000000000000"), + (1e21, "1e+21"), + (-12345.0, "-12345"), + (-1e-7, "-1e-7"), + (-1e20, "-100000000000000000000"), + (1, "1"), # int path + (0, "0"), + # Product safe-integer edges (Number.MAX/MIN_SAFE_INTEGER = ±(2**53-1)). + (9007199254740991, "9007199254740991"), + (-9007199254740991, "-9007199254740991"), +] + +# String forms match RFC 8785 / JCS (raw Unicode, not ensure_ascii escapes). +# Control chars / quotes / backslash stay escaped. +CANONICAL_STRING_VECTORS = [ + ("", '""'), + ("ascii", '"ascii"'), + ("中文键", '"中文键"'), + ("中文值", '"中文值"'), + ("😀", '"😀"'), + ("a😀b", '"a😀b"'), + ("\u2028", '"\u2028"'), # LINE SEPARATOR — raw, not \u2028 escape + ("\u2029", '"\u2029"'), # PARAGRAPH SEPARATOR + ("\u2028\u2029", '"\u2028\u2029"'), + ('say "hi"\\ok', '"say \\"hi\\"\\\\ok"'), + ("\n\t\r\b\f", '"\\n\\t\\r\\b\\f"'), + ("\x00\x1f", '"\\u0000\\u001f"'), +] + +# Exact integers outside the product safe-integer domain — rejected by policy +# (unique lossless integer identity for generic ECMAScript consumers). This is +# intentional domain rejection, not a claim that every magnitude must round +# under binary64 (see float(2**53) vectors below, which remain legal). +UNSAFE_INTEGERS = [ + 9007199254740992, # exact int 2**53 — outside product int domain + 9007199254740993, # not uniquely representable as binary64 integer + 9223372036854775807, + -9007199254740992, + -9007199254740993, + -9223372036854775808, +] + + +class _WeirdStrInt(int): + """int subclass whose dunder hooks deliberately lie about the value.""" + + def __index__(self) -> int: + return 99 + + def __int__(self) -> int: + return 99 + + def __lt__(self, other) -> bool: + return False + + def __gt__(self, other) -> bool: + return False + + def __le__(self, other) -> bool: + return False + + def __ge__(self, other) -> bool: + return False + + def __eq__(self, other) -> bool: + return False + + def __str__(self) -> str: + return "not-a-json-number" + + def __repr__(self) -> str: + return "not-a-json-number" + + def __format__(self, format_spec: str) -> str: + return "99" + + +class _WeirdFloat(float): + """float subclass whose dunder hooks deliberately lie about the value.""" + + def __float__(self) -> float: + return -88.0 + + def __abs__(self): + return 88.0 + + def __lt__(self, other) -> bool: + return True + + def __eq__(self, other) -> bool: + return False + + def __repr__(self) -> str: + return "-88" + + def __format__(self, format_spec: str) -> str: + return "-88" + + +class _WeirdStr(str): + """str subclass whose dunder hooks deliberately lie about the value. + + Without base ``str.__str__`` normalization, ``__iter__`` can hide + unpaired surrogates from the rejection scan and ``encode`` can corrupt + JCS UTF-16 object-key order (e.g. flip ``"10"`` / ``"2"``). + """ + + def __str__(self) -> str: + return "hostile" + + def __iter__(self): + # Hide underlying content (including unpaired surrogates). + return iter("") + + def encode(self, encoding: str = "utf-8", errors: str = "strict") -> bytes: + # Flip UTF-16 code-unit order of keys "10" and "2" if consulted. + underlying = str.__str__(self) + if encoding == "utf-16-be": + if underlying == "10": + return "9".encode("utf-16-be") + if underlying == "2": + return "1".encode("utf-16-be") + return ("z" * max(1, len(underlying))).encode("utf-16-be") + return "hostile".encode(encoding, errors) + + +class _TaggedStr(str): + """Adversarial str subclass: tag/identity hash/eq allow coexistence. + + Two instances can share the same underlying text (so base + ``str.__str__`` normalizes both to one plain key) while tag-based + ``__eq__``/``__hash__`` let them both live in one Python dict. A + naive writer would emit duplicate JSON names; the fingerprint + helpers must fail closed with ValueError. + """ + + def __new__(cls, text: str, tag: object): + obj = str.__new__(cls, text) + obj.tag = tag + return obj + + def __eq__(self, other): + if isinstance(other, _TaggedStr): + return (self.tag == other.tag + and str.__str__(self) == str.__str__(other)) + return NotImplemented + + def __hash__(self): + return hash((_TaggedStr, str.__str__(self), self.tag)) + + def __str__(self) -> str: + return "hostile" + + +class _SampleIntEnum(enum.IntEnum): + """IntEnum-style int subclass regression (Py3.9 str was Enum.NAME).""" + + ALPHA = 7 + EDGE = 9007199254740991 + +# Unpaired UTF-16 surrogates — invalid I-JSON / UTF-8; fail closed. +UNPAIRED_SURROGATES = [ + "\ud800", + "\udfff", + "a\ud800b", + "\ud800\ud800", +] + +# Compact representative set from RFC 8785 Appendix B (ES6 Number samples): +# (big-endian IEEE-754 binary64 hex, expected JCS / NumberToString text). +RFC8785_APPENDIX_B_BINARY64 = [ + ("0000000000000000", "0"), # +0 + ("8000000000000000", "0"), # -0 + ("0000000000000001", "5e-324"), # min subnormal + ("000FFFFFFFFFFFFF", "2.225073858507201e-308"), # max subnormal + ("0010000000000000", "2.2250738585072014e-308"), # min normal + ("7FEFFFFFFFFFFFFF", "1.7976931348623157e+308"), # max finite + ("4340000000000000", "9007199254740992"), # 2**53 + ("4430000000000000", "295147905179352830000"), + ("44B52D02C7E14AF5", "9.999999999999997e+22"), + ("44B52D02C7E14AF6", "1e+23"), + ("44B52D02C7E14AF7", "1.0000000000000001e+23"), + ("444B1AE4D6E2EF4E", "999999999999999700000"), + ("444B1AE4D6E2EF4F", "999999999999999900000"), + ("444B1AE4D6E2EF50", "1e+21"), +] + +# Float sources whose canonical tokens look like large integers — plain +# json.loads yields Python int outside the product int domain; IEEE-754 parse +# is required for re-canonicalization tests. Exact-int forms of these +# magnitudes remain rejected (product integer domain), but the binary64 +# values themselves are deliberately legal. +IEEE754_TOKEN_ROUNDTRIP_VECTORS = [ + (1e20, "100000000000000000000"), + (9007199254740992.0, "9007199254740992"), # float 2**53 legal; int 2**53 not +] + + +def _helper_block(path: Path) -> str: + text = path.read_text(encoding="utf-8") + start = text.index("# >>> fingerprint helpers") + end = text.index("# <<< fingerprint helpers") + len("# <<< fingerprint helpers") + return text[start:end] + + +def _ieee754_json_loads(text: str): + """Decode JSON with a binary64 number model (RFC 8785 / ES Number). + + Default json.loads promotes integral tokens to unbounded Python int, which + is the wrong model for JCS re-canonicalization tests (e.g. 1e20). + """ + return json.loads(text, parse_int=float) + + +def _f64_from_be_hex(hex64: str) -> float: + return struct.unpack(">d", bytes.fromhex(hex64))[0] + # Fixture: a ctor that sets a SUBSET of strategy() fields (process_orders_on_close # and close_entries_rule are intentionally absent — they default in the engine # base class), a set_strategy_override with a decoy `initial_capital_ =` line @@ -143,12 +375,318 @@ def check(name: str, cond: bool) -> None: decoded = json.loads(base64.b64decode(fp["token"])) check(f"{label}: token decodes to provenance", decoded == prov) check(f"{label}: digest prefixed sha256:", fp["digest"].startswith("sha256:")) - canonical = json.dumps(prov, sort_keys=True, separators=(",", ":")) + canonical = m._canonical_fingerprint_json(prov) check(f"{label}: digest matches canonical sha256", fp["digest"] == "sha256:" + hashlib.sha256(canonical.encode()).hexdigest()) fp2 = m.build_fingerprint(prov) check(f"{label}: deterministic token", fp["token"] == fp2["token"]) + # --- canonical number / nest vectors (language-independent) ------ + for value, expected in CANONICAL_NUMBER_VECTORS: + got = m._canonical_fingerprint_json(value) + check(f"{label}: number vector {value!r} -> {expected!r}", + got == expected) + + # RFC 8785 Appendix B binary64 samples via exact bit patterns. + for hex64, expected in RFC8785_APPENDIX_B_BINARY64: + value = _f64_from_be_hex(hex64) + got = m._canonical_fingerprint_json(value) + check(f"{label}: RFC8785 App.B {hex64} -> {expected!r}", + got == expected) + + # Canonical-token round trips that require IEEE-754 number parsing: + # token bytes are authoritative for digest; re-canonicalization uses + # parse_int=float. Source Python ints outside the safe range still fail. + for value, expected in IEEE754_TOKEN_ROUNDTRIP_VECTORS: + canon = m._canonical_fingerprint_json(value) + check(f"{label}: ieee token form {value!r} -> {expected!r}", + canon == expected) + fp_n = m.build_fingerprint({"n": value}) + token_bytes = base64.b64decode(fp_n["token"]) + token_text = token_bytes.decode("utf-8") + check(f"{label}: ieee digest from token bytes {value!r}", + fp_n["digest"] + == "sha256:" + hashlib.sha256(token_bytes).hexdigest()) + # Plain json.loads yields a Python int outside the safe range. + plain = json.loads(token_text)["n"] + check(f"{label}: plain loads of {expected!r} is int (not float)", + isinstance(plain, int) and not isinstance(plain, bool)) + plain_rejected = False + try: + m._canonical_fingerprint_json(plain) + except ValueError: + plain_rejected = True + check(f"{label}: plain-loaded int of {expected!r} rejected", + plain_rejected) + # Verifier reconstruction: binary64 number model + re-encode. + rebuilt = m._canonical_fingerprint_json( + _ieee754_json_loads(token_text)) + check(f"{label}: ieee re-canon round-trip {value!r}", + rebuilt == token_text) + rebuilt_fp = m.build_fingerprint(_ieee754_json_loads(token_text)) + check(f"{label}: ieee re-canon digest matches {value!r}", + rebuilt_fp["digest"] == fp_n["digest"] + and rebuilt_fp["token"] == fp_n["token"]) + + for value, expected in CANONICAL_STRING_VECTORS: + got = m._canonical_fingerprint_json(value) + check(f"{label}: string vector {value!r} -> {expected!r}", + got == expected) + # Raw Unicode must not be ensure_ascii-escaped. + if any(ord(ch) >= 0x80 for ch in value): + check(f"{label}: string {value!r} is raw Unicode (no \\u)", + "\\u" not in got) + + # Chinese object + value: JCS string form (raw Unicode). + chinese = {"中文键": "中文值"} + chinese_canon = m._canonical_fingerprint_json(chinese) + check(f"{label}: Chinese object is raw Unicode JSON", + chinese_canon == '{"中文键":"中文值"}') + + # U+10000 sorts *before* U+E000 under UTF-16 code units (JCS/JS), + # but *after* under Unicode code-point order (Python sorted()). + key_order = {"\U00010000": 1, "\uE000": 2} + key_order_canon = m._canonical_fingerprint_json(key_order) + check(f"{label}: UTF-16 key order U+10000 before U+E000", + key_order_canon == '{"\U00010000":1,"\uE000":2}') + # Sanity: code-point sort would reverse these two keys. + check(f"{label}: UTF-16 order differs from code-point order", + sorted(["\U00010000", "\uE000"]) + == ["\uE000", "\U00010000"]) + + # Array-index-looking keys + "__proto__" must all be retained in + # lexical UTF-16 order. A naive JS object-rebuild + JSON.stringify + # reorders integer-index keys and drops assignment to __proto__. + special_keys = { + "2": 2, + "10": 10, + "__proto__": "own", + } + special_keys_canon = m._canonical_fingerprint_json(special_keys) + check(f"{label}: special keys retained in lexical UTF-16 order", + special_keys_canon == '{"10":10,"2":2,"__proto__":"own"}') + check(f"{label}: special keys all present (no __proto__ drop)", + '"10":10' in special_keys_canon + and '"2":2' in special_keys_canon + and '"__proto__":"own"' in special_keys_canon + and special_keys_canon.index('"10"') + < special_keys_canon.index('"2"') + < special_keys_canon.index('"__proto__"')) + special_nested = { + "z": [{"2": 1, "10": 2, "__proto__": 3}], + "2": 0, + "10": -1, + "__proto__": {"nested": True}, + } + special_nested_canon = m._canonical_fingerprint_json(special_nested) + check(f"{label}: nested special keys lexical UTF-16 + array recurse", + special_nested_canon == ( + '{"10":-1,"2":0,"__proto__":{"nested":true},' + '"z":[{"10":2,"2":1,"__proto__":3}]}')) + + nested = { + "z": [0.0, -0.0, 12345.0, {"inner": 1e-6}], + "a": {"initial_capital": 12345.0, "qty": 1.5}, + "emoji": "😀", + "sep": "\u2028\u2029", + "中文键": "中文值", + "\U00010000": "supplementary", + "\uE000": "pua", + } + nested_canon = m._canonical_fingerprint_json(nested) + # UTF-16 key order: a, emoji, sep, z, 中文键, U+10000, U+E000 + check(f"{label}: nested keys UTF-16-sorted + numbers/strings canonical", + nested_canon == ( + '{"a":{"initial_capital":12345,"qty":1.5},' + '"emoji":"😀",' + '"sep":"\u2028\u2029",' + '"z":[0,0,12345,{"inner":0.000001}],' + '"中文键":"中文值",' + '"\U00010000":"supplementary",' + '"\uE000":"pua"}')) + + nonfinite_rejected = True + for bad in (float("nan"), float("inf"), float("-inf")): + try: + m._canonical_fingerprint_json({"x": bad}) + nonfinite_rejected = False + except ValueError: + pass + check(f"{label}: non-finite numbers rejected", nonfinite_rejected) + + unsafe_int_rejected = True + for bad in UNSAFE_INTEGERS: + try: + m._canonical_fingerprint_json(bad) + unsafe_int_rejected = False + except ValueError: + pass + try: + m._canonical_fingerprint_json({"n": bad}) + unsafe_int_rejected = False + except ValueError: + pass + check(f"{label}: out-of-domain exact integers rejected", + unsafe_int_rejected) + + # Product policy: exact int 2**53 is outside the integer domain, while + # isolated binary64 float(2**53) remains legal NumberToString output. + # This is intentional accepted-type policy, not "every such value + # must round." + check(f"{label}: float 2**53 is legal binary64 token", + m._canonical_fingerprint_json(9007199254740992.0) + == "9007199254740992") + int_2_53_rejected = False + try: + m._canonical_fingerprint_json(9007199254740992) + except ValueError: + int_2_53_rejected = True + check(f"{label}: exact int 2**53 rejected (product int domain)", + int_2_53_rejected) + # Out-of-domain provenance yields no fingerprint under callers: + # build_fingerprint raises before emitting token/digest. + no_fp = False + try: + m.build_fingerprint({"n": 9007199254740992}) + except ValueError: + no_fp = True + check(f"{label}: out-of-domain provenance yields no fingerprint", + no_fp) + + # Int subclasses must emit underlying decimal digits even when + # __index__/__int__/comparisons/__str__/__repr__/__format__ are + # hostile (custom int) or Enum-style (IntEnum on Python 3.9; str + # was Enum.NAME). Domain checks use the normalized plain int. + check(f"{label}: custom int subclass ignores hostile int hooks", + m._canonical_fingerprint_json(_WeirdStrInt(42)) == "42" + and m._canonical_fingerprint_json(_WeirdStrInt(0)) == "0" + and m._canonical_fingerprint_json(_WeirdStrInt(-7)) == "-7" + and m._canonical_fingerprint_json(_WeirdStrInt(3)) == "3") + check(f"{label}: IntEnum emits decimal digits not Enum.NAME", + m._canonical_fingerprint_json(_SampleIntEnum.ALPHA) == "7" + and m._canonical_fingerprint_json(_SampleIntEnum.EDGE) + == "9007199254740991") + weird_oob_rejected = False + try: + m._canonical_fingerprint_json(_WeirdStrInt(9007199254740992)) + except ValueError: + weird_oob_rejected = True + check(f"{label}: custom int subclass still enforces safe-int domain", + weird_oob_rejected) + + # Float subclasses: normalize via float.__float__ so hostile + # __float__/__abs__/comparisons/repr cannot rewrite NumberToString. + check(f"{label}: custom float subclass ignores hostile float hooks", + m._canonical_fingerprint_json(_WeirdFloat(1.5)) == "1.5" + and m._canonical_fingerprint_json(_WeirdFloat(-0.0)) == "0" + and m._canonical_fingerprint_json(_WeirdFloat(0.0)) == "0") + weird_float_nonfinite_rejected = True + for bad in (float("nan"), float("inf"), float("-inf")): + try: + m._canonical_fingerprint_json(_WeirdFloat(bad)) + weird_float_nonfinite_rejected = False + except ValueError: + pass + check(f"{label}: custom float subclass still rejects nonfinite", + weird_float_nonfinite_rejected) + + # Str subclasses: normalize via str.__str__ so hostile + # __str__/__iter__/encode cannot rewrite emission, key order, or + # hide unpaired surrogates from the rejection scan. + check(f"{label}: custom str subclass ignores hostile str hooks", + m._canonical_fingerprint_json(_WeirdStr("good")) == '"good"' + and m._canonical_fingerprint_json({ + _WeirdStr("10"): 1, + _WeirdStr("2"): 2, + }) == '{"10":1,"2":2}') + weird_str_surrogate_rejected = False + try: + m._canonical_fingerprint_json(_WeirdStr("\ud800")) + except ValueError: + weird_str_surrogate_rejected = True + try: + m._canonical_fingerprint_json({_WeirdStr("\ud800"): "x"}) + weird_str_surrogate_rejected = False + except ValueError: + pass + check(f"{label}: custom str subclass still rejects unpaired surrogates", + weird_str_surrogate_rejected) + + # Distinct str-subclass keys can override __eq__/__hash__ so both + # coexist in a Python dict while str.__str__ yields the same plain + # text. Emitting both would produce duplicate JSON names (JS drops + # one); helpers must fail closed on the collision. + tagged_a = _TaggedStr("same", "a") + tagged_b = _TaggedStr("same", "b") + tagged_dup = {tagged_a: 1, tagged_b: 2} + check(f"{label}: tagged str keys coexist pre-normalization", + len(tagged_dup) == 2 + and str.__str__(tagged_a) == str.__str__(tagged_b) == "same") + tagged_dup_rejected = False + try: + m._canonical_fingerprint_json(tagged_dup) + except ValueError: + tagged_dup_rejected = True + check(f"{label}: duplicate normalized object keys rejected", + tagged_dup_rejected) + + # Booleans must remain distinct from the int path (bool subclasses int). + check(f"{label}: bool true/false stay boolean tokens", + m._canonical_fingerprint_json(True) == "true" + and m._canonical_fingerprint_json(False) == "false" + and m._canonical_fingerprint_json([True, False, 0, 1]) + == "[true,false,0,1]") + + surrogate_rejected = True + for bad in UNPAIRED_SURROGATES: + try: + m._canonical_fingerprint_json(bad) + surrogate_rejected = False + except ValueError: + pass + try: + m._canonical_fingerprint_json({bad: "x"}) + surrogate_rejected = False + except ValueError: + pass + try: + m._canonical_fingerprint_json({"k": bad}) + surrogate_rejected = False + except ValueError: + pass + check(f"{label}: unpaired surrogates rejected", surrogate_rejected) + + # Live bug: integral float must not produce a trailing ".0" that a + # JS NumberToString path would drop, breaking token/digest verification. + live = {"strategy": {"initial_capital": 12345.0}} + live_fp = m.build_fingerprint(live) + live_token_json = base64.b64decode(live_fp["token"]).decode("utf-8") + check(f"{label}: live integral-float token has no trailing .0", + '"initial_capital":12345' in live_token_json + and "12345.0" not in live_token_json) + # Safe integral tokens re-parse under either model; prefer binary64 + # for verifier reconstruction consistency with large-number vectors. + recomputed = m.build_fingerprint(_ieee754_json_loads(live_token_json)) + check(f"{label}: digest stable after token JSON re-parse", + recomputed["digest"] == live_fp["digest"] + and recomputed["token"] == live_fp["token"]) + # Authoritative path: hash decoded token bytes (no re-serialization). + live_token_bytes = live_token_json.encode("utf-8") + check(f"{label}: digest equals sha256 of token bytes", + live_fp["digest"] + == "sha256:" + hashlib.sha256(live_token_bytes).hexdigest()) + + # Unicode provenance survives token base64 + UTF-8 round-trip. + uni_live = {"中文键": "中文值", "emoji": "😀\u2028"} + uni_fp = m.build_fingerprint(uni_live) + uni_token_json = base64.b64decode(uni_fp["token"]).decode("utf-8") + check(f"{label}: Unicode token is raw UTF-8 (no ensure_ascii)", + "中文键" in uni_token_json and "\\u" not in uni_token_json) + uni_recomputed = m.build_fingerprint(_ieee754_json_loads(uni_token_json)) + check(f"{label}: Unicode digest stable after token re-parse", + uni_recomputed["digest"] == uni_fp["digest"] + and uni_recomputed["token"] == uni_fp["token"]) + # --- the effective execution gate must participate in the fingerprint runtime_kwargs = { "input_tf": "15", @@ -426,10 +964,188 @@ def provenance_rejects(module, digest) -> bool: check("copies agree: effective_inputs", rj.effective_inputs(FIXTURE_CPP, {"Fast EMA": "8"}) == rs.effective_inputs(FIXTURE_CPP, {"Fast EMA": "8"})) + check("copies agree: helper source is byte-identical", + _helper_block(RUN_JSON) == _helper_block(RUN_STRATEGY)) + + # --- JS RFC 8785/JCS direct-writer boundary (Node optional) ---------- + # Deterministic core vectors above are always mandatory. Node is optional: + # when present, this bridge checks a JS direct JCS writer (not plain + # JSON.stringify) against the Python encoder; when absent, print a skip + # note and do NOT count the bridge as a pass (pure-Python CI still + # enforces all core vectors). Token bytes remain authoritative. + node = shutil.which("node") + bridge_payload = { + "strategy": { + "initial_capital": 12345.0, + "commission_value": 0.0, + "default_qty_value": 1.5, + "safe_int": 9007199254740991, + }, + "nested": {"arr": [0.0, -0.0, 1e-6, 1e20, 1e21, -1e-7]}, + "flags": {"on": True, "off": False, "empty": None}, + # Array-index keys + __proto__: object-rebuild+stringify is wrong. + "10": 10, + "2": 2, + "__proto__": "own", + "indexish": {"2": 20, "10": 10, "__proto__": {"x": 1}}, + "arr_special": [{"10": 1, "2": 2, "__proto__": 3}], + # Unicode / JCS coverage beyond ASCII+numbers: + "中文键": "中文值", + "emoji": "😀", + "sep": "\u2028\u2029", + "ctrl": "\n\t\x00", + "\U00010000": "supplementary", + "\uE000": "pua", + } + py_canon = rs._canonical_fingerprint_json(bridge_payload) + py_digest = "sha256:" + hashlib.sha256(py_canon.encode("utf-8")).hexdigest() + # Pin expected UTF-16 key order on the bridge payload itself (always). + check("bridge payload: UTF-16 key order (U+10000 before U+E000)", + '"\U00010000":"supplementary","\uE000":"pua"' in py_canon + and py_canon.index('"\U00010000"') < py_canon.index('"\uE000"')) + check("bridge payload: Chinese/emoji/U+2028 raw (no ensure_ascii)", + "中文键" in py_canon and "😀" in py_canon + and "\u2028" in py_canon and "\\u4e2d" not in py_canon + and "\\ud83d" not in py_canon and "\\u2028" not in py_canon) + # Lexical UTF-16: "10" < "2" < "__proto__" — all retained as own keys. + check("bridge payload: special keys 10/2/__proto__ lexical order", + '"10":10' in py_canon + and '"2":2' in py_canon + and '"__proto__":"own"' in py_canon + and py_canon.index('"10":10') < py_canon.index('"2":2') + < py_canon.index('"__proto__":"own"') + and '"indexish":{"10":10,"2":20,"__proto__":{"x":1}}' in py_canon + and '"arr_special":[{"10":1,"2":2,"__proto__":3}]' in py_canon) + if node is None: + # Not a check() pass — optional bridge simply not exercised. + print(" SKIP JS bridge: node not available " + "(optional; core deterministic vectors remain mandatory)") + else: + # Emit non-canonical Python JSON that a Worker would re-canonicalize + # with a JCS direct writer (not plain JSON.stringify). ensure_ascii + # mangles non-ASCII; sort_keys uses code-point order so the JS path + # must re-sort U+10000 vs U+E000 into UTF-16 order. + py_legacy = json.dumps(bridge_payload, sort_keys=True, + separators=(",", ":"), ensure_ascii=True) + check("bridge: legacy Python JSON uses ensure_ascii + code-point sort", + "\\u4e2d" in py_legacy + and py_legacy.index("\\ue000") < py_legacy.index("\\ud800\\udc00")) + special_payload = { + "10": 10, + "2": 2, + "__proto__": "own", + "nested": [{"10": 1, "2": 2, "__proto__": 3}], + } + py_special = rs._canonical_fingerprint_json(special_payload) + script = r""" +const fs = require('fs'); +const input = JSON.parse(fs.readFileSync(0, 'utf8')); +// RFC 8785/JCS direct canonical writer over JS-parsed values. +// Do NOT rebuild a plain object (array-index key reorder; __proto__ drop). +function canonical(v) { + if (Array.isArray(v)) { + let s = '['; + for (let i = 0; i < v.length; i++) { + if (i) s += ','; + s += canonical(v[i]); + } + return s + ']'; + } + if (v !== null && typeof v === 'object') { + const keys = Object.keys(v).sort(); + let s = '{'; + for (let i = 0; i < keys.length; i++) { + const k = keys[i]; + if (i) s += ','; + s += JSON.stringify(k) + ':' + canonical(v[k]); + } + return s + '}'; + } + return JSON.stringify(v); +} +const fromLegacy = canonical(JSON.parse(input.legacy)); +const fromCanon = canonical(JSON.parse(input.canonical)); +const specialKeys = canonical(JSON.parse(input.special)); +// Direct Unicode/key vectors independent of the payload shape. +const direct = { + chinese: canonical({"中文键": "中文值"}), + emoji: canonical("😀"), + seps: canonical("\u2028\u2029"), + keyOrder: canonical( + Object.fromEntries( + ["\u{10000}", "\uE000"].sort().map((k, i) => [k, i + 1]) + ) + ), +}; +process.stdout.write(JSON.stringify({fromLegacy, fromCanon, specialKeys, direct})); +""" + proc = subprocess.run( + [node, "-e", script], + input=json.dumps({ + "legacy": py_legacy, + "canonical": py_canon, + "special": json.dumps( + special_payload, separators=(",", ":"), + ensure_ascii=False), + }, ensure_ascii=False), + text=True, capture_output=True, check=False) + if proc.returncode != 0: + check("JS bridge: node execution", False) + print(proc.stderr) + else: + out = json.loads(proc.stdout) + check("JS bridge: direct JCS writer matches Python canonical", + out["fromLegacy"] == py_canon + and out["fromCanon"] == py_canon) + # Digest recomputed from the JS-emitted text must match. + js_digest = "sha256:" + hashlib.sha256( + out["fromLegacy"].encode("utf-8")).hexdigest() + check("JS bridge: digest stable across JS boundary", + js_digest == py_digest + and rs.build_fingerprint(bridge_payload)["digest"] + == py_digest) + check("JS bridge: special keys 10/2/__proto__ match Python", + out["specialKeys"] == py_special + and out["specialKeys"] + == ('{"10":10,"2":2,"__proto__":"own",' + '"nested":[{"10":1,"2":2,"__proto__":3}]}') + and '"__proto__"' in out["specialKeys"] + and out["specialKeys"].index('"10"') + < out["specialKeys"].index('"2"') + < out["specialKeys"].index('"__proto__"')) + check("JS bridge: direct Chinese object matches", + out["direct"]["chinese"] + == rs._canonical_fingerprint_json({"中文键": "中文值"})) + check("JS bridge: direct emoji matches", + out["direct"]["emoji"] + == rs._canonical_fingerprint_json("😀")) + check("JS bridge: direct U+2028/U+2029 matches", + out["direct"]["seps"] + == rs._canonical_fingerprint_json("\u2028\u2029")) + check("JS bridge: direct UTF-16 key order matches", + out["direct"]["keyOrder"] + == rs._canonical_fingerprint_json( + {"\U00010000": 1, "\uE000": 2})) + # Fail-closed companions (out-of-domain / invalid I-JSON). + check("JS bridge companion: NaN rejected in Python contract", + all(_raises_value_error(rs, x) + for x in (math.nan, math.inf, -math.inf))) + check("JS bridge companion: out-of-domain ints rejected", + all(_raises_value_error(rs, x) for x in UNSAFE_INTEGERS)) + check("JS bridge companion: unpaired surrogates rejected", + all(_raises_value_error(rs, x) for x in UNPAIRED_SURROGATES)) print(f"\n{passed} passed, {failed} failed") return 1 if failed else 0 +def _raises_value_error(module, value) -> bool: + try: + module._canonical_fingerprint_json(value) + except ValueError: + return True + return False + + if __name__ == "__main__": sys.exit(main()) diff --git a/scripts/run_strategy.py b/scripts/run_strategy.py index 0542ec6..3886b91 100644 --- a/scripts/run_strategy.py +++ b/scripts/run_strategy.py @@ -307,8 +307,225 @@ def build_provenance(engine: dict, cpp_path, transpiled: bool, } +# Product accepted-type domain for exact Python integers: JavaScript +# Number.MAX/MIN_SAFE_INTEGER (= ±(2**53 - 1)). Rejecting larger exact ints +# guarantees unique lossless integer identity across generic ECMAScript +# consumers. This is not a claim that every such magnitude must round as +# binary64 — e.g. float(2**53) is exactly representable and remains legal +# on the float path, while exact int(2**53) is deliberately outside the +# product integer domain. Out-of-domain provenance yields no fingerprint +# under existing callers (exception → fingerprint None / skipped). +# Booleans are handled separately (bool subclasses int). +_JS_MAX_SAFE_INTEGER = 9007199254740991 +_JS_MIN_SAFE_INTEGER = -9007199254740991 + + +def _canonical_json_number(num: float) -> str: + """Serialize a finite IEEE-754 float via ECMAScript NumberToString. + + Matches ECMAScript NumberToString (RFC 8785 / JCS numeric form): + integral values have no trailing ``.0``, ``±0`` is ``0``, and + scientific notation uses ES exponent thresholds (``e`` when the + exponent is < -6 or >= 21). Non-finite values raise ValueError. + Float subclasses are normalized via base ``float.__float__`` first so + hooks such as ``__float__``/``__abs__``/comparisons/``__repr__`` cannot + alter the underlying binary64 value used for math and emission. + """ + # Plain built-in float: ignore subclass __float__/__abs__/__lt__/... + num = float.__float__(num) + if not math.isfinite(num): + raise ValueError( + "non-finite numbers are not permitted in fingerprint JSON") + if num == 0.0: + return "0" + negative = num < 0 + r = repr(abs(num)) + if "e" in r or "E" in r: + mant, exp_s = r.lower().split("e") + exp = int(exp_s) + if "." in mant: + whole, frac = mant.split(".") + digits_raw = whole + frac + n = exp + len(whole) + else: + digits_raw = mant + n = exp + len(digits_raw) + else: + if "." in r: + whole, frac = r.split(".") + digits_raw = whole + frac + n = len(whole) + else: + digits_raw = r + n = len(digits_raw) + # Leading-zero strip adjusts n so value = int(digits)*10^(n-k) holds. + lead = len(digits_raw) - len(digits_raw.lstrip("0")) + digits = digits_raw.lstrip("0") or "0" + if digits != "0": + n -= lead + while len(digits) > 1 and digits[-1] == "0": + digits = digits[:-1] + k = len(digits) + sign = "-" if negative else "" + if 0 < n <= 21: + if k <= n: + return sign + digits + ("0" * (n - k)) + return sign + digits[:n] + "." + digits[n:] + if -6 < n <= 0: + return sign + "0." + ("0" * (-n)) + digits + exp = n - 1 + exp_s = f"+{exp}" if exp >= 0 else str(exp) + if k == 1: + return sign + digits + "e" + exp_s + return sign + digits[0] + "." + digits[1:] + "e" + exp_s + + +def _reject_unpaired_surrogates(s: str, *, what: str) -> None: + """Fail closed on unpaired UTF-16 surrogates (invalid I-JSON / UTF-8).""" + for ch in s: + cp = ord(ch) + if 0xD800 <= cp <= 0xDFFF: + raise ValueError( + f"unpaired UTF-16 surrogate in fingerprint JSON {what}") + + +def _canonical_json_string(s: str) -> str: + """Serialize a string in RFC 8785 / JCS string form. + + JSON control characters, quotes, and backslashes are escaped; valid + Unicode (including non-ASCII, emoji, and U+2028/U+2029) is emitted as + raw code points (not ``ensure_ascii`` ``\\uXXXX`` escapes). Unpaired + surrogates raise ValueError rather than producing invalid I-JSON. + Str subclasses are normalized via base ``str.__str__`` first so + ``__str__``/``__iter__`` hooks cannot redirect iteration or emission. + """ + # Plain built-in str: ignore subclass __str__/__iter__/... + s = str.__str__(s) + _reject_unpaired_surrogates(s, what="string") + return json.dumps(s, ensure_ascii=False) + + +def _utf16_code_unit_key(s: str) -> bytes: + """Object-key sort key matching JCS / ECMAScript UTF-16 code unit order. + + Str subclasses are normalized via base ``str.__str__`` first so + ``__str__``/``__iter__``/``encode`` hooks cannot corrupt key order or + hide unpaired surrogates. + """ + # Plain built-in str: ignore subclass __str__/__iter__/encode hooks. + s = str.__str__(s) + _reject_unpaired_surrogates(s, what="object key") + return s.encode("utf-16-be") + + +def _canonical_fingerprint_json(value) -> str: + """Canonical JSON text for fingerprint token/digest bytes. + + Direct RFC 8785 / JCS-style canonical writer over the accepted Python + value tree. The resulting UTF-8 token bytes are authoritative for + ``digest``; verifiers should hash those bytes rather than assuming plain + ``JSON.stringify`` is itself JCS (it does not sort keys or implement + full JCS). Over values parsed under a JavaScript / IEEE-754 binary64 + number model, this matches a JCS direct encoder for the accepted types: + - objects: each key is normalized once via base ``str.__str__`` to a + plain built-in str; keys sorted by UTF-16 code unit order on those + normalized names; no whitespace. Duplicate normalized names raise + ValueError (fail closed — distinct str-subclass keys can override + ``__eq__``/``__hash__`` so both coexist in a Python dict while + ``str.__str__`` yields the same text; emitting both would produce + duplicate JSON names that JS silently drops). Non-str keys raise + TypeError. The retained original key is used only for value lookup. + - arrays: element order preserved; no whitespace + - numbers (float): ECMAScript NumberToString for every finite IEEE-754 + value after base ``float.__float__`` normalization (subclass hooks + ignored); non-finite numbers raise ValueError + - numbers (int): decimal digits only for exact integers inside the + product safe-integer domain [-(2**53-1), 2**53-1]. This is a strict + accepted-type policy guaranteeing unique lossless integer identity + across generic ECMAScript consumers — not a claim that every larger + individual value is unrepresentable as binary64. Exact integers + outside the domain raise ValueError (e.g. int 2**53); isolated + binary64 floats such as float(2**53) remain legal via the float path. + Int subclasses (e.g. IntEnum) are normalized via base ``int.__index__`` + to a plain built-in int before domain checks and digit emission, so + ``__index__``/``__int__``/comparison/``__str__``/``__repr__``/ + ``__format__`` hooks cannot change the value or bypass rejection. + Booleans are not integers. + - strings: JCS form — control/quote/backslash escapes, raw valid + Unicode; unpaired surrogates raise ValueError. Str subclasses are + normalized via base ``str.__str__`` so ``__str__``/``__iter__``/ + ``encode`` hooks cannot change emission, key order, or bypass + surrogate rejection. + - bools/null: ``true`` / ``false`` / ``null`` + """ + if value is None: + return "null" + if value is True: + return "true" + if value is False: + return "false" + if isinstance(value, str): + return _canonical_json_string(value) + if isinstance(value, int) and not isinstance(value, bool): + # Plain built-in int via base slot: subclass __index__/__int__/ + # comparison hooks must not bypass domain checks or alter digits + # (Python 3.9 IntEnum str was "Enum.NAME"). + value = int.__index__(value) + if value < _JS_MIN_SAFE_INTEGER or value > _JS_MAX_SAFE_INTEGER: + raise ValueError( + "integers outside the JavaScript safe-integer range " + f"[{_JS_MIN_SAFE_INTEGER}, {_JS_MAX_SAFE_INTEGER}] " + "are not permitted in fingerprint JSON") + return int.__repr__(value) + if isinstance(value, float): + return _canonical_json_number(value) + if isinstance(value, list): + return "[" + ",".join( + _canonical_fingerprint_json(v) for v in value) + "]" + if isinstance(value, dict): + # Normalize each original key exactly once to a plain built-in str. + # Distinct str subclasses can override __eq__/__hash__ so two keys + # coexist while str.__str__ yields the same text; emit the + # normalized name, look up values via the retained original key, + # and fail closed on duplicate normalized names. + items = [] # (normalized_key, original_key) + seen_normalized = set() + for k in value.keys(): + if not isinstance(k, str): + raise TypeError( + "fingerprint JSON object keys must be str, " + f"got {type(k).__name__}") + nk = str.__str__(k) + if nk in seen_normalized: + raise ValueError( + "duplicate fingerprint JSON object key after " + f"str normalization: {nk!r}") + seen_normalized.add(nk) + items.append((nk, k)) + parts = [] + for nk, orig in sorted(items, key=lambda ik: _utf16_code_unit_key(ik[0])): + parts.append( + _canonical_json_string(nk) + ":" + + _canonical_fingerprint_json(value[orig])) + return "{" + ",".join(parts) + "}" + raise TypeError( + f"fingerprint JSON cannot encode {type(value).__name__}") + + def build_fingerprint(provenance: dict) -> dict: - canonical = json.dumps(provenance, sort_keys=True, separators=(",", ":")) + """Build ``{token, digest, provenance}`` for a provenance dict. + + ``token`` is base64 of the canonical UTF-8 JSON bytes; ``digest`` is + ``sha256:`` + hex of those same bytes. Verifiers should treat the decoded + token bytes as authoritative and hash them directly. Re-canonicalizing a + decoded value tree requires an RFC 8785/JCS direct encoder with an + IEEE-754 binary64 number model — default Python ``json.loads`` may yield + ints outside the product integer domain for tokens such as ``1e20`` + (``100000000000000000000``). Project tests that rebuild from token text + use ``json.loads(text, parse_int=float)``. Out-of-domain inputs raise + here; existing callers yield no fingerprint (``None`` / skip). + """ + canonical = _canonical_fingerprint_json(provenance) raw = canonical.encode("utf-8") return { "token": base64.b64encode(raw).decode("ascii"),