RFC(DRAFT): versioned ObservationEvidence digest and safe legacy verification #196

RFC status

DRAFT — proposal only; NOT accepted, implemented, merged, or released. This is the Core review required by GOVERNANCE.md and RFC process. Linked implementation bug: #192. Characterization tests: Draft PR #195 (exact-head CI, Docs Site, Security all passed on 9d5d025ea0a3bd11916969cc024afae78cb5d700).

Problem and repository evidence

On main 89cfdcb10ab407f5c1ef84b8cda05894b5dd4a2a, packages/evidence-normalizer/src/normalizer.ts computes sha256(JSON.stringify(evidencePayload, Object.keys(evidencePayload).sort())). The array replacer recursively filters nested property names, so differences in nested result.exitCode, context.environmentVariables, etc. may vanish from the preimage (this is not a SHA collision). The existing TCK normalizer test checks digest shape but not nested-field sensitivity. No general external exploit or all-bundle impact is established.

Existing source of truth: Docs/evidence/canonicalization-migration-matrix.json already identifies evidence-normalizer-digest as IDENTITY_CRITICAL, migrationStrategy: NOT_SAFE_TO_MIGRATE_YET, historicalArtifacts: HISTORICAL_ARTIFACTS_NOT_FOUND. It explicitly requires a consumer/verifier contract and nested-key tests. Inventory IDs and migration flags must be preserved, not silently updated to imply authorization.

Proposed decision (subject to approval)

1. Define a dedicated immutable legacy verification profile for the actual array-replacer bytes, e.g. observation-top-level-replacer-v0. Never alias this to legacy-ts-v0; that existing profile uses recursive key sorting and is different.
2. Define a dedicated profile binding for new ObservationEvidence artifacts based on the existing semantiq-canonical-json-v1 canonicalizer, not a novel serializer. Its hash subject is a documented, explicit projection of released/redacted ObservationEvidence, excluding evidenceDigest and its algorithm/profile metadata (no circular hashing). Exact projection and optional-member rules must be frozen in cross-language-compatible fixtures. undefined optional members require an explicit, pre-declared omission rule, not JSON round-trip coercion; unsupported numbers and values fail closed.
3. New artifacts include explicit profile metadata in a versioned/approved envelope or compatible schema extension (do not silently repurpose sha256:<hex> or infer identity from matching bytes). Verification requires recognized explicit profile; unknown/absent profile has no automatic fallback. For historical untagged artifacts require separately authenticated legacy-provenance context; absent that, classification is UNKNOWN/NOT ESTABLISHED, not verification PASS.
4. Preserve historical digests unchanged. If a valid old artifact is explicitly identified as legacy, reproduce the exact historical preimage for verification; this is legacy consistency only and does not attest omitted nested fields. Never upgrade an old digest by silently recomputing it or asserting it covered information it never hashed.
5. Address privacy before V1 generation: current ObservationEvidence.context.environmentVariables and decision.commandArray can carry raw values while stream/diff redaction is pattern-based. Specify field-aware allowlisting/redaction and what is intended to be committed, plus an internal/public disclosure boundary. Raw credentials must not appear in published evidence, fixtures, CI logs, or public metadata. Hashing a secret is not a substitute for redaction.
6. Keep default EvidenceNormalizer.normalize() and current public contract unchanged until this RFC, a data-sensitivity policy, migration strategy and verifier semantics are approved. A prospective, non-routed pure proof-of-concept is permitted for test coverage only, not as production acceptance.

Alternatives to adjudicate

• A — Explicit V1 opt-in + immutable legacy verification (recommended for review): preserves historical interoperability and fail-closed semantics; needs profile/tag envelope design.
• B — Replace digest hashing in place: rejected as proposed because same sha256:<hex> wire identity silently changes and legacy artifacts cannot be distinguished.
• C — Keep current hash as sufficient: rejected as evidence-identity assurance because nested payload mutations can be invisible; may remain temporarily only as explicitly limited legacy behavior.
• D — Hash whole raw bundle: deferred: risks secrets/PII and does not match the sanitized/public evidence identity scope.

Acceptance tests / review gates

• Approve producer/consumer/verifier inventory, including direct callers, serialized artifacts, schema/version boundaries and any historical persisted evidence; absence of public archive is not absence of private history.
• Exact historical byte/hash vectors for legacy, including top-level order and optional field handling, and explicit statement of what is not bound by old hashes.
• V1 fixture: changing each material nested redacted evidence field changes digest; equivalent nested object order does not; arrays retain order; optional fields behave explicitly.
• Reject V1 undefined in arrays, unsafe floats, nonfinite numbers, negative zero, cycles, unsupported objects/Unicode; no silent coercion/fallback.
• Verify V1 under both TS and Python canonical implementation using shared fixtures before making cross-language interoperability claims.
• Redaction tests for env vars/commands/streams/diffs/reasoning; no secrets in published subject/CI logs.
• Unknown/missing/mismatched profile fail-closed; malformed digest rejects; unverified legacy surface never promoted to VERIFIED_V1.
• Obtain CODEOWNER approval and all protected-branch CI; update canonicalization registry/migration inventory only after decision and implementation evidence.
• Keep schema identity, benchmark scoring, releases, promotions, and main unchanged until review authorizes a versioned implementation.

Scientific / policy boundary

Integrity does not prove observation truth, goal completion, adoption, independent replication or security. Preserve Observed ≠ Inferred, Absence ≠ Counterevidence, Bundle integrity ≠ truth.

Ownership / dependencies

Core/contract/security review by current @Logorythmus-org/semantiq-maintainers per CODEOWNERS. Related: #188, #189, #191, #192 and roadmap #177. This RFC is intentionally separate from speculative OTel exports (#190).

Unresolved decisions

Which released fields are identity-critical after redaction? Is an explicit digestProfile field backwards compatible for all readers, or must an envelope version change? What is the trusted source of a legacy tag for unversioned artifacts? Are any persisted legacy evidence artifacts actually in use outside public repository history?