feat: topic-config restore (#136), Phase 1 header preflight (#137), evidence contract v2 (#138)

[sionsmith]

Summary

Cyber-recovery hardening for the restore/validation path. One integrated, fully verified commit closing three issues (per-issue boundaries run through shared files — config.rs, manifest.rs, restore/engine.rs — so splitting would have produced non-building intermediate commits).

Closes #136, closes #137, closes #138.

#136 — topic configuration capture and restore

• Backup captures allowlisted mutable topic configuration overrides + replication-factor metadata into the manifest.
• Restore applies captured configuration to created topics, with existing_topic_config_policy for pre-existing targets, per-topic overrides, and drift reporting.
• Supporting Kafka client / partition-router / storage / TLS fixes.

#137 — real Phase 1 header preflight

Replaces the placeholder ThreePhaseRestore::validate_phase1_headers (always returned success with sample_records_checked: 0, no production call sites).

• New restore/preflight.rs: scans every selected topic/partition/segment (binary + legacy JSON formats, honoring partition/PITR filters); structured per-partition states full/partial/missing/empty/data_missing/corrupt/indeterminate; zero records is never a positive pass.
• Consumer-offset recovery (reset_consumer_offsets/auto_consumer_groups) fails before the engine connects to the target — before topic creation, config restore, purge, or produce — when required tracking metadata is missing/partial/corrupt/unprovable. auto_consumer_groups now requires a valid consumer-groups snapshot; a reset request that could never act now fails instead of silently skipping.
• Wired into RestoreEngine::run, run_all_phases (ThreePhaseReport.header_preflight), validate_phase1_headers, dry_run/validate-restore (DryRunReport.header_preflight, exit 1 on blocking findings), and the CLI output.
• restore.header_preflight: auto|full|skip — skip is the documented emergency escape hatch (pre-PR behavior, loud UNVERIFIED warnings).
• Docs: docs/restore-preflight.md. Evidence: issue comment.

#138 — versioned byte-level evidence contract (v2)

Eliminates the self-referential report_sha256 (stored/signed JSON carried an empty field while the PDF displayed a digest matching no stored artifact).

• Report schema 1.1 never embeds its own digest; the digest + ECDSA P-256 signature live in a detached envelope (kafka-backup/evidence-envelope/v2) computed as a DSSE v1 signature over PAE(payloadType, exact stored report bytes) (spec PAE vector pinned by test).
• Single serialization: stored JSON, digest, envelope, CLI display, and PDF display all refer to the same bytes; verification hashes stored bytes as read (no re-serialization/canonicalization). Legacy v1 text .sig artifacts — including ones produced by the original bug — verify via an explicitly tested path; unknown schemas are rejected, never reinterpreted.
• Artifact creation/verification centralized in evidence::emit/evidence::envelope; the CLI is a thin wrapper and the enterprise distribution consumes the same functions (cross-binary verification demonstrated).
• to_canonical_json → to_deterministic_json, documented and unit-proven not RFC 8785 (divergences pinned: -0.0, 1e-6, UTF-16 vs UTF-8 key order, >2^53 integers).
• Docs: docs/evidence-contract.md (byte-level contract, RFC 8785 / DSSE / in-toto citations, deliberate deviations). Evidence: issue comment.

Verification

• cargo test --workspace: 421 passed / 0 failed / 91 ignored (Docker-gated); new suites: preflight_contract_test 17/17, evidence_contract_test 10/10.
• Docker E2E: preflight_e2e 2/2 (failed preflight leaves the target unchanged + mutation-detecting control), evidence 3/3, restore-validate 2/2, issue_67 10/10, offset_semantics 4/4.
• cargo fmt --check clean; cargo clippy --workspace --all-targets 0 warnings; cargo audit no vulnerabilities (3 known informational advisories); release build OK.
• Release-binary demo: signed evidence emitted, digest == shasum of stored bytes == envelope == PDF; cross-verified by the enterprise binary; tampered copy fails (exit 1); header-less backup refused before any target mutation.

Notes for reviewers

• Behavior changes (fail-closed, documented in docs/restore-preflight.md): auto_consumer_groups with a missing/invalid snapshot and reset_consumer_offsets with nothing to act on now fail before mutation; header_preflight: skip restores old behavior explicitly.
• Evidence .sig objects are now JSON envelopes; old binaries fail to parse them explicitly (never a wrong verdict). New binaries verify both formats.
• Branch is based on ec2995a; origin/main has advanced since — happy to merge main into the branch if you want it current before review.

🤖 Generated with Claude Code