From 2fff0b9054864673c4d545e544299c81642d3fce Mon Sep 17 00:00:00 2001 From: Brian Gyss Date: Mon, 2 Feb 2026 22:12:37 -0800 Subject: [PATCH] Add XCI intake and DKCR HD milestone specs --- PLANS.md | 51 +++++++++++++++++++ RESEARCH.md | 6 +++ specs/README.md | 3 ++ specs/SPEC-180-XCI-INTAKE.md | 65 ++++++++++++++++++++++++ specs/SPEC-190-VIDEO-BASED-VALIDATION.md | 61 ++++++++++++++++++++++ specs/SPEC-200-DKCR-HD-FIRST-LEVEL.md | 57 +++++++++++++++++++++ 6 files changed, 243 insertions(+) create mode 100644 specs/SPEC-180-XCI-INTAKE.md create mode 100644 specs/SPEC-190-VIDEO-BASED-VALIDATION.md create mode 100644 specs/SPEC-200-DKCR-HD-FIRST-LEVEL.md diff --git a/PLANS.md b/PLANS.md index fc6d9ca..9bfdd74 100644 --- a/PLANS.md +++ b/PLANS.md @@ -20,6 +20,9 @@ This file tracks implementation work derived from specs that do not yet have a c - SPEC-150 Homebrew Asset Packaging - SPEC-160 AArch64 Decode Coverage - SPEC-170 Function Discovery and Control-Flow Graph +- SPEC-180 XCI Title Intake +- SPEC-190 Video-Based Validation +- SPEC-200 DKCR HD First-Level Milestone (macOS/aarch64) ## SPEC-000: Project Charter and Ethics Outcome @@ -271,3 +274,51 @@ Exit criteria (from SPEC-170) - A synthetic binary with a conditional branch yields at least two blocks and correct edges. - Direct call targets are discovered and lifted as separate functions. - The lifted module is deterministic when run twice on the same input. + +## SPEC-180: XCI Title Intake +Outcome +- Intake XCI inputs with user-supplied keys and extract code/assets deterministically. + +Work items +- [ ] Define the XCI intake CLI path and config schema extensions. +- [ ] Integrate keyset validation and explicit Program NCA selection. +- [ ] Extract ExeFS/NSO into deterministic segment blobs with hashes recorded. +- [ ] Emit RomFS assets to a separate asset output root and record in manifest. +- [ ] Add non-proprietary tests for intake validation and asset separation rules. + +Exit criteria (from SPEC-180) +- XCI intake emits deterministic ExeFS/NSO outputs and a manifest with hashes. +- Missing keys or ambiguous Program NCA selection fails with clear errors. +- Code outputs and assets remain separated in all outputs. + +## SPEC-190: Video-Based Validation +Outcome +- Validate the recompiled output against a reference gameplay video without emulator traces. + +Work items +- [ ] Define a reference timeline for the first level and store it in `reference_video.toml`. +- [ ] Implement a capture workflow for macOS/aarch64 runtime output. +- [ ] Add a comparison step that computes video and audio similarity metrics. +- [ ] Generate a `validation-report.json` with pass/fail and drift summaries. +- [ ] Document manual review steps for mismatches. + +Exit criteria (from SPEC-190) +- A single run produces a validation report for the first level. +- Similarity metrics are stable across two consecutive runs. +- Validation artifacts remain external and are referenced via provenance metadata. + +## SPEC-200: DKCR HD First-Level Milestone (macOS/aarch64) +Outcome +- Produce a macOS/aarch64 static recompilation of DKCR HD that reaches and plays the first level. + +Work items +- [ ] Complete XCI intake for the DKCR HD title (SPEC-180 inputs and outputs). +- [ ] Identify required OS services and implement or stub them in the runtime. +- [ ] Implement the minimal GPU translation path needed for the first level. +- [ ] Create a per-title config and patch set for DKCR HD. +- [ ] Run video-based validation against the first level (SPEC-190). + +Exit criteria (from SPEC-200) +- The macOS/aarch64 build boots and reaches the first playable level. +- First-level gameplay matches the reference video within defined tolerances. +- No proprietary assets or keys are stored in the repo or build outputs. diff --git a/RESEARCH.md b/RESEARCH.md index 3c6433e..4828e79 100644 --- a/RESEARCH.md +++ b/RESEARCH.md @@ -77,6 +77,12 @@ Needed research: - Switch NSO format and compression: https://switchbrew.org/wiki/NSO - Homebrew ABI entrypoint and loader config: https://switchbrew.org/wiki/Homebrew_ABI - NACP title metadata format: https://switchbrew.org/wiki/NACP +- Switch NCA container format: https://switchbrew.org/wiki/NCA +- hactool (XCI/NCA extraction and keyset handling): https://github.com/SciresM/hactool +- hactoolnet (XCI/NCA extraction with user keys): https://github.com/Thealexbarney/hactoolnet +- nstool (XCI/NCA/NSO extraction): https://github.com/jakcron/nstool +- Ghidra SLEIGH language reference (p-code semantics): https://github.com/NationalSecurityAgency/ghidra/blob/master/GhidraDocs/languages/html/sleigh.html +- sleigh library (p-code lifting implementation): https://github.com/lifting-bits/sleigh ## Research Deliverables - A research summary for each category with sources. diff --git a/specs/README.md b/specs/README.md index bf0f8eb..4d557a3 100644 --- a/specs/README.md +++ b/specs/README.md @@ -26,6 +26,9 @@ This folder contains the project specs for the Switch static recompilation prese - SPEC-150-HOMEBREW-ASSET-PACKAGING.md - SPEC-160-AARCH64-DECODE-COVERAGE.md - SPEC-170-FUNCTION-DISCOVERY-CFG.md +- SPEC-180-XCI-INTAKE.md +- SPEC-190-VIDEO-BASED-VALIDATION.md +- SPEC-200-DKCR-HD-FIRST-LEVEL.md ## Template - SPEC-TEMPLATE.md diff --git a/specs/SPEC-180-XCI-INTAKE.md b/specs/SPEC-180-XCI-INTAKE.md new file mode 100644 index 0000000..2144b7a --- /dev/null +++ b/specs/SPEC-180-XCI-INTAKE.md @@ -0,0 +1,65 @@ +# SPEC-180: XCI Title Intake + +## Status +Draft v0.1 + +## Purpose +Define how the pipeline ingests a user-supplied XCI and extracts code and assets while preserving legal separation and deterministic outputs. + +## Goals +- Accept XCI inputs with provenance metadata and deterministic hashing. +- Extract Program NCAs, ExeFS, and NSO segments for recompilation. +- Extract RomFS assets into a separate, user-managed directory. + +## Non-Goals +- Distributing keys, firmware, or proprietary assets. +- Circumventing encryption without user-provided keys. +- Packaging assets into the repo or build outputs. + +## Background +- Retail titles are distributed as NCAs inside XCI images. +- NCAs are encrypted and require user-supplied keys for extraction. +- NSO executables may contain compressed segments that must be decompressed before lifting. + +## Requirements +- Intake must require a valid provenance record before processing an XCI. +- Intake must accept a user-provided keyset path and fail with clear errors when keys are missing. +- Program NCA selection must be explicit and logged (TitleID, content type, version). +- ExeFS and NSO segments must be extracted deterministically and hashed. +- RomFS assets must be emitted to a separate asset output root, never mixed with code outputs. +- The intake manifest must record tool versions, hashes, and extracted file sizes. + +## Interfaces and Data +- Inputs + - `input.xci_path` and `input.keys_path` in provenance metadata. + - Optional `title.toml` overrides for main program selection. +- Outputs + - `intake/` directory with NCA metadata, ExeFS, and NSO segment blobs. + - `assets/` directory for RomFS extraction (external to code output). + - `module.json` and `manifest.json` with hashes and offsets. + +## Deliverables +- CLI intake command that accepts an XCI plus keyset and emits deterministic extraction outputs. +- A validator that enforces asset separation and provenance requirements for XCI inputs. +- Documentation describing the intake flow and supported keyset formats. + +## Open Questions +- How should update and DLC NCAs be layered or merged? +- Should the pipeline support NSP inputs in addition to XCI? +- What is the minimal metadata set needed to select the correct Program NCA? + +## Acceptance Criteria +- Given a user-supplied XCI and keyset, the intake emits ExeFS/NSO and RomFS outputs deterministically. +- Missing keys or ambiguous Program NCA selection results in a clear, actionable error. +- Code outputs and asset outputs are separated and hashed in the manifest. + +## Risks +- Legal and policy risk if asset separation is violated. +- Toolchain drift if external extraction tools change output formats. +- Titles with multiple Program NCAs may require manual selection rules. + +## References +- https://github.com/SciresM/hactool +- https://github.com/Thealexbarney/hactoolnet +- https://github.com/jakcron/nstool +- https://switchbrew.org/wiki/NCA diff --git a/specs/SPEC-190-VIDEO-BASED-VALIDATION.md b/specs/SPEC-190-VIDEO-BASED-VALIDATION.md new file mode 100644 index 0000000..045326f --- /dev/null +++ b/specs/SPEC-190-VIDEO-BASED-VALIDATION.md @@ -0,0 +1,61 @@ +# SPEC-190: Video-Based Validation + +## Status +Draft v0.1 + +## Purpose +Define a validation workflow that compares recompiled output against a reference gameplay video when no instrumented emulator is available. + +## Goals +- Provide a repeatable comparison method using video and audio captures. +- Establish a first-level milestone gate based on reference footage. +- Produce a machine-readable report for regressions. + +## Non-Goals +- Pixel-perfect or audio-sample-perfect matching. +- Replacing manual playtesting for qualitative issues. +- Distributing copyrighted reference footage. + +## Background +- The project does not currently have access to instrumented Switch emulation traces. +- A long-form gameplay video is available as a behavioral reference. +- Video comparison can detect major regressions in rendering, timing, and audio. + +## Requirements +- A reference timeline must define the first-level start and completion timecodes. +- Validation must capture native macOS output at a stable resolution and frame rate. +- A comparison step must compute frame similarity (perceptual hash or SSIM) and audio similarity. +- The report must highlight drift, dropped frames, or audio desync beyond thresholds. +- Validation artifacts must remain outside the repo and be referenced via provenance metadata. + +## Interfaces and Data +- `reference_video.toml` with: + - input video path + - timecodes for key events + - expected resolution and frame rate +- `validation-report.json` with: + - similarity metrics + - timecode drift data + - pass/fail summary + +## Deliverables +- A capture script that records the recompiled runtime output. +- A comparison script that generates a validation report from reference and capture. +- Documentation describing the expected workflow and thresholds. + +## Open Questions +- What similarity thresholds best indicate a playable first level? +- How should camera motion or cutscenes be treated in the comparison? +- Which input playback method yields the most deterministic run? + +## Acceptance Criteria +- A reference timeline for the first level is defined and versioned. +- A single run produces a validation report with explicit pass/fail for the first level. +- Reported metrics are stable within tolerance across two consecutive runs. + +## Risks +- Frame pacing differences may cause false mismatches. +- Audio compression or capture devices may skew similarity metrics. + +## References +- TBD diff --git a/specs/SPEC-200-DKCR-HD-FIRST-LEVEL.md b/specs/SPEC-200-DKCR-HD-FIRST-LEVEL.md new file mode 100644 index 0000000..7b8b774 --- /dev/null +++ b/specs/SPEC-200-DKCR-HD-FIRST-LEVEL.md @@ -0,0 +1,57 @@ +# SPEC-200: DKCR HD First-Level Milestone (macOS/aarch64) + +## Status +Draft v0.1 + +## Purpose +Define the first title milestone for the DKCR HD XCI on macOS/aarch64, using video-based validation to confirm the first level is playable. + +## Goals +- Produce a macOS/aarch64 static recompilation that boots and reaches the first level. +- Validate the run against the reference gameplay video for timing, visuals, and audio. +- Keep all proprietary assets external and user-supplied. + +## Non-Goals +- Completing or validating the full game. +- Achieving exact performance parity with Switch hardware. +- Distributing assets, keys, or copyrighted footage. + +## Background +- A retail XCI and a long-form gameplay reference video are available locally. +- The project already defines pipeline, runtime, and validation scaffolding for homebrew. +- This milestone focuses on proving the retail-title path works end to end. + +## Requirements +- Intake must extract Program NCA, ExeFS, and NSO segments from the XCI using user keys. +- Recompiled output must build on macOS/aarch64 and link against the runtime ABI. +- Runtime must implement enough OS services, GPU translation, audio, and input to reach the first level. +- RomFS assets must be loaded from an external, user-managed path. +- Validation must compare the first level segment against the reference video and record results. + +## Interfaces and Data +- `title.toml` for DKCR HD configuration (stubbed services, patches, asset paths). +- `provenance.toml` for XCI and reference video inputs. +- `validation-report.json` for the first-level comparison results. + +## Deliverables +- Title-specific configuration and patch set to reach the first playable level. +- A reproducible build and run command for macOS/aarch64. +- A validation report demonstrating the first-level gate. + +## Open Questions +- Which services are the minimum set required to reach the first level? +- What frame rate and resolution should be treated as the baseline for comparison? +- Which shader or GPU features are required for the first level? + +## Acceptance Criteria +- The macOS/aarch64 build boots and reaches the first playable level. +- The first-level run is playable and passes the video-based validation threshold. +- No proprietary assets are committed to the repo or build outputs. + +## Risks +- GPU translation gaps may block rendering. +- Missing OS services may cause boot failure. +- Validation thresholds may need tuning for frame pacing variance. + +## References +- TBD