feat: add kernel experiment workflow preflight

.gitignore

@@ -2,6 +2,7 @@
 runs/
 system/
 .worktrees/
+experiments/
 *.local
 __pycache__/
 .pytest_cache/

AGENTS.md

@@ -12,6 +12,8 @@ Build a repeatable, read-only telemetry and benchmark harness for the Surface Bo
 - Suspend tests must be operator-gated until explicitly approved.
 - Do not collect secrets or unrelated personal data.
 - Preserve raw artifacts; reports must be reproducible from them.
+- Kernel workflow tools must not install packages, sign UKIs, edit Limine, set a boot default, or boot a candidate without explicit human approval.
+- Preserve the known-good `linux-surface` package and boot entry while testing separately named `linux-eiger` candidates.
 
 ## Workflow
 

README.md

@@ -18,6 +18,10 @@ Project bootstrap. The implementation plan is in `docs/plan/eiger-telemetry.md`.
 
 Idle, browse, video, burst, and suspend/S0ix. AutoFDO/perf profiling is deliberately deferred until workload definitions are stable.
 
+## Kernel development workflow
+
+A safe Arch/linux-surface → `linux-eiger` build, Secure-Boot/Limine boot, benchmark, review, and rollback workflow is in [`docs/kernel-development-workflow.md`](docs/kernel-development-workflow.md). Its read-only build-provenance preflight tool is documented in [`kernel/README.md`](kernel/README.md).
+
 ## Delivery workflow
 
 Work is managed on the Hermes Kanban board and implemented in isolated worktrees. Changes flow through PM specification, implementation, independent QA/review, and a human merge gate. See `AGENTS.md` and `CONTRIBUTING.md`.

docs/kernel-development-workflow.md

@@ -0,0 +1,255 @@
+# Eiger Kernel Development and Measurement Workflow
+
+This runbook defines the safe path from a known-good `linux-surface` baseline to a custom Eiger kernel, then to evidence-based promotion. It is tailored to this host:
+
+- Arch Linux / Omarchy
+- `linux-surface` 6.19.8.arch1-3 baseline
+- Limine boot loader
+- Secure Boot enabled
+- measured UKI boot path
+- Surface Book 2 with two independently reported batteries
+
+The workflow is deliberately split into **build**, **boot**, and **measurement** gates. A package that builds is not automatically bootable. A kernel that boots is not automatically comparable. A benchmark that completes is not automatically evidence of an improvement.
+
+## Non-negotiable rules
+
+1. Keep the vendor `linux-surface` package and its known-good UKI/boot entry intact until an Eiger candidate has passed the complete acceptance sequence.
+2. Build as an unprivileged user. Do not use `sudo make install` and do not replace the active kernel in place.
+3. Package custom work under a distinct package base/name such as `linux-eiger` and `linux-eiger-headers`; use a distinct kernel local version such as `-eiger.r1`.
+4. Boot selection, package installation, Limine/UKI updates, Secure-Boot signing, kernel removal, and release/publishing are human gates.
+5. Kernel comparison changes one intended independent variable at a time. Every candidate has a pinned source revision, patch manifest, configuration hash, package hash, and boot identity.
+6. Benchmark collection is read-only. Experimental controls are recorded, never silently tuned.
+
+## Roles and durable artifacts
+
+| Stage | Owner | Required durable artifact |
+| --- | --- | --- |
+| Hypothesis | PM + owner | issue/card, expected metric and risk |
+| Baseline | QA | accepted raw run directories + report |
+| Source/config | engineer | source revision, patch series, config hash/diff |
+| Build | engineer | preflight JSON, build log, package SHA-256, package list |
+| Install/boot | owner | package transaction log, signing evidence, boot entry, rollback entry |
+| Smoke test | QA | boot fingerprint, hardware/journal checks |
+| Experiment | QA | paired benchmark artifacts and exclusion ledger |
+| Review/promotion | reviewer + owner | comparison report, decision record, human merge/release approval |
+
+## 0. One-time preparation
+
+### Source policy
+
+Base Eiger packaging on the official `linux-surface` Arch packaging tree, which tracks the Arch Linux kernel packaging plus Surface patches. Pin the exact Git commit used for each experiment; do not build from an unnamed moving checkout.
+
+Keep local sources outside the telemetry repository, for example:
+
+```text
+~/src/linux-surface-packaging/       # pinned upstream packaging checkout
+~/src/linux-eiger-packaging/         # Eiger packaging fork/worktree
+~/build/eiger/<experiment-id>/       # disposable build output/logs
+~/eiger-telemetry/                   # protocol, collector, reports, immutable runs
+```
+
+The `eiger-telemetry` repository must store manifests, configuration fragments, patch references, checksums, and reports—not multi-gigabyte kernel sources, object files, packages, or raw personal telemetry.
+
+### Build dependencies
+
+The current host has `make`, `gcc`, `bison`, and `pahole`, but strict preflight currently reports missing `bc` and `flex`. Install only the reviewed build dependencies before attempting a package build. On Arch, the typical baseline is `base-devel bc flex` plus any package-specific dependencies declared by the pinned PKGBUILD.
+
+Do not install dependencies during a formal benchmark matrix. Dependency changes belong to a recorded preparation phase followed by a new baseline fingerprint.
+
+### Secure Boot and boot safety
+
+This host has Secure Boot enabled and boots a measured UKI through Limine. Before an Eiger candidate is installed, document:
+
+- how the existing `linux-surface` package creates/signs its UKI;
+- the custom Eiger package's UKI path and signing step;
+- the exact Limine entry title for both candidate and known-good baseline;
+- one-shot boot selection/recovery procedure;
+- how to recover from a failed candidate boot without removing `linux-surface`.
+
+Do not assume `sbctl` automatically signs a custom package. Verify the generated artifact and signing status after installation. This is a human-gated operation.
+
+## 1. Create a hypothesis card
+
+Every kernel change starts with a single falsifiable statement, for example:
+
+> On the Surface Book 2, patch set X reduces idle wakeups and raises package C10 residency without regressing burst energy or resume reliability.
+
+The card records:
+
+- intended source/config/patch change;
+- expected metric direction and scenario;
+- primary metric and guardrail metrics;
+- kernel A (baseline) and B (candidate) labels;
+- known risks and the rollback entry;
+- benchmark protocol revision and required number of paired runs.
+
+Do not bundle unrelated scheduler, driver, compiler, config, or firmware changes into one candidate.
+
+## 2. Capture the baseline before changing source
+
+1. Boot the baseline `linux-surface` entry.
+2. Run `eiger-kernel-preflight` against the pinned baseline source/config and save its JSON alongside the experiment record.
+3. Capture a full system fingerprint.
+4. Collect the protocol-defined baseline matrix for the relevant power profile and observed charge state.
+5. Preserve raw runs, hash manifests, summaries, excluded runs, and environment controls.
+6. Require at least three accepted A/B-comparable repetitions before making a performance claim. Five is the target.
+
+The first provisional capture is a smoke artifact only. It must not be used as a baseline claim because it lacks controlled conditions and battery energy resolution.
+
+## 3. Create a reproducible candidate
+
+For each candidate, create an experiment directory such as:
+
+```text
+experiments/2026-08-31-idle-wakeup-r1/
+├── experiment.json
+├── baseline-preflight.json
+├── candidate-preflight.json
+├── source.lock                 # upstream URL + exact revision
+├── patches/series              # references/checksums, not copied upstream trees
+├── config/
+│   ├── baseline.config.sha256
+│   ├── candidate.config
+│   └── config.diff
+├── build/
+│   ├── command.txt
+│   ├── environment.allowlist.json
+│   ├── build.log
+│   ├── packages.sha256
+│   └── package-file-list.txt
+└── results/                    # links/IDs to immutable telemetry runs
+```
+
+Required candidate identity:
+
+- pinned linux-surface packaging revision and upstream kernel revision;
+- patch series names and SHA-256 values;
+- candidate config SHA-256 and audited config diff;
+- compiler and linker versions; build command; source-tree cleanliness;
+- package filename/version, package SHA-256, and module `pkgbase` value;
+- expected `uname -r` / local version.
+
+The preflight command is intentionally read-only:
+
+```sh
+kernel/bin/eiger-kernel-preflight \
+  --source ~/src/linux-eiger-packaging/kernel/src/linux-* \
+  --config ~/src/linux-eiger-packaging/kernel/src/linux-*/.config \
+  --output experiments/<experiment-id>/candidate-preflight.json \
+  --baseline-label surface-6.19.8 \
+  --candidate-label eiger-6.19.8-r1 \
+  --strict
+```
+
+It creates a provenance artifact, checks a source `Makefile` and exact configuration hash, records source Git identity/dirty state, checks build-tool availability, and warns about insufficient output-filesystem space. It does not compile, install, invoke sudo, or modify the source tree.
+
+## 4. Build packages, never install the raw kernel
+
+From a clean, pinned Arch package worktree:
+
+1. Change `pkgbase` and package names to `linux-eiger`; do not conflict with `linux-surface`.
+2. Apply exactly the experiment's named patch/config changes.
+3. Run strict preflight and preserve the JSON result.
+4. Build as the normal user with the pinned package recipe, recording the exact command and complete output log.
+5. Generate SHA-256 checksums and package file lists.
+6. Verify the package includes a distinct kernel/module path and, if relevant, matching headers.
+7. Do not run `pacman -U`, modify `/boot`, or boot the candidate until a reviewer has inspected the build artifacts and the owner approves the install gate.
+
+A build failure is data: retain the log and preflight context. Do not repair a source/config issue by making undocumented changes.
+
+## 5. Human-gated installation and one-shot boot
+
+After approval:
+
+1. Confirm backup and known-good `linux-surface` package/boot entry remain available.
+2. Install the `linux-eiger` packages without removing `linux-surface`.
+3. Verify generated UKI/initramfs location, signing status, and Limine entry using the actual system hooks.
+4. Record package transaction output and artifact hashes.
+5. Set a one-shot candidate boot, never change the permanent default first.
+6. Boot candidate B and verify `uname -r`, boot ID, cmdline, module `pkgbase`, and UKI/entry identity.
+7. If it fails to boot or breaks essential hardware, select the known-good baseline entry and record the failure. Do not continue performance testing.
+
+## 6. Candidate smoke gate
+
+Before performance collection, QA verifies a minimal non-performance acceptance set:
+
+- boot and filesystem mount success;
+- display, keyboard/trackpad, Wi-Fi, audio, camera/tablet or other relevant Surface hardware;
+- battery/AC telemetry still visible for both batteries;
+- no unexpected kernel errors/warnings during settling;
+- fingerprint captured with candidate identity;
+- privileged `turbostat` path available through the reviewed narrow wrapper, if package-power metrics are required;
+- suspend remains separately gated and is not started automatically.
+
+The smoke gate catches regressions. It does not establish an efficiency improvement.
+
+## 7. Measure A/B performance and power
+
+Use `docs/benchmark-protocol.md` and its power/charge-state extension. The comparability tuple is:
+
+```text
+kernel × scenario × power_profile × observed_charge_state × repetition
+```
+
+For each pair:
+
+1. Precompute and record a balanced A/B ordering seed.
+2. Boot the named kernel condition, then settle.
+3. Confirm observed power profile and natural charge state. Skip—not substitute—when they do not match.
+4. Run explicit workload/phase markers and read-only capture.
+5. Verify hashes, duration, completeness, accepted/excluded status, and all control values.
+6. Reboot before switching kernel condition.
+7. Retain outliers and excluded runs with reasons.
+
+Required metrics depend on the hypothesis but generally include:
+
+- battery energy/power where reliable, per battery and combined;
+- package/core/GFX/RAM watts from privileged turbostat, never mislabelled as laptop watts;
+- C-state/package-state/GPU RC6 residency;
+- wakeup, interrupt, softirq, and PCI runtime-PM deltas;
+- thermal maximum and throttling evidence;
+- completion time and energy-per-task for burst workloads;
+- suspend elapsed time, S0ix counters, and battery loss only in the separately approved suspend flow.
+
+## 8. Analysis and decision
+
+The report must:
+
+- show all raw run IDs, accepted pairs, skipped/excluded pairs, and reasons;
+- reject comparisons with changed protocol, collector, source/config identity, controls, kernel boot identity, profile, or charge state;
+- show median, mean, dispersion, sample count, and individual pair deltas;
+- distinguish a correlation from a causal claim;
+- state metric availability limitations and observer effects;
+- report regressions in guardrails even when the primary metric improves.
+
+A candidate is **not promoted** if it has fewer than three accepted pairs, lacks the primary measurement, regresses reliability, or has unexplained control/identity differences.
+
+## 9. Promotion and rollback
+
+Human review chooses one of:
+
+- **reject:** retain evidence; remove only the experimental package after confirming the known-good kernel boots;
+- **iterate:** create a new candidate card with one documented delta;
+- **provisional keep:** retain both boot entries and continue long-term historian observation;
+- **promote:** set the Eiger entry as default only after a documented human approval and rollback test.
+
+Never delete the baseline kernel or its signed boot path as part of an experiment. Keep the last known-good package and package checksum available for rollback.
+
+## Mandatory human gates
+
+| Gate | Why it is human-only |
+| --- | --- |
+| Installing local kernel packages | Changes bootable system state |
+| Secure-Boot signing/enrollment | Changes security/boot trust state |
+| Editing Limine entries/defaults | Can make the host unbootable |
+| First boot of a candidate | Requires recovery judgement |
+| Suspend experiment | May strand the device or conceal a resume failure |
+| Merging/publishing a kernel package or release | External/release commitment |
+
+## Workflow status on this host
+
+- Baseline kernel and boot path: identified.
+- Preflight tooling: implemented and tested on the workflow branch.
+- Telemetry collector: integrated in a separate uncommitted worktree but QA has open defects; not ready for formal comparison.
+- Power/charge preflight, narrow turbostat wrapper, overnight scheduling, historian, markers: separate Kanban work remains queued/in progress.
+- First custom compile: intentionally blocked until package-source/config selection, missing build dependencies, and the Secure-Boot/UKI installation plan are reviewed.

kernel/README.md

@@ -0,0 +1,25 @@
+# Kernel workflow tooling
+
+This directory contains **read-only preparation tools** for Eiger kernel experiments. It does not build, install, sign, boot, or remove a kernel.
+
+## `eiger-kernel-preflight`
+
+Generate a machine-readable provenance artifact before a build:
+
+```sh
+kernel/bin/eiger-kernel-preflight \
+  --source ~/src/linux-eiger-packaging/kernel/src/linux-* \
+  --config ~/src/linux-eiger-packaging/kernel/src/linux-*/.config \
+  --output experiments/<experiment-id>/candidate-preflight.json \
+  --baseline-label surface-6.19.8 \
+  --candidate-label eiger-6.19.8-r1 \
+  --strict
+```
+
+It validates safe experiment labels, source-tree structure, the supplied kernel configuration hash, source Git identity/dirty state, required build-tool availability, and output-filesystem free space. It writes an atomic JSON artifact.
+
+`--strict` exits nonzero if any required build tool is unavailable. The tool never invokes a build command, `sudo`, package manager, bootloader, signer, or telemetry collector.
+
+Required tools are `make`, `gcc`, `bc`, `bison`, `flex`, and `pahole`. `clang` and `ccache` are recorded as optional. The exact package recipe may require additional dependencies; treat its pinned PKGBUILD as authoritative.
+
+See [`../docs/kernel-development-workflow.md`](../docs/kernel-development-workflow.md) for the complete build, boot, measurement, review, and rollback process.