hyperpolymath · hyperpolymath · Sep 27, 2026 · Sep 27, 2026 · Sep 27, 2026
@@ -2,16 +2,18 @@
 # SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell <j.d.a.jewell@open.ac.uk>
 #
 # Files that `just check-kyaml` does NOT require to be canonical KYAML, because a
-# tool that is not this repository writes them. One path prefix per line; `#`
-# starts a comment. Every entry needs a reason and an owner, and the list is
-# reviewed with docs/pilots/kyaml-pilot.md — an exemption without a reason is drift.
+# tool that is not this repository writes them. This is a check-only exemption:
+# `use-kyaml` and `use-yaml` still convert and roll these files back. One path
+# prefix per line; `#` starts a comment. Every entry needs a reason and an owner,
+# and the list is reviewed with docs/pilots/kyaml-pilot.md — an exemption without
+# a reason is drift.
 #
 # Owner ruling, 2026-09-26: the drift is ACCEPTED IN WRITING (standards
 # YAML-POLICY.adoc §5 step 6 requires either KYAML-emitting bots or an explicit
-# written acceptance). The files below ARE converted to KYAML — they are simply
-# not gated, because Dependabot and `gh actions-lock` rewrite their `uses:` pins
-# in block style, and a gate that goes red on a bot's schedule trains everyone to
-# ignore it. Reconcile after a bot PR with: just use-kyaml
+# written acceptance). These paths remain in the conversion scope, but are not
+# gated: Dependabot and `gh actions-lock` rewrite their `uses:` pins in block
+# style, and a gate that goes red on a bot's schedule trains everyone to ignore
+# it. Reconcile after a bot PR with: just use-kyaml
 #
 # .github/workflows/ci.yml      Dependabot + gh actions-lock rewrite `uses:` pins
 # .github/workflows/ui.yml      same

@@ -16,6 +16,11 @@ empty and a normal merge surfaces all 159 shared‑but‑different files at once
 Nothing in this guide changes upstream's operational behaviour unless you choose
 it to; the default is "behave exactly like upstream".
 
+For the separate KYAML authoring-format pilot, see the [upstream architecture and
+transition audit](./kyaml-upstream-adoption.md). It identifies the runtime config readers,
+user-editable YAML writers, GitHub Actions boundary, and the evidence required before
+changing any tracked files.
+
 ## The idea, in layers
 
 0. **Re‑anchor** — replay the fork's commits one‑by‑one onto upstream so git

@@ -0,0 +1,133 @@
+<!--
+SPDX-License-Identifier: CC-BY-SA-4.0
+SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) <j.d.a.jewell@open.ac.uk>
+-->
+# KYAML adoption: upstream architecture and a low-risk path
+
+**Purpose:** explain where an authoring-format change would touch Joshua's upstream
+repository, separate safe formatting from runtime behavior, and give each review a small,
+explicit scope. This is an architecture review, not a request to adopt KYAML or a claim that
+an upstream change has been tested or merged.
+
+**Snapshot reviewed:** `JoshuaJewell/MetaManifold-WebUI` `main` at
+`ecefb1c72b3d2515e7086024b14227ef13329605` (2026-09-27). Recheck paths and tests against
+the target commit when preparing a PR.
+
+## Bottom line
+
+KYAML is a source-authoring convention, not a replacement parser or a new application data
+format. Upstream already reads configuration with `YAML.jl` 0.4. If a file remains valid
+YAML and its parsed values are unchanged, its existing consumers need no new dependency and
+no API or pipeline change. **Keep `YAML.jl`, existing serializers, and user configuration
+formats unchanged during the pilot.**
+
+The boundary that matters is who owns and writes each file:
+
+| Files / surface | Current consumer or writer | Safe transition rule |
+|---|---|---|
+| `config/defaults/*.yml` | Julia `YAML.load_file`; pipeline defaults feed the default → global → study → group → run cascade in `src/core/config.jl`. | Convert as repository-owned inputs; compare the values read by `YAML.jl` before/after. Do not change the cascade or hashes as part of a format-only patch. |
+| `config/ci/*.yml`, `bench/**.yml`, tracked sample `data/MiSeq_SOP/pipeline.yml` | CI/bootstrap, benchmark, or example-data readers. | Convert as authored inputs and run their existing consumers. Keep the sample's values and provenance intact. |
+| `.github/workflows/*.yml` | GitHub Actions' workflow parser—not the application YAML loader. Several files contain multiline shell `run:` blocks; Dependabot and `gh actions-lock` also rewrite workflow pins. | Convert only after an actual GitHub Actions parse/run probe. Preserve trigger, job IDs, `needs`, permissions, matrix values, environment, working directory, shell, and `GITHUB_ENV` behavior. Bot drift may exempt these files from the canonicality gate, but not from conversion or rollback. |
+| `config/pipeline.yml`, `config/primers.yml`, `config/databases.yml`, `config/composition.yml`, `config/tools.yml`, and per-study `pipeline.yml` overrides | Machine/user-editable overlays; routes in `src/server/routes/config.jl`, `composition.jl`, and `results.jl` write YAML with `YAML.write`. Root `.gitignore` excludes the machine-level config files. | Do not reformat user state or change save behavior in the pilot. The reader continues to accept YAML; runtime writes may remain block-style YAML. Keep generated/user files outside the tracked-file gate. |
+| `run_config.yml`, provenance sidecars, result/filter files | Generated from the config cascade or written as run output; `src/core/config.jl` and `src/core/provenance.jl` own these paths. | Leave generated artifacts and their writers alone. They are not repository-authored source configuration. |
+
+This means Joshua does **not** have to take a KYAML dependency or immediately migrate
+user-generated files. A format-only change to the tracked defaults can be read by the
+existing YAML loader; an edited user overlay can continue to be written in ordinary YAML.
+
+## Risks that deserve explicit evidence
+
+1. **Semantic equivalence, not visual similarity.** Quoting is intentional: KYAML quotes
+   strings and keys that YAML readers may otherwise interpret differently (`on`, `no`,
+   version-like strings). The gate must compare the actual parsed data from the production
+   `YAML.jl` reader before and after emission for every tracked configuration document.
+   The emitter's own YAML→KYAML→YAML round-trip is not enough by itself.
+2. **GitHub workflow interpretation.** The quoted `"on"` key must still trigger the
+   workflow. Unit tests cannot establish this. A real PR run must report the expected jobs
+   under their unchanged stable names, with no missing required status context.
+3. **Shell extraction is a separate behavior change.** Moving `run: |` blocks to
+   `scripts/ci/*.sh` can change the working directory, shell options, expression expansion,
+   environment, exit behavior, permissions, or writes to `GITHUB_ENV`. Keep that work in a
+   separate reviewed change before the workflow-format rewrite, with a shell-level test or
+   the same real CI execution.
+4. **Bot rewrites.** A bot reverting a workflow to block style should not make the quality
+   gate flaky, but operators must still be able to canonicalize and roll back those files.
+   The exemption must apply only to the gate—not to the converter.
+5. **User data and serializer scope.** Applying the converter to ignored machine configs,
+   run directories, or provenance outputs would create churn and could rewrite user state.
+   The source-format pilot should enumerate tracked YAML only and must not alter the
+   application's YAML serialization paths.
+6. **Reversibility.** Preserve a byte-exact Git revert for the format-only conversion.
+   `just use-yaml` is a canonical re-emission, not a promise to recover every original
+   whitespace byte; the Git revert is the exact escape hatch.
+
+## Small, reviewable sequence
+
+### 0. Hardening patch — no source YAML rewritten
+
+Land the KYAML tool correction and its regression tests first. The drift list is a gate
+allowlist, not a migration exclusion. Test the converter and rollback against an exempt
+workflow path, and parse every tracked YAML document. Compare application-consumed
+configuration with the production `YAML.jl` reader; validate workflow files with GitHub
+Actions itself. Correct the corpus census and document the actual mutable-file boundary.
+This patch changes no workflow or application behavior.
+
+### 1. Fix and establish the CI verdict
+
+Resolve the current fork CI `startup_failure` before treating CI as evidence. A run with
+zero jobs has no test verdict. Then establish a passing baseline for the KYAML unit tests,
+workflow syntax checks, and existing full test suite on the exact review commit.
+
+### 2. Extract CI shell blocks (only if still required)
+
+If the project retains the pilot's script-extraction decision, do it separately from
+formatting. For every moved step, record the old and new `working-directory`, shell,
+`env`, GitHub expression, permissions, `set -e`/pipefail, and outputs. Run the workflow
+before changing its YAML style.
+
+### 3. Format-only conversion
+
+Convert only tracked, repository-authored `.yml`/`.yaml` files. Keep the migration as one
+format-only change (or one clearly named commit within the PR) so its purpose and revert
+boundary stay obvious. Its review description should report:
+
+- the exact file list and parser-equivalence result;
+- every decision count from `just kyaml-report` (`null`, quoting, comment movement);
+- the GitHub Actions run IDs that parsed and executed the converted workflows;
+- the required status contexts observed;
+- files intentionally not gated because of bot rewriting; and
+- confirmation that `Project.toml`, serializers, APIs, config-cascade behavior, run hashes,
+  and user data are unchanged.
+
+Enable the canonicality gate in the same migration change. Do not claim the GitHub workflow
+proof is complete merely because the ordinary YAML parser accepts the file.
+
+## Findings from this checkout that must be resolved before migration
+
+- The repository has **17 tracked YAML files**, not 16. Two workflow paths are listed in
+  `config/kyaml/drift.txt`; that makes 15 files subject to the canonicality gate after
+  conversion, while all 17 still need parser/round-trip coverage.
+- Before the current hardening change, the CLI filtered the drift paths out of every mode.
+  That meant the workflows would not be converted by `use-kyaml` and would not be restored
+  by `use-yaml`, despite the pilot saying they are in scope. The implementation now applies
+  those exemptions only to `--check`; a regression test covers conversion, gate exemption,
+  and rollback.
+- The CLI also referred to `Stats` from outside its `KYAML` module without qualifying the
+  name. It now uses `KYAML.Stats`; the new CLI-level regression test exercises that path.
+- The old parser-corpus test skipped the two workflows. It now includes every tracked file,
+  compares both emitted forms with `YAML.jl` for application-consumed YAML, and leaves
+  workflow acceptance to GitHub Actions itself. The current Actions workflow contains only
+  a placeholder comment about adding the canonicality gate; it does not yet run
+  `check-kyaml`. Add the real gate in the same change that converts the corpus.
+- **These Julia tests have not yet been executed in this sandbox**: Julia is absent, and the
+  latest fork CI run ended in `startup_failure` before starting any jobs. Therefore this
+  audit does not certify that all 17 files pass or that GitHub has accepted a converted
+  workflow.
+
+## Upstream source pointers
+
+- [`Project.toml`](https://github.com/JoshuaJewell/MetaManifold-WebUI/blob/main/Project.toml) — YAML.jl 0.4 remains a runtime dependency.
+- [`src/core/config.jl`](https://github.com/JoshuaJewell/MetaManifold-WebUI/blob/main/src/core/config.jl) — defaults and override cascade; generated `run_config.yml`.
+- [`src/server/routes/config.jl`](https://github.com/JoshuaJewell/MetaManifold-WebUI/blob/main/src/server/routes/config.jl) — config reads and atomic YAML writes for editable overlays.
+- [`src/core/primers_library.jl`](https://github.com/JoshuaJewell/MetaManifold-WebUI/blob/main/src/core/primers_library.jl) and [`src/core/databases_library.jl`](https://github.com/JoshuaJewell/MetaManifold-WebUI/blob/main/src/core/databases_library.jl) — user-edited library files and normalization/validation boundaries.
+- [`.github/workflows/ci.yml`](https://github.com/JoshuaJewell/MetaManifold-WebUI/blob/main/.github/workflows/ci.yml) — CI parser, shell steps, Julia matrix, and tool installation.
@@ -9,7 +9,9 @@ use-kyaml` / `just use-yaml`, and the escape hatch is one `git revert`.
 
 Authority: `hyperpolymath/standards`, `3-practice/YAML-POLICY.adoc` (rules Y-1, Y-2, Y-3,
 adoption order §5). This document is the operating manual for the pilot; the policy stays
-the authority, and where the two disagree the policy wins and this file is wrong.
+the authority, and where the two disagree the policy wins and this file is wrong. The
+upstream application touchpoints and the low-risk adoption sequence are mapped in
+[`docs/integration/kyaml-upstream-adoption.md`](../integration/kyaml-upstream-adoption.md).
 
 ## 1. Why this is worth doing, in one paragraph
 
@@ -80,7 +82,7 @@ Refused, by name and line number, leaving the file untouched:
 | Multi-line plain scalars | Folded into one line by YAML rules, so the source bytes are not recoverable. |
 | An end-of-line comment on a block scalar | The scalar owns the rest of the line; there is nowhere lossless to put the comment. |
 
-This repository's corpus needs none of those: a census of the 16 tracked YAML files found
+This repository's corpus needs none of those: a census of the 17 tracked YAML files found
 **zero** anchors, aliases, tags, multi-document streams or directives; block scalars in two
 files (`ci.yml`, 22 of them; `.github/ISSUE_TEMPLATE/bug_report.yml`, 2); 12 `~` nulls; and
 comment-bearing lines concentrated in `ci.yml` (313), `config/defaults/pipeline.yml` (97) and

@@ -1234,7 +1234,7 @@ function _kyaml_cli(argv::Vector{String})::Int
 
             Options:
               --report              print the decisions taken per file
-              --skip-file PATH      file listing paths to leave alone (default config/kyaml/drift.txt)
+              --skip-file PATH      --check-only exemptions (default config/kyaml/drift.txt)
               --expect kyaml|yaml   style --check compares against (default kyaml)
 
             Exit codes: 0 ok, 1 a check failed, 2 a file was refused (nothing was written).
@@ -1256,10 +1256,18 @@ function _kyaml_cli(argv::Vector{String})::Int
         end
     end
     isempty(paths) && (paths = KYAML.git_yaml_paths())
-    kept = [p for p in paths if !any(s -> startswith(p, s), skip)]
+    # Drift exemptions are a gate policy, not a migration boundary. Bot-owned
+    # workflows are still converted (and can be rolled back); only --check omits
+    # them, so an external formatter cannot make the canonicality gate flap.
+    skipped = String[]
+    kept = paths
+    if mode == "check"
+        skipped = [p for p in paths if any(s -> startswith(p, s), skip)]
+        kept = [p for p in paths if !any(s -> startswith(p, s), skip)]
+    end
 
     failures = String[]
-    reports = Dict{String,Stats}()
+    reports = Dict{String,KYAML.Stats}()
     rendered = Dict{String,String}()
     for p in kept
         stats = KYAML.Stats()
@@ -1285,7 +1293,7 @@ function _kyaml_cli(argv::Vector{String})::Int
     if mode == "check"
         if isempty(failures)
             println("kyaml check: $(length(kept)) file(s) in canonical $expect form" *
-                    (isempty(skip) ? "" : ", $(length(skip)) exempt"))
+                    (isempty(skipped) ? "" : ", $(length(skipped)) exempt"))
             return 0
         end
         println(stderr, "kyaml check: $(length(failures)) file(s) are not canonical $expect:")
@@ -1302,7 +1310,7 @@ function _kyaml_cli(argv::Vector{String})::Int
         written += 1
     end
     println("kyaml $mode: $(written) of $(length(kept)) file(s) rewritten" *
-            (isempty(skip) ? "" : ", $(length(skip)) exempt"))
+            (isempty(skipped) ? "" : ", $(length(skipped)) exempt"))
     if report
         for p in sort(collect(keys(reports)))
             s = reports[p]

@@ -15,20 +15,9 @@ const KYAML_REPO_ROOT = normpath(joinpath(@__DIR__, "..", ".."))
 
 include(KYAML_TOOL_PATH)
 
+using YAML
 using .KYAML: parse_document, render_kyaml, render_yaml, check, git_yaml_paths, KyamlError
 
-function kyaml_exempt_prefixes()::Vector{String}
-    drift = joinpath(KYAML_REPO_ROOT, "config", "kyaml", "drift.txt")
-    isfile(drift) || return String[]
-    prefixes = String[]
-    for line in eachline(drift)
-        t = strip(line)
-        (isempty(t) || startswith(t, "#")) && continue
-        push!(prefixes, t)
-    end
-    return prefixes
-end
-
 @testset "KYAML switch" begin
 
     @testset "comments keep their association" begin
@@ -131,19 +120,61 @@ end
         end
     end
 
-    @testset "the repository's own YAML is inside the subset" begin
-        exempt = kyaml_exempt_prefixes()
+    @testset "drift exemptions affect the gate, not conversion or rollback" begin
+        mktempdir() do dir
+            rel = ".github/workflows/ci.yml"
+            path = joinpath(dir, rel)
+            mkpath(dirname(path))
+            source = "# retained workflow comment\nname: \"CI\"\n"
+            write(path, source)
+            drift = joinpath(dir, "drift.txt")
+            write(drift, rel * "\n")
+
+            cd(dir) do
+                @test _kyaml_cli(["--to-kyaml", "--skip-file", drift, rel]) == 0
+                kyaml = read(path, String)
+                @test kyaml != source
+                @test kyaml == render_kyaml(parse_document(rel, source))
+                @test _kyaml_cli(["--check", "--skip-file", drift, rel]) == 0
+                @test _kyaml_cli(["--to-yaml", "--skip-file", drift, rel]) == 0
+                restored = read(path, String)
+                @test restored != kyaml
+                @test restored == render_yaml(parse_document(rel, kyaml))
+            end
+        end
+    end
+
+    @testset "tracked YAML parses; application files preserve YAML.jl semantics" begin
+        # Drift exemptions govern the canonicality gate only. They are still parsed
+        # here because conversion and rollback must cover every tracked YAML file.
+        # Compare with the production YAML.jl reader, not only this tool's own parser:
+        # a self-round-trip cannot prove that the application or CI sees the same data.
         failures = String[]
         for rel in git_yaml_paths(KYAML_REPO_ROOT)
-            any(prefix -> startswith(rel, prefix), exempt) && continue
             path = joinpath(KYAML_REPO_ROOT, rel)
             isfile(path) || continue
             try
-                doc = parse_document(path, read(path, String))
-                render_kyaml(doc)
-                render_yaml(doc)
+                source = read(path, String)
+                doc = parse_document(path, source)
+                kyaml = render_kyaml(doc)
+                yaml = render_yaml(doc)
+                mktempdir() do dir
+                    original_path = joinpath(dir, "original.yml")
+                    kyaml_path = joinpath(dir, "converted.kyaml")
+                    yaml_path = joinpath(dir, "roundtrip.yml")
+                    write(original_path, source)
+                    write(kyaml_path, kyaml)
+                    write(yaml_path, yaml)
+                    # GitHub Actions is the consumer of workflow files; its actual
+                    # parser/run is tested by CI, not approximated with YAML.jl.
+                    # Compare all application-consumed YAML with the production reader.
+                    if !startswith(rel, ".github/workflows/")
+                        original_value = YAML.load_file(original_path)
+                        @test YAML.load_file(kyaml_path) == original_value
+                        @test YAML.load_file(yaml_path) == original_value
+                    end
+                end
             catch err
-                err isa KyamlError || rethrow()
                 push!(failures, rel * " -> " * sprint(showerror, err))
             end
         end