diff --git a/config/kyaml/drift.txt b/config/kyaml/drift.txt index 75399ac..ef0e192 100644 --- a/config/kyaml/drift.txt +++ b/config/kyaml/drift.txt @@ -2,16 +2,18 @@ # SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell # # Files that `just check-kyaml` does NOT require to be canonical KYAML, because a -# tool that is not this repository writes them. One path prefix per line; `#` -# starts a comment. Every entry needs a reason and an owner, and the list is -# reviewed with docs/pilots/kyaml-pilot.md — an exemption without a reason is drift. +# tool that is not this repository writes them. This is a check-only exemption: +# `use-kyaml` and `use-yaml` still convert and roll these files back. One path +# prefix per line; `#` starts a comment. Every entry needs a reason and an owner, +# and the list is reviewed with docs/pilots/kyaml-pilot.md — an exemption without +# a reason is drift. # # Owner ruling, 2026-09-26: the drift is ACCEPTED IN WRITING (standards # YAML-POLICY.adoc §5 step 6 requires either KYAML-emitting bots or an explicit -# written acceptance). The files below ARE converted to KYAML — they are simply -# not gated, because Dependabot and `gh actions-lock` rewrite their `uses:` pins -# in block style, and a gate that goes red on a bot's schedule trains everyone to -# ignore it. Reconcile after a bot PR with: just use-kyaml +# written acceptance). These paths remain in the conversion scope, but are not +# gated: Dependabot and `gh actions-lock` rewrite their `uses:` pins in block +# style, and a gate that goes red on a bot's schedule trains everyone to ignore +# it. Reconcile after a bot PR with: just use-kyaml # # .github/workflows/ci.yml Dependabot + gh actions-lock rewrite `uses:` pins # .github/workflows/ui.yml same diff --git a/docs/integration/README.md b/docs/integration/README.md index 7de6376..c30f57b 100644 --- a/docs/integration/README.md +++ b/docs/integration/README.md @@ -16,6 +16,11 @@ empty and a normal merge surfaces all 159 shared‑but‑different files at once Nothing in this guide changes upstream's operational behaviour unless you choose it to; the default is "behave exactly like upstream". +For the separate KYAML authoring-format pilot, see the [upstream architecture and +transition audit](./kyaml-upstream-adoption.md). It identifies the runtime config readers, +user-editable YAML writers, GitHub Actions boundary, and the evidence required before +changing any tracked files. + ## The idea, in layers 0. **Re‑anchor** — replay the fork's commits one‑by‑one onto upstream so git diff --git a/docs/integration/kyaml-upstream-adoption.md b/docs/integration/kyaml-upstream-adoption.md new file mode 100644 index 0000000..272994c --- /dev/null +++ b/docs/integration/kyaml-upstream-adoption.md @@ -0,0 +1,133 @@ + +# KYAML adoption: upstream architecture and a low-risk path + +**Purpose:** explain where an authoring-format change would touch Joshua's upstream +repository, separate safe formatting from runtime behavior, and give each review a small, +explicit scope. This is an architecture review, not a request to adopt KYAML or a claim that +an upstream change has been tested or merged. + +**Snapshot reviewed:** `JoshuaJewell/MetaManifold-WebUI` `main` at +`ecefb1c72b3d2515e7086024b14227ef13329605` (2026-09-27). Recheck paths and tests against +the target commit when preparing a PR. + +## Bottom line + +KYAML is a source-authoring convention, not a replacement parser or a new application data +format. Upstream already reads configuration with `YAML.jl` 0.4. If a file remains valid +YAML and its parsed values are unchanged, its existing consumers need no new dependency and +no API or pipeline change. **Keep `YAML.jl`, existing serializers, and user configuration +formats unchanged during the pilot.** + +The boundary that matters is who owns and writes each file: + +| Files / surface | Current consumer or writer | Safe transition rule | +|---|---|---| +| `config/defaults/*.yml` | Julia `YAML.load_file`; pipeline defaults feed the default → global → study → group → run cascade in `src/core/config.jl`. | Convert as repository-owned inputs; compare the values read by `YAML.jl` before/after. Do not change the cascade or hashes as part of a format-only patch. | +| `config/ci/*.yml`, `bench/**.yml`, tracked sample `data/MiSeq_SOP/pipeline.yml` | CI/bootstrap, benchmark, or example-data readers. | Convert as authored inputs and run their existing consumers. Keep the sample's values and provenance intact. | +| `.github/workflows/*.yml` | GitHub Actions' workflow parser—not the application YAML loader. Several files contain multiline shell `run:` blocks; Dependabot and `gh actions-lock` also rewrite workflow pins. | Convert only after an actual GitHub Actions parse/run probe. Preserve trigger, job IDs, `needs`, permissions, matrix values, environment, working directory, shell, and `GITHUB_ENV` behavior. Bot drift may exempt these files from the canonicality gate, but not from conversion or rollback. | +| `config/pipeline.yml`, `config/primers.yml`, `config/databases.yml`, `config/composition.yml`, `config/tools.yml`, and per-study `pipeline.yml` overrides | Machine/user-editable overlays; routes in `src/server/routes/config.jl`, `composition.jl`, and `results.jl` write YAML with `YAML.write`. Root `.gitignore` excludes the machine-level config files. | Do not reformat user state or change save behavior in the pilot. The reader continues to accept YAML; runtime writes may remain block-style YAML. Keep generated/user files outside the tracked-file gate. | +| `run_config.yml`, provenance sidecars, result/filter files | Generated from the config cascade or written as run output; `src/core/config.jl` and `src/core/provenance.jl` own these paths. | Leave generated artifacts and their writers alone. They are not repository-authored source configuration. | + +This means Joshua does **not** have to take a KYAML dependency or immediately migrate +user-generated files. A format-only change to the tracked defaults can be read by the +existing YAML loader; an edited user overlay can continue to be written in ordinary YAML. + +## Risks that deserve explicit evidence + +1. **Semantic equivalence, not visual similarity.** Quoting is intentional: KYAML quotes + strings and keys that YAML readers may otherwise interpret differently (`on`, `no`, + version-like strings). The gate must compare the actual parsed data from the production + `YAML.jl` reader before and after emission for every tracked configuration document. + The emitter's own YAML→KYAML→YAML round-trip is not enough by itself. +2. **GitHub workflow interpretation.** The quoted `"on"` key must still trigger the + workflow. Unit tests cannot establish this. A real PR run must report the expected jobs + under their unchanged stable names, with no missing required status context. +3. **Shell extraction is a separate behavior change.** Moving `run: |` blocks to + `scripts/ci/*.sh` can change the working directory, shell options, expression expansion, + environment, exit behavior, permissions, or writes to `GITHUB_ENV`. Keep that work in a + separate reviewed change before the workflow-format rewrite, with a shell-level test or + the same real CI execution. +4. **Bot rewrites.** A bot reverting a workflow to block style should not make the quality + gate flaky, but operators must still be able to canonicalize and roll back those files. + The exemption must apply only to the gate—not to the converter. +5. **User data and serializer scope.** Applying the converter to ignored machine configs, + run directories, or provenance outputs would create churn and could rewrite user state. + The source-format pilot should enumerate tracked YAML only and must not alter the + application's YAML serialization paths. +6. **Reversibility.** Preserve a byte-exact Git revert for the format-only conversion. + `just use-yaml` is a canonical re-emission, not a promise to recover every original + whitespace byte; the Git revert is the exact escape hatch. + +## Small, reviewable sequence + +### 0. Hardening patch — no source YAML rewritten + +Land the KYAML tool correction and its regression tests first. The drift list is a gate +allowlist, not a migration exclusion. Test the converter and rollback against an exempt +workflow path, and parse every tracked YAML document. Compare application-consumed +configuration with the production `YAML.jl` reader; validate workflow files with GitHub +Actions itself. Correct the corpus census and document the actual mutable-file boundary. +This patch changes no workflow or application behavior. + +### 1. Fix and establish the CI verdict + +Resolve the current fork CI `startup_failure` before treating CI as evidence. A run with +zero jobs has no test verdict. Then establish a passing baseline for the KYAML unit tests, +workflow syntax checks, and existing full test suite on the exact review commit. + +### 2. Extract CI shell blocks (only if still required) + +If the project retains the pilot's script-extraction decision, do it separately from +formatting. For every moved step, record the old and new `working-directory`, shell, +`env`, GitHub expression, permissions, `set -e`/pipefail, and outputs. Run the workflow +before changing its YAML style. + +### 3. Format-only conversion + +Convert only tracked, repository-authored `.yml`/`.yaml` files. Keep the migration as one +format-only change (or one clearly named commit within the PR) so its purpose and revert +boundary stay obvious. Its review description should report: + +- the exact file list and parser-equivalence result; +- every decision count from `just kyaml-report` (`null`, quoting, comment movement); +- the GitHub Actions run IDs that parsed and executed the converted workflows; +- the required status contexts observed; +- files intentionally not gated because of bot rewriting; and +- confirmation that `Project.toml`, serializers, APIs, config-cascade behavior, run hashes, + and user data are unchanged. + +Enable the canonicality gate in the same migration change. Do not claim the GitHub workflow +proof is complete merely because the ordinary YAML parser accepts the file. + +## Findings from this checkout that must be resolved before migration + +- The repository has **17 tracked YAML files**, not 16. Two workflow paths are listed in + `config/kyaml/drift.txt`; that makes 15 files subject to the canonicality gate after + conversion, while all 17 still need parser/round-trip coverage. +- Before the current hardening change, the CLI filtered the drift paths out of every mode. + That meant the workflows would not be converted by `use-kyaml` and would not be restored + by `use-yaml`, despite the pilot saying they are in scope. The implementation now applies + those exemptions only to `--check`; a regression test covers conversion, gate exemption, + and rollback. +- The CLI also referred to `Stats` from outside its `KYAML` module without qualifying the + name. It now uses `KYAML.Stats`; the new CLI-level regression test exercises that path. +- The old parser-corpus test skipped the two workflows. It now includes every tracked file, + compares both emitted forms with `YAML.jl` for application-consumed YAML, and leaves + workflow acceptance to GitHub Actions itself. The current Actions workflow contains only + a placeholder comment about adding the canonicality gate; it does not yet run + `check-kyaml`. Add the real gate in the same change that converts the corpus. +- **These Julia tests have not yet been executed in this sandbox**: Julia is absent, and the + latest fork CI run ended in `startup_failure` before starting any jobs. Therefore this + audit does not certify that all 17 files pass or that GitHub has accepted a converted + workflow. + +## Upstream source pointers + +- [`Project.toml`](https://github.com/JoshuaJewell/MetaManifold-WebUI/blob/main/Project.toml) — YAML.jl 0.4 remains a runtime dependency. +- [`src/core/config.jl`](https://github.com/JoshuaJewell/MetaManifold-WebUI/blob/main/src/core/config.jl) — defaults and override cascade; generated `run_config.yml`. +- [`src/server/routes/config.jl`](https://github.com/JoshuaJewell/MetaManifold-WebUI/blob/main/src/server/routes/config.jl) — config reads and atomic YAML writes for editable overlays. +- [`src/core/primers_library.jl`](https://github.com/JoshuaJewell/MetaManifold-WebUI/blob/main/src/core/primers_library.jl) and [`src/core/databases_library.jl`](https://github.com/JoshuaJewell/MetaManifold-WebUI/blob/main/src/core/databases_library.jl) — user-edited library files and normalization/validation boundaries. +- [`.github/workflows/ci.yml`](https://github.com/JoshuaJewell/MetaManifold-WebUI/blob/main/.github/workflows/ci.yml) — CI parser, shell steps, Julia matrix, and tool installation. diff --git a/docs/pilots/kyaml-pilot.md b/docs/pilots/kyaml-pilot.md index e2d555e..71c2e9e 100644 --- a/docs/pilots/kyaml-pilot.md +++ b/docs/pilots/kyaml-pilot.md @@ -9,7 +9,9 @@ use-kyaml` / `just use-yaml`, and the escape hatch is one `git revert`. Authority: `hyperpolymath/standards`, `3-practice/YAML-POLICY.adoc` (rules Y-1, Y-2, Y-3, adoption order §5). This document is the operating manual for the pilot; the policy stays -the authority, and where the two disagree the policy wins and this file is wrong. +the authority, and where the two disagree the policy wins and this file is wrong. The +upstream application touchpoints and the low-risk adoption sequence are mapped in +[`docs/integration/kyaml-upstream-adoption.md`](../integration/kyaml-upstream-adoption.md). ## 1. Why this is worth doing, in one paragraph @@ -80,7 +82,7 @@ Refused, by name and line number, leaving the file untouched: | Multi-line plain scalars | Folded into one line by YAML rules, so the source bytes are not recoverable. | | An end-of-line comment on a block scalar | The scalar owns the rest of the line; there is nowhere lossless to put the comment. | -This repository's corpus needs none of those: a census of the 16 tracked YAML files found +This repository's corpus needs none of those: a census of the 17 tracked YAML files found **zero** anchors, aliases, tags, multi-document streams or directives; block scalars in two files (`ci.yml`, 22 of them; `.github/ISSUE_TEMPLATE/bug_report.yml`, 2); 12 `~` nulls; and comment-bearing lines concentrated in `ci.yml` (313), `config/defaults/pipeline.yml` (97) and diff --git a/scripts/kyaml/KYAML.jl b/scripts/kyaml/KYAML.jl index f390901..7668ff5 100644 --- a/scripts/kyaml/KYAML.jl +++ b/scripts/kyaml/KYAML.jl @@ -1234,7 +1234,7 @@ function _kyaml_cli(argv::Vector{String})::Int Options: --report print the decisions taken per file - --skip-file PATH file listing paths to leave alone (default config/kyaml/drift.txt) + --skip-file PATH --check-only exemptions (default config/kyaml/drift.txt) --expect kyaml|yaml style --check compares against (default kyaml) Exit codes: 0 ok, 1 a check failed, 2 a file was refused (nothing was written). @@ -1256,10 +1256,18 @@ function _kyaml_cli(argv::Vector{String})::Int end end isempty(paths) && (paths = KYAML.git_yaml_paths()) - kept = [p for p in paths if !any(s -> startswith(p, s), skip)] + # Drift exemptions are a gate policy, not a migration boundary. Bot-owned + # workflows are still converted (and can be rolled back); only --check omits + # them, so an external formatter cannot make the canonicality gate flap. + skipped = String[] + kept = paths + if mode == "check" + skipped = [p for p in paths if any(s -> startswith(p, s), skip)] + kept = [p for p in paths if !any(s -> startswith(p, s), skip)] + end failures = String[] - reports = Dict{String,Stats}() + reports = Dict{String,KYAML.Stats}() rendered = Dict{String,String}() for p in kept stats = KYAML.Stats() @@ -1285,7 +1293,7 @@ function _kyaml_cli(argv::Vector{String})::Int if mode == "check" if isempty(failures) println("kyaml check: $(length(kept)) file(s) in canonical $expect form" * - (isempty(skip) ? "" : ", $(length(skip)) exempt")) + (isempty(skipped) ? "" : ", $(length(skipped)) exempt")) return 0 end println(stderr, "kyaml check: $(length(failures)) file(s) are not canonical $expect:") @@ -1302,7 +1310,7 @@ function _kyaml_cli(argv::Vector{String})::Int written += 1 end println("kyaml $mode: $(written) of $(length(kept)) file(s) rewritten" * - (isempty(skip) ? "" : ", $(length(skip)) exempt")) + (isempty(skipped) ? "" : ", $(length(skipped)) exempt")) if report for p in sort(collect(keys(reports))) s = reports[p] diff --git a/test/unit/test_kyaml.jl b/test/unit/test_kyaml.jl index eed3129..4396fb3 100644 --- a/test/unit/test_kyaml.jl +++ b/test/unit/test_kyaml.jl @@ -15,20 +15,9 @@ const KYAML_REPO_ROOT = normpath(joinpath(@__DIR__, "..", "..")) include(KYAML_TOOL_PATH) +using YAML using .KYAML: parse_document, render_kyaml, render_yaml, check, git_yaml_paths, KyamlError -function kyaml_exempt_prefixes()::Vector{String} - drift = joinpath(KYAML_REPO_ROOT, "config", "kyaml", "drift.txt") - isfile(drift) || return String[] - prefixes = String[] - for line in eachline(drift) - t = strip(line) - (isempty(t) || startswith(t, "#")) && continue - push!(prefixes, t) - end - return prefixes -end - @testset "KYAML switch" begin @testset "comments keep their association" begin @@ -131,19 +120,61 @@ end end end - @testset "the repository's own YAML is inside the subset" begin - exempt = kyaml_exempt_prefixes() + @testset "drift exemptions affect the gate, not conversion or rollback" begin + mktempdir() do dir + rel = ".github/workflows/ci.yml" + path = joinpath(dir, rel) + mkpath(dirname(path)) + source = "# retained workflow comment\nname: \"CI\"\n" + write(path, source) + drift = joinpath(dir, "drift.txt") + write(drift, rel * "\n") + + cd(dir) do + @test _kyaml_cli(["--to-kyaml", "--skip-file", drift, rel]) == 0 + kyaml = read(path, String) + @test kyaml != source + @test kyaml == render_kyaml(parse_document(rel, source)) + @test _kyaml_cli(["--check", "--skip-file", drift, rel]) == 0 + @test _kyaml_cli(["--to-yaml", "--skip-file", drift, rel]) == 0 + restored = read(path, String) + @test restored != kyaml + @test restored == render_yaml(parse_document(rel, kyaml)) + end + end + end + + @testset "tracked YAML parses; application files preserve YAML.jl semantics" begin + # Drift exemptions govern the canonicality gate only. They are still parsed + # here because conversion and rollback must cover every tracked YAML file. + # Compare with the production YAML.jl reader, not only this tool's own parser: + # a self-round-trip cannot prove that the application or CI sees the same data. failures = String[] for rel in git_yaml_paths(KYAML_REPO_ROOT) - any(prefix -> startswith(rel, prefix), exempt) && continue path = joinpath(KYAML_REPO_ROOT, rel) isfile(path) || continue try - doc = parse_document(path, read(path, String)) - render_kyaml(doc) - render_yaml(doc) + source = read(path, String) + doc = parse_document(path, source) + kyaml = render_kyaml(doc) + yaml = render_yaml(doc) + mktempdir() do dir + original_path = joinpath(dir, "original.yml") + kyaml_path = joinpath(dir, "converted.kyaml") + yaml_path = joinpath(dir, "roundtrip.yml") + write(original_path, source) + write(kyaml_path, kyaml) + write(yaml_path, yaml) + # GitHub Actions is the consumer of workflow files; its actual + # parser/run is tested by CI, not approximated with YAML.jl. + # Compare all application-consumed YAML with the production reader. + if !startswith(rel, ".github/workflows/") + original_value = YAML.load_file(original_path) + @test YAML.load_file(kyaml_path) == original_value + @test YAML.load_file(yaml_path) == original_value + end + end catch err - err isa KyamlError || rethrow() push!(failures, rel * " -> " * sprint(showerror, err)) end end