ADC-687: add an honest audited-open M4 conformance gate

.github/workflows/ci.yml

@@ -225,12 +225,18 @@ jobs:
               - 'tests/CMakeLists.txt'
               - 'tests/test_manifest.toml'
               - 'scripts/ci_select_tests.py'
+              # The M4 closure is executed inside the installed MPI/VTK lane below. Any edit to
+              # its ledger, runner, source fence, or workflow must therefore require this job on
+              # the PR that changes the proof, never wait for a later nightly run.
+              - 'tests/gates/m4_runtime_io.toml'
+              - 'tests/python/architecture/test_m4_runtime_io_gate.py'
+              - 'scripts/run_m4_gate.py'
+              - '.github/workflows/ci.yml'
               - 'cmake/**'
               - 'CMakeLists.txt'
               - 'CMakePresets.json'
-            # Les changements de workflows/actions CI sont valides par revue + lint YAML local,
-            # et ne compilent pas Kokkos par defaut. Ajouter `ci-kokkos` pour forcer les gates
-            # Serial, ou `ci-full` pour MPI + OpenMP.
+            # Les autres changements de workflows/actions CI sont valides par revue + lint YAML
+            # local. Le workflow CI lui-meme fait exception car il porte la lane M4 executable.
             # full : un push master ne lance la suite COMPLETE (MPI + Kokkos OpenMP) que si un
             # chemin build/backend a bouge. Conservateur a dessein -- couvre tout ce qui peut affecter
             # l'artefact compile OU le chemin DSL/production exerce par le job kokkos-openmp (dsl.py et
@@ -715,6 +721,9 @@ jobs:
       - name: M3 AMR and multi-layout gate manifest
         run: python3 scripts/run_m3_gate.py --check-only
 
+      - name: M4 native runtime and scientific I/O gate manifest
+        run: python3 scripts/run_m4_gate.py --check-only
+
       - name: Generated component catalog
         env:
           PYTHONPATH: ${{ github.workspace }}/python
@@ -1595,7 +1604,7 @@ jobs:
     # The native build, processor-grouped CTest plan, and Python MPI contract
     # fence run sequentially. Each C++ launch retains its configured bounded
     # TIMEOUT; grouping removes PROCESSORS head-of-line blocking without skips.
-    timeout-minutes: 70
+    timeout-minutes: 180
     needs: [set-mode, changes, gate-mpi-prewarm]
     # Suite complete, ou PR qui modifie directement le chemin distribue/MPI.
     if: needs.set-mode.outputs.mpi_required == 'true'
@@ -1632,7 +1641,7 @@ jobs:
           sudo apt-get install -y --no-install-recommends \
             ccache libeigen3-dev libhdf5-openmpi-dev libopenmpi-dev ninja-build openmpi-bin \
             pybind11-dev python3-dev python3-h5py \
-            python3-numpy python3-pytest
+            python3-numpy python3-pytest python3-vtk9
 
       - name: Resolve runner and compiler cache identity
         id: kokkos-platform
@@ -1717,7 +1726,7 @@ jobs:
           test -s build-mpi/mpi-ctest-groups.tsv
 
       - name: Configure + build (MPI + Kokkos Serial)
-        timeout-minutes: 22
+        timeout-minutes: 35
         # Flags : preset ci-mpi (source unique, cf. CMakePresets.json) ; Kokkos_ROOT vient de
         # $KOKKOS_PREFIX (env du job, install en cache). ccache auto-detecte.
         run: |
@@ -1772,6 +1781,10 @@ jobs:
             --build-dir build-mpi \
             --verify-contracts "${compile_contracts[@]}"
           read -r -a mpi_targets <<< "${{ steps.mpi-test-plan.outputs.cpp_label_targets }}"
+          mapfile -t m4_targets < <(
+            /usr/bin/python3 scripts/run_m4_gate.py --list-ctest-targets
+          )
+          test "${#m4_targets[@]}" -gt 0
           export NINJA_STATUS='[%f/%t elapsed=%es active=%r] '
           # The monolithic Python module link is memory-heavy. Keep it isolated
           # from test compilation/linking so a small hosted runner cannot evict
@@ -1780,6 +1793,8 @@ jobs:
             cmake --build --preset ci-mpi --parallel 1 --target _pops
           run_with_heartbeat "MPI native test build" 8m \
             cmake --build --preset ci-mpi --parallel 4 --target "${mpi_targets[@]}"
+          run_with_heartbeat "M4 native test build" 10m \
+            cmake --build --preset ci-mpi --parallel 4 --target "${m4_targets[@]}"
 
       - name: Installed package smoke (MPI-only + collective HDF5)
         run: |
@@ -1935,6 +1950,41 @@ jobs:
             timeout --signal=TERM --kill-after=30s 25m \
               /usr/bin/python3 -m pytest -q -ra --maxfail=1 "$mpi_orchestrator"
           done < build-mpi/python-mpi-orchestrators.txt
+
+      - name: M4 complete native runtime and scientific I/O gate
+        timeout-minutes: 45
+        env:
+          PYTHONPATH: ${{ github.workspace }}/build-mpi/python-package:${{ github.workspace }}
+          POPS_INCLUDE: ${{ github.workspace }}/include
+          POPS_KOKKOS_ROOT: ${{ github.workspace }}/.kokkos-install
+          Kokkos_ROOT: ${{ github.workspace }}/.kokkos-install
+          POPS_CACHE_DIR: ${{ github.workspace }}/.pops-ci/m4-dsl-cache
+          POPS_KEEP_GENERATED: "1"
+          POPS_REQUIRE_MPI_TESTS: "1"
+          POPS_REQUIRE_NATIVE_TESTS: "1"
+          # Ubuntu 24.04 OpenMPI 4 OMPIO selects sharedfp/lockedfile during
+          # HDF5 MPI_File_open and aborts inside its fortified sprintf path.
+          # ROMIO is the packaged OpenMPI MPI-IO component and exercises the
+          # same collective HDF5 contract without that implementation defect.
+          OMPI_MCA_io: "^ompio"
+        run: |
+          # These readers are mandatory capabilities of this lane. Imports happen before the gate
+          # so a missing apt module cannot masquerade as a scientific skip.
+          /usr/bin/python3 - <<'PY'
+          import h5py
+          import numpy
+          from vtkmodules.vtkIOXML import (
+              vtkXMLPUnstructuredGridReader,
+              vtkXMLUnstructuredGridReader,
+          )
+
+          print("M4 readers:", numpy.__version__, h5py.__version__)
+          print(vtkXMLPUnstructuredGridReader, vtkXMLUnstructuredGridReader)
+          PY
+          /usr/bin/python3 scripts/run_m4_gate.py \
+            --build-dir build-mpi \
+            --mpi-exec mpiexec
+
       - name: ccache stats (MPI)
         if: always()
         run: ccache -s

docs/design/exact-output-consumers.md

@@ -53,6 +53,28 @@ ScientificOutput(
 )
 ```
 
+When the native runtime owns an AMR reflux correction and/or an authored projection, the ledger can
+delegate those exact terms instead of requiring zero placeholders. `component` is the exact
+conservative index shared by the explicit Program sums and native evidence (it defaults to zero for
+a scalar state); the optional typed role is checked against that index at bind:
+
+```python
+from pops.physics.roles import Density
+
+mass = BalanceLedger(
+    "mass",
+    role=Density(),
+    component=0,
+    automatic_terms=("projection", "reflux"),
+)
+program.record_balance(
+    mass,
+    storage_change=storage_increment,
+    outward_boundary_flux=boundary_flux_increment,
+    sources=source_increment,
+)
+```
+
 Le fournisseur possède l'extension. Une cible comme `solution/tracer.vtu` est refusée dès
 l'authoring, avant le bind ; elle empêcherait le changement de format et entrerait en collision au
 deuxième échantillon. Chaque pas accepté dû publie immédiatement un fichier distinct sous le chemin
@@ -451,15 +473,20 @@ schedule and transaction. Its reductions are completed on the simulation thread
 the post-commit worker receives only immutable arrays and scalar payloads, never the native mailbox
 or communicator facade.
 
-Each argument to `record_balance` is a signed, time-integrated native Program sum/dot reduction,
-or scalar arithmetic composed only from such reductions and exact literals.
+Each non-automatic argument to `record_balance` is a signed, time-integrated native Program sum/dot
+reduction, or scalar arithmetic composed only from such reductions and exact literals. When any
+term is delegated to a native producer, every explicit term must instead be composed from
+component-qualified `sum` reductions for the ledger's exact `component`; an all-state dot product
+cannot be reconciled with one component's reflux/projection evidence.
 The reported residual is `storage_change + outward_boundary_flux - sources - reflux - projection`.
 The native attempt mailbox accumulates repeated cadence/substep invocations, rejects missing or
 non-finite terms, and is cleared before the next attempt. The consumer reads it only while the
 outer accepted-step transaction still retains the pre-step image. Python therefore packages the
 five returned scalars and residual but never traverses arrays, invents a zero term, or reuses a
-previous step. A rejected attempt or failed consumer publication restores the mailbox with the
-rest of the native transaction.
+previous step. Selected automatic terms are resolved by exact runtime block, active hierarchy level
+and conservative component. A missing coordinate, a non-finite value, or simultaneous Program and
+native authority for one term fails the accepted transaction. A rejected attempt or failed consumer
+publication restores both mailboxes with the rest of the native transaction.
 
 The `pops.balance-term` namespace is reserved. Ordinary `Program.record_scalar(...)` authoring and
 the Python runtime diagnostic binding both reject it; generated `record_balance` code reaches a
@@ -476,8 +503,9 @@ by an OR of their exact accepted-step periods. `Always` and `when(True)` are per
 The compiler traces the complete reduction/scalar chain rather than scheduling only the terminal
 records. If a value is also consumed by an ordinary Program diagnostic or another non-balance
 operation, that shared producer remains unconditional so cadence fusion cannot change unrelated
-semantics. A `Balance` consumer with no matching five-term `Program.record_balance` producer fails
-before native code generation. Program stride/substeps use one attempt-local outer accepted-step
+semantics. A `Balance` consumer with no complete matching `Program.record_balance` producer for all
+non-automatic terms fails before native code generation. Program stride/substeps use one
+attempt-local outer accepted-step
 target, so every substep of one due public step sees the same decision and accumulates into the same
 attempt mailbox. The cadence is authored once as part of the Program identity, for example
 `program.cadence(substeps=2, stride=3)`, then authenticated and installed before runtime freeze on
@@ -498,12 +526,22 @@ balance reductions are not yet skipped. This fallback can add work but cannot su
 evidence. A zero-step run has no accepted native occurrence: its coincident start/end moment cannot
 publish an accepted-step consumer, including `Balance`.
 
-This route is explicit evidence, not automatic numerical instrumentation: a Program that cannot
-produce its actual reflux or projection increment cannot declare `Balance`. In particular, the
-generic automatic extraction of AMR reflux/projection contributions from the internal native
-operator ledgers remains separate work. On an adaptive layout the recorded values must already be
-composite and coverage-corrected; an ordinary sum of every per-level state would double-count
-covered coarse cells. Neither `Balance` nor `BalanceTerms` silently claims otherwise.
+The selected public route now consumes signed AMR reflux corrections and before/after projection
+deltas from the separate qualified attempt mailbox. Uniform Cartesian projection uses the
+authenticated cell measure and embedded-boundary mask; AMR projection excludes covered coarse cells
+and performs one component-vector collective per participating level. A reflux selection requires
+an adaptive hierarchy and expects one contribution for every active parent/fine interface;
+projection expects one for every selected active level. Generated code publishes the OR of the exact
+due route decisions before the first Program operator; the marker is monotone for the attempt,
+disabled during replay, and reset at attempt entry. Consequently off-cadence steps do not pay for
+automatic operator reductions.
+
+The capability remains deliberately bounded. Polar projection is rejected because no exact
+per-cell polar volume provider exists on this path. Automatic physical-boundary flux and source
+evidence are not yet producers and therefore remain explicit `Program.record_balance` arguments.
+The native selector never substitutes a missing automatic value with zero (except the exact reflux
+identity for a hierarchy with no coarse/fine interface), and the legacy all-explicit ledger route
+retains its original identity and behavior.
 
 Checkpoint remains a separate restart effect. These consumers do not define a checkpoint schema or
 reader and do not call the scientific-output manifest a restart identity. The checkpoint provider