Ask rustc whether the emitted Rust compiles (#198, step 2)

docs/shape-matrix.md

@@ -44,7 +44,8 @@ not a rewrite.
 | # | Step | State |
 |---|---|---|
 | 1 | The enumerator, both targets, committed report + regen gate | **landed** ([#400](https://github.com/milyin/prebindgen/pull/400)) |
-| 2 | Receipts: `ToolchainCompiled` and `RuntimeExercised` | not started |
+| 2 | Receipts: rustc accepts the emitted Rust | **landed** ([#403](https://github.com/milyin/prebindgen/pull/403)) |
+| 2b | The rest of `ToolchainCompiled`, and `RuntimeExercised` | not started |
 | 3 | The minimum-guarantees table | not started |
 | 4 | Multi-parameter aliasing fixtures | not started |
 | 5 | The adapter-policy axis | JNI half ready; C half blocked |
@@ -75,20 +76,29 @@ until new cells are classified" would be vacuous on its own — a regression tha
 flips a working cell to `rejected` would be recorded as a successful
 classification.
 
-### 2. Receipts — the two states that need a toolchain
+### 2. Receipts — rustc — landed
 
-`plan` means generation succeeded. Nothing compiles, links or runs the result
-yet, so the two strongest states are uncollected.
+Every cell that produced Rust is compiled, and the state is a **receipt**: the
+cell is written to its own file, the crate is checked in one pass, and each
+diagnostic is attributed back by the file rustc names. Nothing maps a cell to a
+fixture by hand — that mapping is what let #175's test pass without creating its
+own precondition.
 
-The rule from #198 stands: these states are **derived from mechanical receipts,
-never declared**. A hand-written *cell → fixture-name* mapping can claim coverage
-for a fixture that never touches the cell — the #175 failure exactly. A fixture
-emits its cell id only *after* the relevant assertion has executed, and a cell
-with no receipt stays `PlanSupported` whatever any table claims.
+Compiler messages stay out of the committed report: they vary by toolchain, and
+the report has to be identical on every one that builds it.
 
-`examples/emitcheck` is the precedent for the compile half: it exists so rustc
-judges emitted Rust. The runtime half rides the JVM covertest and the C smoke
-tests.
+Turning the compiler on immediately found ten cells whose generated Rust does not
+compile, and three defects in this harness — each of which had been reporting a
+confident wrong answer. That is the argument for this step in one sentence:
+`plan` was worth less than it looked.
+
+### 2b. The rest of the toolchain
+
+`cbindgen` does not run, so a C cell that compiles has not been shown to produce
+a valid header; the Kotlin compiler does not run either. `RuntimeExercised` needs
+the JVM covertest and the C smoke tests, and the same receipt rule applies — a
+fixture emits its cell id only *after* the relevant assertion has executed, and a
+cell with no receipt keeps the weaker state whatever any table claims.
 
 ### 3. The minimum-guarantees table
 

examples/shape-matrix/src/check.rs

@@ -0,0 +1,265 @@
+//! Does the emitted Rust actually compile?
+//!
+//! `plan` only says the generator produced a file. An emission can be
+//! well-formed, contain every substring a unit test looks for, and still not
+//! type-check — [`examples/emitcheck`](../../emitcheck) exists because that
+//! happened once with 41 of 41 tests green over it. This stage asks rustc, per
+//! cell.
+//!
+//! # The answer is a receipt, not a claim
+//!
+//! A cell is recorded as compiling only if **rustc says so about that cell's own
+//! file**. Every cell is written to `<id>.rs`, the whole crate is checked in one
+//! pass, and each diagnostic is attributed back by the file path the compiler
+//! reports. Nothing maps a cell to a fixture by name — a name-keyed mapping can
+//! claim coverage for a fixture that never touched the cell, which is the defect
+//! #175 was.
+//!
+//! # What it does not cover
+//!
+//! rustc, and rustc only. Neither `cbindgen` nor the Kotlin compiler runs here,
+//! so a cell that compiles has not been shown to produce a valid C header or a
+//! loadable JVM class. Those are the rest of `ToolchainCompiled`, and they are
+//! not collected yet.
+
+use std::{
+    collections::{BTreeMap, BTreeSet},
+    path::{Path, PathBuf},
+    process::Command,
+};
+
+/// One cell's emitted Rust, ready to be checked.
+pub struct Unit {
+    /// `<shape>__<position>__<target>`, the receipt key.
+    pub id: String,
+    /// The fixture's own items — what the source crate would contain.
+    pub fixture: String,
+    /// What the generator emitted for it.
+    pub emitted: String,
+}
+
+/// Which cells rustc accepted, and what it said about the rest.
+#[derive(Default)]
+pub struct Checked {
+    pub compiled: BTreeSet<String>,
+    /// Cell id → the diagnostics rustc reported for it. **Not** rendered into
+    /// the committed report: a message is a property of the compiler version,
+    /// and the report has to be identical on every toolchain that builds it.
+    pub failed: BTreeMap<String, Vec<String>>,
+}
+
+/// Check every unit in one crate, and attribute the result per cell.
+///
+/// `Err` is reserved for the check itself failing to run — a missing cargo, an
+/// unwritable directory. That is not a verdict about any cell, and callers must
+/// not record it as one.
+/// `workspace` names the directory this batch is checked in. Two batches must
+/// not share one: the report's run and the self-test run concurrently under
+/// `cargo test`, and a shared directory would have them overwriting each
+/// other's sources. A *stable* name per batch rather than a unique one per
+/// call, so the dependencies stay compiled between runs and the emitted Rust
+/// stays on disk to be read after a failure.
+pub fn check(workspace: &str, units: &[Unit]) -> Result<Checked, String> {
+    if units.is_empty() {
+        return Ok(Checked::default());
+    }
+    let root = crate_dir()?.join(workspace);
+    write_crate(&root, units)?;
+
+    let output = Command::new(std::env::var("CARGO").unwrap_or_else(|_| "cargo".into()))
+        .arg("check")
+        .arg("--quiet")
+        .arg("--message-format=json")
+        .arg("--manifest-path")
+        .arg(root.join("Cargo.toml"))
+        // Its own target directory: the parent build may still hold the
+        // workspace one, and a nested cargo blocking on that lock would look
+        // like a hang rather than a queue.
+        .arg("--target-dir")
+        .arg(root.join("target"))
+        .output()
+        .map_err(|e| format!("running cargo check: {e}"))?;
+
+    let mut checked = Checked::default();
+    for unit in units {
+        checked.compiled.insert(unit.id.clone());
+    }
+    for line in String::from_utf8_lossy(&output.stdout).lines() {
+        let Some((file, message)) = diagnostic(line) else {
+            continue;
+        };
+        let Some(id) = cell_of(&file) else { continue };
+        checked.compiled.remove(&id);
+        checked.failed.entry(id).or_default().push(message);
+    }
+
+    // A crate that failed to build with no attributable diagnostic means the
+    // failure was the harness's, not a cell's — a bad `Cargo.toml`, a missing
+    // dependency. Reporting every cell as compiling would be a lie in the
+    // direction that hides defects.
+    if !output.status.success() && checked.failed.is_empty() {
+        return Err(format!(
+            "cargo check failed with nothing attributable to a cell:\n{}",
+            String::from_utf8_lossy(&output.stderr)
+        ));
+    }
+    Ok(checked)
+}
+
+/// The file and message of one rustc error, or `None` for anything else on the
+/// JSON stream.
+///
+/// Deliberately hand-parsed rather than pulled in with `serde_json`: this reads
+/// two fields of a stable format, and the alternative is a dependency in a crate
+/// whose whole point is to have no opinions of its own.
+fn diagnostic(line: &str) -> Option<(String, String)> {
+    if !line.contains("\"level\":\"error\"") {
+        return None;
+    }
+    let rendered = field(line, "\"rendered\":\"")?;
+    let file = rendered
+        .split(&['\\', '"'][..])
+        .find(|part| part.ends_with(".rs") && part.contains(CELL_SEP))
+        .or_else(|| {
+            rendered
+                .split_whitespace()
+                .find(|w| w.contains(".rs") && w.contains(CELL_SEP))
+        })?
+        .to_string();
+    let message = rendered.lines().next().unwrap_or(&rendered).to_string();
+    Some((file, message))
+}
+
+fn field(line: &str, key: &str) -> Option<String> {
+    let start = line.find(key)? + key.len();
+    let rest = &line[start..];
+    let mut out = String::new();
+    let mut chars = rest.chars();
+    while let Some(c) = chars.next() {
+        match c {
+            '"' => break,
+            '\\' => match chars.next() {
+                Some('n') => out.push('\n'),
+                Some('"') => out.push('"'),
+                Some('\\') => out.push('\\'),
+                Some(other) => out.push(other),
+                None => break,
+            },
+            other => out.push(other),
+        }
+    }
+    Some(out)
+}
+
+/// The separator between a cell id's parts, chosen so a file name cannot be
+/// mistaken for anything else on a diagnostic line.
+const CELL_SEP: &str = "__";
+
+fn cell_of(file: &str) -> Option<String> {
+    let stem = Path::new(file.trim_matches(|c: char| !c.is_ascii_graphic()))
+        .file_stem()?
+        .to_str()?;
+    stem.contains(CELL_SEP).then(|| stem.to_string())
+}
+
+/// Where the generated crate lives: under the workspace target directory, so it
+/// is already ignored by git and cleaned by `cargo clean`.
+fn crate_dir() -> Result<PathBuf, String> {
+    let manifest = PathBuf::from(env!("CARGO_MANIFEST_DIR"));
+    let workspace = manifest
+        .parent()
+        .and_then(Path::parent)
+        .ok_or("locating the workspace root")?;
+    let target = std::env::var("CARGO_TARGET_DIR")
+        .map(PathBuf::from)
+        .unwrap_or_else(|_| workspace.join("target"));
+    Ok(target.join("shape-matrix-check"))
+}
+
+fn write_crate(root: &Path, units: &[Unit]) -> Result<(), String> {
+    let src = root.join("src");
+    std::fs::create_dir_all(&src).map_err(|e| format!("creating {}: {e}", src.display()))?;
+
+    let manifest = PathBuf::from(env!("CARGO_MANIFEST_DIR"));
+    let workspace = manifest.parent().and_then(Path::parent).expect("workspace");
+    write(
+        &root.join("Cargo.toml"),
+        &manifest_toml(&workspace.display().to_string()),
+    )?;
+
+    let mut lib = String::from(LIB_HEADER);
+    for (n, unit) in units.iter().enumerate() {
+        let file = format!("{}.rs", unit.id);
+        write(&src.join(&file), &cell_source(unit))?;
+        lib.push_str(&format!("#[path = \"{file}\"]\npub mod cell_{n};\n"));
+    }
+    write(&src.join("lib.rs"), &lib)
+}
+
+fn write(path: &Path, contents: &str) -> Result<(), String> {
+    std::fs::write(path, contents).map_err(|e| format!("writing {}: {e}", path.display()))
+}
+
+/// One cell as a Rust module: the source crate, then the generated file.
+///
+/// This mirrors how a binding crate is actually built — `pub mod myflat;` plus
+/// `include!("generated_bindings.rs")` in `emitcheck`, the same two lines every
+/// consumer writes.
+///
+/// The imports go **inside** the fixture module and nowhere else. A source crate
+/// writing `Cow<'static, str>` has imported `Cow`; the generated file is a
+/// separate scope, and if it needs an import nobody gave it, that is a finding
+/// about the generator rather than something for this harness to paper over.
+fn cell_source(unit: &Unit) -> String {
+    format!(
+        "// Generated by shape-matrix. Do not edit.\n\
+         #![allow(clippy::all, dead_code, unused_imports, unused_variables)]\n\
+         \n\
+         pub mod {} {{\n\
+         use std::borrow::Cow;\n\
+         use std::mem::MaybeUninit;\n\
+         {}\n\
+         }}\n\
+         \n\
+         {}\n",
+        crate::run::SOURCE_CRATE,
+        unit.fixture,
+        unit.emitted
+    )
+}
+
+fn manifest_toml(workspace: &str) -> String {
+    format!(
+        r#"# Generated by shape-matrix. Do not edit.
+[package]
+name = "shape-matrix-check"
+version = "0.0.0"
+edition = "2021"
+publish = false
+
+[lib]
+path = "src/lib.rs"
+
+# Exactly what generated code calls into — the dependencies a real binding
+# crate declares, and nothing else.
+[dependencies]
+prebindgen-jni-runtime = {{ path = "{workspace}/prebindgen-jni-runtime" }}
+prebindgen-c-runtime = {{ path = "{workspace}/prebindgen-c-runtime" }}
+# Pinned exactly, and to what the workspace already resolves: this crate has
+# its own lockfile, so a caret range would let a new upstream release change a
+# cell's answer with nothing in this repo having changed.
+jni = "=0.21.1"
+tracing = "=0.1.44"
+konst = "=0.3.17"
+
+[workspace]
+"#
+    )
+}
+
+const LIB_HEADER: &str = "\
+// Generated by shape-matrix. Do not edit.
+//
+// One module per cell that produced Rust, each in its own file so a diagnostic
+// names the cell it belongs to.
+";

examples/shape-matrix/src/corpus.rs

@@ -24,18 +24,37 @@ pub enum Need {
 
 impl Need {
     /// The Rust the fixture declares for it.
+    ///
+    /// Everything derives `Clone`, as the types in a real source crate do. That
+    /// is not incidental: several generator paths clone — a borrowed handle
+    /// crossing out becomes an owned one — so a fixture whose types were not
+    /// `Clone` would spend its cells measuring that constraint instead of
+    /// whether the shape crosses.
     pub fn source(self) -> &'static str {
         match self {
-            Need::Record => "pub struct Rec { pub id: u64, pub tag: u32 }",
-            Need::Handle => "pub struct Handle { pub id: u64 }",
-            Need::Sum => "pub enum Sum { Num(u64), Nothing }",
-            Need::UnitEnum => "pub enum Mode { On = 0, Off = 1 }",
+            Need::Record => "#[derive(Clone)] pub struct Rec { pub id: u64, pub tag: u32 }",
+            Need::Handle => "#[derive(Clone)] pub struct Handle { pub id: u64 }",
+            // `Display` too: `result_sum_err` puts this in an error position,
+            // where both targets render the error as text.
+            Need::Sum => {
+                "#[derive(Clone)] pub enum Sum { Num(u64), Nothing }\n\
+                          impl std::fmt::Display for Sum {\n\
+                          fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {\n\
+                          write!(f, \"sum\") } }"
+            }
+            Need::UnitEnum => "#[derive(Clone, Copy)] pub enum Mode { On = 0, Off = 1 }",
             // Both targets need a way to render an error, so the accessor is
             // part of the declaration rather than something a cell goes
             // without.
+            // An error type is `Clone` and `Display` because that is what an
+            // error type is; a fixture without them spends its fallible cells
+            // measuring that requirement instead of whether the shape crosses.
             Need::Error => {
-                "pub struct ZError { pub code: u64 }\n\
-                            pub fn zerror_message(e: &ZError) -> String { unimplemented!() }"
+                "#[derive(Clone)] pub struct ZError { pub code: u64 }\n\
+                 impl std::fmt::Display for ZError {\n\
+                 fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {\n\
+                 write!(f, \"error {}\", self.code) } }\n\
+                 pub fn zerror_message(e: &ZError) -> String { unimplemented!() }"
             }
         }
     }
@@ -90,6 +109,17 @@ impl Position {
             Position::Payload => "enum payload",
         }
     }
+
+    /// The part of a cell id this position contributes — file-safe, since a
+    /// cell id names the file rustc reports diagnostics against.
+    pub fn slug(self) -> &'static str {
+        match self {
+            Position::Param => "param",
+            Position::Return => "ret",
+            Position::Field => "field",
+            Position::Payload => "payload",
+        }
+    }
 }
 
 /// The shapes.

examples/shape-matrix/src/lib.rs

@@ -36,8 +36,13 @@
 //!   legitimately answer differently, and one combined verdict would hide
 //!   exactly the gaps this exists to find.
 //!
+//! Every cell that produces Rust is then handed to rustc ([`check`]), because
+//! "the generator produced a file" and "the file compiles" are different claims
+//! and only the second is worth much.
+//!
 //! Run it with `cargo run -p shape-matrix`, which rewrites `REPORT.md`.
 
+pub mod check;
 pub mod corpus;
 pub mod report;
 pub mod run;