Context (real unguarded security path — exists today). An operator can point analysis.mode=llm-only at any fine-tuned checkpoint via LocalInferenceProviderSpec and route every policy decision through it, yet the only safety net is health_check (a deliberate no-op for local, local_provider.py:59-71) and the determinism harness — which checks self-consistency, not correctness. Because estimated_sensitivity feeds the deny/scope decision, a checkpoint that under-estimates sensitivity downgrades a security control with no operator signal.
Proposal. Document the gap, then add a correctness harness alongside the determinism one.
- The locked fixture
tests/fixtures/llm_determinism/intent_prompts_100.jsonl already carries ground-truth labels (expected_data_types/estimated_sensitivity). Replay it through the candidate provider's real client (not the fake) and score agreement on the security-critical fields (data_types_needed, estimated_sensitivity).
- Asymmetric penalty for sensitivity UNDER-estimation (mirroring
intent_v1.txt rule 6's minimal-interpretation bias).
- Surface a pass/fail threshold operators must clear before flipping to
llm-only.
Where.
local_provider.py:59-71 (health_check no-op) + :38-54 (verbatim-model acceptance); tests/integration/test_llm_determinism_harness.py (determinism-only); model on rkm/validator/scoring.py:score (which has no correctness analog for providers).
Acceptance criteria.
Caveat. Too-strict blocks good checkpoints; too-loose gives false assurance. Anchor on the deterministic fixture labels (the graph is the oracle), not frontier-generated labels, to avoid the closed-loop-judge trap.
Source: arXiv:2605.22502v1 — "Compiling Agentic Workflows into LLM Weights" (subterranean agents). Distilled from arXiv-research/2605.22502v1/analysis.md; file refs verified against current main by the analysis pass.
Context (real unguarded security path — exists today). An operator can point
analysis.mode=llm-onlyat any fine-tuned checkpoint viaLocalInferenceProviderSpecand route every policy decision through it, yet the only safety net ishealth_check(a deliberate no-op for local,local_provider.py:59-71) and the determinism harness — which checks self-consistency, not correctness. Becauseestimated_sensitivityfeeds the deny/scope decision, a checkpoint that under-estimates sensitivity downgrades a security control with no operator signal.Proposal. Document the gap, then add a correctness harness alongside the determinism one.
tests/fixtures/llm_determinism/intent_prompts_100.jsonlalready carries ground-truth labels (expected_data_types/estimated_sensitivity). Replay it through the candidate provider's real client (not the fake) and score agreement on the security-critical fields (data_types_needed,estimated_sensitivity).intent_v1.txtrule 6's minimal-interpretation bias).llm-only.Where.
local_provider.py:59-71(health_check no-op) + :38-54 (verbatim-model acceptance);tests/integration/test_llm_determinism_harness.py(determinism-only); model onrkm/validator/scoring.py:score(which has no correctness analog for providers).Acceptance criteria.
llm-onlypromotion.Caveat. Too-strict blocks good checkpoints; too-loose gives false assurance. Anchor on the deterministic fixture labels (the graph is the oracle), not frontier-generated labels, to avoid the closed-loop-judge trap.
Source: arXiv:2605.22502v1 — "Compiling Agentic Workflows into LLM Weights" (subterranean agents). Distilled from
arXiv-research/2605.22502v1/analysis.md; file refs verified against currentmainby the analysis pass.