feat(evidence): seal model identity into the bundle — #4-wedge (PR-2)#3
Closed
gesh75 wants to merge 2 commits into
Closed
feat(evidence): seal model identity into the bundle — #4-wedge (PR-2)#3gesh75 wants to merge 2 commits into
gesh75 wants to merge 2 commits into
Conversation
Records WHICH model produced a change and seals it under the bundle integrity hash — the #4 model-identity wedge, the input the CROSS-3 seal consumes. Built on the REAL evidence layer (build_bundle / run_preflight / integrity.sha256). - evidence/bundler.py: build_bundle gains model_identity=; bundle_version 1.0->1.1; change.model_identity sealed into integrity.sha256 (tamper-evident); honest _UNKNOWN_IDENTITY (provider "none") for config_import / no-model runs. Synthetic identities use a fixed resolved_at_utc so stress INV-3 determinism holds. - evidence/schema/evidence_bundle.schema.json: bundle_version const 1.0->1.1; change requires model_identity (provider enum local|cloud|simulator|none, model_hash hex64|null, identity-claim never faked); compliance framework adds eu_ai_act. - core/orchestrator/pipeline.py: run_preflight(model_identity=) passthrough; nl_intent attests via backend.model_identity(); config_import -> _UNKNOWN. - core/backends/simulator.py: deterministic model_identity() (provider "simulator"). - core/llm/identity.py + __init__: IdentitySink — the egress after_response seam. - tests/wedge_test.py: 9 tests. Checks green: wedge 9/9; stress PASS (3000 runs — schema INV-1, hash INV-2, determinism INV-3 hold); promote/twin/contract PASS; PR-1 egress 29/29. Stacked on #2 (PR-1).
Reviewer's GuideImplements bundle version 1.1 that seals the generating model’s identity into the evidence bundle (covered by the integrity hash), threads model_identity through the pipeline, adds an IdentitySink egress hook to capture it, introduces a deterministic simulator identity and UNKNOWN identity for non-model changes, updates the evidence schema to require model_identity and pin bundle_version, and adds tests covering sealing, schema behavior, and the egress→sink→bundle round-trip. Sequence diagram for sealing model identity into the evidence bundlesequenceDiagram
actor Operator
participant Egress as LLM_Egress
participant IdentitySink
participant Pipeline as run_preflight
participant Backend as Backend
participant Bundler as build_bundle
Operator->>Egress: generate(intent)
Egress-->>IdentitySink: capture(provider, identity)
IdentitySink-->>IdentitySink: last = identity
Operator->>Pipeline: run_preflight(intent, backend, model_identity=IdentitySink.as_bundle_dict())
alt model_identity not provided
Pipeline->>Backend: model_identity()
Backend-->>Pipeline: model_identity dict
end
Pipeline->>Bundler: build_bundle(..., model_identity)
Bundler-->>Bundler: _unknown_identity() if model_identity is None
Bundler-->>Operator: bundle (bundle_version 1.1 with change.model_identity)
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've found 3 issues, and left some high level feedback:
- The fixed synthetic timestamp string for synthetic identities (
1970-01-01T00:00:00+00:00) is duplicated inevidence.bundler._SYNTHETIC_RESOLVED_ATandSimulatorBackend.model_identity; consider centralizing this in a single constant to avoid drift. - In
run_preflight, thehasattr(backend, "model_identity")check will happily accept non-callable attributes; usingcallable(getattr(backend, "model_identity", None))would make the intent clearer and failure modes safer if backends evolve.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- The fixed synthetic timestamp string for synthetic identities (`1970-01-01T00:00:00+00:00`) is duplicated in `evidence.bundler._SYNTHETIC_RESOLVED_AT` and `SimulatorBackend.model_identity`; consider centralizing this in a single constant to avoid drift.
- In `run_preflight`, the `hasattr(backend, "model_identity")` check will happily accept non-callable attributes; using `callable(getattr(backend, "model_identity", None))` would make the intent clearer and failure modes safer if backends evolve.
## Individual Comments
### Comment 1
<location path="evidence/bundler.py" line_range="108" />
<code_context>
+
+# Fixed timestamp for synthetic identities (UNKNOWN / simulator) so stress determinism
+# (INV-3: same intent -> identical bundle) holds. Real backends carry their own resolved_at.
+_SYNTHETIC_RESOLVED_AT = "1970-01-01T00:00:00+00:00"
+
+
</code_context>
<issue_to_address>
**suggestion:** Avoid duplicating the synthetic resolved_at timestamp literal across modules.
This same fixed timestamp is defined here as `_SYNTHETIC_RESOLVED_AT` and also hard-coded in `core/backends/simulator.py::model_identity`. To avoid drift if it changes, centralize the constant in a shared module or have the simulator backend reference this constant so both synthetic UNKNOWN and simulator identities remain aligned for determinism.
Suggested implementation:
```python
# Fixed timestamp for synthetic identities (UNKNOWN / simulator) so stress determinism
# (INV-3: same intent -> identical bundle) holds. Real backends carry their own resolved_at.
SYNTHETIC_RESOLVED_AT = "1970-01-01T00:00:00+00:00"
```
To fully centralize this constant and avoid duplication:
1. Update `core/backends/simulator.py::model_identity` to import and use the shared constant instead of the hard-coded literal, e.g.:
- Add: `from evidence.bundler import SYNTHETIC_RESOLVED_AT`
- Replace the `"1970-01-01T00:00:00+00:00"` literal used as the synthetic `resolved_at` with `SYNTHETIC_RESOLVED_AT`.
2. If your layering rules discourage simulator importing from `evidence.bundler`, instead move `SYNTHETIC_RESOLVED_AT` to a small shared module (e.g. `evidence/constants.py`), import it in both `bundler.py` and `simulator.py`, and remove the constant definition from `bundler.py` accordingly.
</issue_to_address>
### Comment 2
<location path="tests/wedge_test.py" line_range="64-70" />
<code_context>
+
+# ── real pipeline bundles now carry + seal model_identity at v1.1 ────────────────
+
+def test_nl_intent_seals_simulator_identity_v11() -> None:
+ b = _bundle()
+ assert b["bundle_version"] == "1.1"
+ mi = b["change"]["model_identity"]
+ assert mi["provider"] == "simulator" and mi["model"] == "deterministic-sim-v1"
+ assert verify(b)
+ _validate(b)
+
+
</code_context>
<issue_to_address>
**suggestion (testing):** Assert deterministic identity fields (e.g. resolved_at_utc) to lock in INV-3 determinism for simulator/UNKNOWN identities.
This only asserts provider/model, but INV-3 determinism also relies on fixed timestamp and capabilities. Please also assert that `mi['resolved_at_utc']` is the simulator’s synthetic constant and that `capabilities` matches the expected value. Likewise, in `test_config_import_seals_unknown_identity`, assert the fixed `resolved_at_utc` and capabilities for `_UNKNOWN_IDENTITY` so changes to those fields can’t silently weaken the determinism contract.
Suggested implementation:
```python
def test_nl_intent_seals_simulator_identity_v11() -> None:
b = _bundle()
assert b["bundle_version"] == "1.1"
mi = b["change"]["model_identity"]
# INV-3 determinism: simulator identity must be fully deterministic, including
# fixed timestamp + capabilities, not just provider/model.
assert mi["provider"] == "simulator" and mi["model"] == "deterministic-sim-v1"
assert mi["resolved_at_utc"] == "2026-06-01T10:30:00+00:00"
assert mi["capabilities"] == ["chat", "json"]
assert verify(b)
_validate(b)
```
I could not see the body of `test_config_import_seals_unknown_identity` in the provided context. To fully implement your suggestion and lock in INV-3 determinism for `_UNKNOWN_IDENTITY`, you should:
1. In `test_config_import_seals_unknown_identity`, after you obtain `mi = b["change"]["model_identity"]` (or equivalent), add assertions analogous to the simulator test:
- `assert mi["resolved_at_utc"] == "2026-06-01T10:30:00+00:00"`
- `assert mi["capabilities"] == ["chat", "json"]`
2. If `_UNKNOWN_IDENTITY` is defined as a constant/dict in the production code, consider asserting against that constant instead of literals, e.g.:
- `assert mi["resolved_at_utc"] == _UNKNOWN_IDENTITY["resolved_at_utc"]`
- `assert mi["capabilities"] == _UNKNOWN_IDENTITY["capabilities"]`
Adjust the exact expected values if the canonical simulator / `_UNKNOWN_IDENTITY` timestamps or capabilities differ in your codebase; the key is to assert both fields so that changes to them cannot silently weaken determinism.
</issue_to_address>
### Comment 3
<location path="tests/wedge_test.py" line_range="73-70" />
<code_context>
+ _validate(b)
+
+
+def test_config_import_seals_unknown_identity() -> None:
+ b = run_preflight("", backend=SimulatorBackend(), lab="single", source="config_import",
+ imported_configs=[{"device": "r1", "vendor": "frr", "config": "router bgp 65000"}])
+ mi = b["change"]["model_identity"]
+ assert mi["provider"] == "none" and mi["model"] == "operator-supplied"
+ assert mi["model_hash"] is None and mi["model_hash_kind"] == "identity-claim"
+ assert verify(b)
+ _validate(b)
+
+
</code_context>
<issue_to_address>
**suggestion (testing):** Add a test for backends without model_identity to verify fallback to the honest UNKNOWN identity path.
We currently test (a) simulator auto-attestation, (b) `config_import` → UNKNOWN, and (c) explicit `model_identity` passthrough, but not the case where `run_preflight` is called with a backend that lacks `model_identity()` and `model_identity=None`. That should also take the `_UNKNOWN_IDENTITY` path for non-`config_import` sources. Please add a small test using a minimal backend without `model_identity` to cover this branch and confirm the bundle stays schema-valid and verifiable in this edge case.
Suggested implementation:
```python
def test_nl_intent_seals_simulator_identity_v11() -> None:
b = _bundle()
assert b["bundle_version"] == "1.1"
mi = b["change"]["model_identity"]
assert mi["provider"] == "simulator" and mi["model"] == "deterministic-sim-v1"
assert verify(b)
_validate(b)
class BackendWithoutModelIdentity:
"""Minimal proxy backend that hides model_identity() to trigger UNKNOWN fallback."""
def __init__(self, delegate: SimulatorBackend) -> None:
self._delegate = delegate
def __getattr__(self, name: str):
# Pretend this backend has no model_identity; everything else is delegated.
if name == "model_identity":
raise AttributeError
return getattr(self._delegate, name)
def test_backend_without_model_identity_falls_back_to_unknown_identity() -> None:
backend = BackendWithoutModelIdentity(SimulatorBackend())
b = run_preflight("", backend=backend, lab="single", source="nl_intent")
mi = b["change"]["model_identity"]
assert mi == _UNKNOWN_IDENTITY
assert verify(b)
_validate(b)
House style: pytest-discoverable test_* + a dep-light __main__ runner
```
1. Ensure `_UNKNOWN_IDENTITY` is imported into this test module from the same place `run_preflight`/`SimulatorBackend` come from (e.g. `from aegis.wedge import _UNKNOWN_IDENTITY`), or adjust the import path to match your existing code.
2. If `SimulatorBackend` is not already imported in `tests/wedge_test.py`, add the appropriate import for it and for `run_preflight` at the top of the file, consistent with how the other tests reference them.
3. If your type-checking configuration is strict and `SimulatorBackend` is not the exact type used for `backend` in `run_preflight`, you may want to widen the type annotation on `BackendWithoutModelIdentity.__init__` (e.g. to a protocol or base class) to match your existing backend interface.
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
|
|
||
| # Fixed timestamp for synthetic identities (UNKNOWN / simulator) so stress determinism | ||
| # (INV-3: same intent -> identical bundle) holds. Real backends carry their own resolved_at. | ||
| _SYNTHETIC_RESOLVED_AT = "1970-01-01T00:00:00+00:00" |
There was a problem hiding this comment.
suggestion: Avoid duplicating the synthetic resolved_at timestamp literal across modules.
This same fixed timestamp is defined here as _SYNTHETIC_RESOLVED_AT and also hard-coded in core/backends/simulator.py::model_identity. To avoid drift if it changes, centralize the constant in a shared module or have the simulator backend reference this constant so both synthetic UNKNOWN and simulator identities remain aligned for determinism.
Suggested implementation:
# Fixed timestamp for synthetic identities (UNKNOWN / simulator) so stress determinism
# (INV-3: same intent -> identical bundle) holds. Real backends carry their own resolved_at.
SYNTHETIC_RESOLVED_AT = "1970-01-01T00:00:00+00:00"To fully centralize this constant and avoid duplication:
- Update
core/backends/simulator.py::model_identityto import and use the shared constant instead of the hard-coded literal, e.g.:- Add:
from evidence.bundler import SYNTHETIC_RESOLVED_AT - Replace the
"1970-01-01T00:00:00+00:00"literal used as the syntheticresolved_atwithSYNTHETIC_RESOLVED_AT.
- Add:
- If your layering rules discourage simulator importing from
evidence.bundler, instead moveSYNTHETIC_RESOLVED_ATto a small shared module (e.g.evidence/constants.py), import it in bothbundler.pyandsimulator.py, and remove the constant definition frombundler.pyaccordingly.
Comment on lines
+64
to
+70
| def test_nl_intent_seals_simulator_identity_v11() -> None: | ||
| b = _bundle() | ||
| assert b["bundle_version"] == "1.1" | ||
| mi = b["change"]["model_identity"] | ||
| assert mi["provider"] == "simulator" and mi["model"] == "deterministic-sim-v1" | ||
| assert verify(b) | ||
| _validate(b) |
There was a problem hiding this comment.
suggestion (testing): Assert deterministic identity fields (e.g. resolved_at_utc) to lock in INV-3 determinism for simulator/UNKNOWN identities.
This only asserts provider/model, but INV-3 determinism also relies on fixed timestamp and capabilities. Please also assert that mi['resolved_at_utc'] is the simulator’s synthetic constant and that capabilities matches the expected value. Likewise, in test_config_import_seals_unknown_identity, assert the fixed resolved_at_utc and capabilities for _UNKNOWN_IDENTITY so changes to those fields can’t silently weaken the determinism contract.
Suggested implementation:
def test_nl_intent_seals_simulator_identity_v11() -> None:
b = _bundle()
assert b["bundle_version"] == "1.1"
mi = b["change"]["model_identity"]
# INV-3 determinism: simulator identity must be fully deterministic, including
# fixed timestamp + capabilities, not just provider/model.
assert mi["provider"] == "simulator" and mi["model"] == "deterministic-sim-v1"
assert mi["resolved_at_utc"] == "2026-06-01T10:30:00+00:00"
assert mi["capabilities"] == ["chat", "json"]
assert verify(b)
_validate(b)I could not see the body of test_config_import_seals_unknown_identity in the provided context. To fully implement your suggestion and lock in INV-3 determinism for _UNKNOWN_IDENTITY, you should:
- In
test_config_import_seals_unknown_identity, after you obtainmi = b["change"]["model_identity"](or equivalent), add assertions analogous to the simulator test:assert mi["resolved_at_utc"] == "2026-06-01T10:30:00+00:00"assert mi["capabilities"] == ["chat", "json"]
- If
_UNKNOWN_IDENTITYis defined as a constant/dict in the production code, consider asserting against that constant instead of literals, e.g.:assert mi["resolved_at_utc"] == _UNKNOWN_IDENTITY["resolved_at_utc"]assert mi["capabilities"] == _UNKNOWN_IDENTITY["capabilities"]
Adjust the exact expected values if the canonical simulator / _UNKNOWN_IDENTITY timestamps or capabilities differ in your codebase; the key is to assert both fields so that changes to them cannot silently weaken determinism.
| mi = b["change"]["model_identity"] | ||
| assert mi["provider"] == "simulator" and mi["model"] == "deterministic-sim-v1" | ||
| assert verify(b) | ||
| _validate(b) |
There was a problem hiding this comment.
suggestion (testing): Add a test for backends without model_identity to verify fallback to the honest UNKNOWN identity path.
We currently test (a) simulator auto-attestation, (b) config_import → UNKNOWN, and (c) explicit model_identity passthrough, but not the case where run_preflight is called with a backend that lacks model_identity() and model_identity=None. That should also take the _UNKNOWN_IDENTITY path for non-config_import sources. Please add a small test using a minimal backend without model_identity to cover this branch and confirm the bundle stays schema-valid and verifiable in this edge case.
Suggested implementation:
def test_nl_intent_seals_simulator_identity_v11() -> None:
b = _bundle()
assert b["bundle_version"] == "1.1"
mi = b["change"]["model_identity"]
assert mi["provider"] == "simulator" and mi["model"] == "deterministic-sim-v1"
assert verify(b)
_validate(b)
class BackendWithoutModelIdentity:
"""Minimal proxy backend that hides model_identity() to trigger UNKNOWN fallback."""
def __init__(self, delegate: SimulatorBackend) -> None:
self._delegate = delegate
def __getattr__(self, name: str):
# Pretend this backend has no model_identity; everything else is delegated.
if name == "model_identity":
raise AttributeError
return getattr(self._delegate, name)
def test_backend_without_model_identity_falls_back_to_unknown_identity() -> None:
backend = BackendWithoutModelIdentity(SimulatorBackend())
b = run_preflight("", backend=backend, lab="single", source="nl_intent")
mi = b["change"]["model_identity"]
assert mi == _UNKNOWN_IDENTITY
assert verify(b)
_validate(b)
House style: pytest-discoverable test_* + a dep-light __main__ runner- Ensure
_UNKNOWN_IDENTITYis imported into this test module from the same placerun_preflight/SimulatorBackendcome from (e.g.from aegis.wedge import _UNKNOWN_IDENTITY), or adjust the import path to match your existing code. - If
SimulatorBackendis not already imported intests/wedge_test.py, add the appropriate import for it and forrun_preflightat the top of the file, consistent with how the other tests reference them. - If your type-checking configuration is strict and
SimulatorBackendis not the exact type used forbackendinrun_preflight, you may want to widen the type annotation onBackendWithoutModelIdentity.__init__(e.g. to a protocol or base class) to match your existing backend interface.
…pplied Security review (MEDIUM, fail-open provenance): for nl_intent the hasattr-guarded fallback let a missing attestation collapse to _UNKNOWN_IDENTITY (provider "none", model "operator-supplied") — indistinguishable from a legitimate config_import, so an AI-generated change could read as operator-authored, defeating the wedge. Fix: nl_intent now seals an explicit UNATTESTED marker (provider "unknown", model "unattested") when the backend cannot attest — NEVER the operator-supplied identity. config_import still seals _UNKNOWN_IDENTITY (truly no model). Non-breaking: keeps live HttpBackend (pre-cutover) working with an auditable gap instead of a hard fail. - evidence/bundler.py: unattested_identity() (provider "unknown" / model "unattested") - core/orchestrator/pipeline.py: explicit 3-way resolution (passthrough | config_import -> UNKNOWN | nl_intent -> attest-or-UNATTESTED) - evidence/schema: provider enum adds "unknown" - tests/stress_test.py: CountingBackend forwards inner.model_identity() (attests sim) - tests/wedge_test.py: regression — nl_intent w/o attestation is unattested != operator Green: wedge 10/10; stress PASS (3000); promote/twin/contract PASS; PR-1 egress 29/29.
gesh75
added a commit
that referenced
this pull request
Jun 1, 2026
…authority gate, ceiling, CROSS-3 seal (#9) Squashed release of the 8-PR stack (#2..#8): the offline, hardware-rootable bounded-autonomy receipt and everything under it. AUTO disabled by default; the only un-built piece is the hardware-PIV signer (docs/PIV_HARDWARE_SIGNER_PLAN.md) — software Ed25519 runs the whole demo. - core/llm: single shared LLM egress + adapter (local-first fallback chain, retry/cooldown, 4-event hook bus; raw httpx, no anthropic SDK; air-gap fail-closed). [#2] - evidence: model-identity wedge — change.model_identity sealed into the bundle hash; honest UNKNOWN / unattested states, never a faked attribution. [#3] - core/risk: deterministic AUTO/HITL/HOTL/BLOCK authority model (severity != authority; AS/RD/RT -> BLOCK; spine/underlay -> >= HOTL) + the no-self-escalation ceiling. [#4] - core/promote: gate rule G5 enforces the ceiling from the sealed required tier; change.authority sealed (bundle_version 1.1->1.2). [#5] - core/seal: CROSS-3 detached, offline-verifiable bounded-autonomy receipt (Ed25519; gates G0..G6; refuses to seal an unbounded change). [#6] - serve: /api/preflight/run emits the seal; GET /api/seal/pubkey + POST /api/seal/verify; examiner PDF shows the seal. [#7] - docs: PIV hardware-signer plan — the final, hardware-gated CROSS-3 pass. [#8] Property-tested safety invariants: no-self-escalation 0-violations, every-seal-is-bounded, tamper detection, offline-keyless verify, air-gap fail-closed.
Owner
Author
|
Superseded by the squashed release #9, now merged to main. This PR's changes are in main; closing for tidiness (branch kept for history). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR-2 — the model-identity wedge (stacked on #2 / PR-1)
Seals which model produced a change into the evidence bundle's integrity hash — the input the CROSS-3 SEAL consumes. Built on the real evidence layer (
build_bundle/run_preflight/integrity.sha256).Changes (7 files, +301/-6)
evidence/bundler.py—build_bundle(model_identity=…);bundle_version1.0→1.1;change.model_identityis covered byintegrity.sha256, so swapping the attested model after the fact is detected (re-hash mismatch). Honest_UNKNOWN_IDENTITY(provider:"none") forconfig_import/no-model runs — never a faked attribution. Synthetic identities use a fixedresolved_at_utcso stress INV-3 determinism still holds.evidence/schema/…json—bundle_versionconst 1.1;changerequiresmodel_identity(provider enum local|cloud|simulator|none,model_hashhex64|null,identity-claimnever faked); compliance framework addseu_ai_act.core/orchestrator/pipeline.py—run_preflight(model_identity=…)passthrough;nl_intentattests viabackend.model_identity(),config_import→_UNKNOWN.core/backends/simulator.py— deterministicmodel_identity()(provider:"simulator").core/llm/identity.py+__init__—IdentitySink: the egressafter_responseseam that captures the generating model for the pipeline to seal.Tests —
tests/wedge_test.py, 9/9 (against the real pipeline + schema)sim identity at v1.1 ·
config_import→UNKNOWN · explicit passthrough · model under the seal / tamper-detected · cloudidentity-claimnever a faked hash · v1.1 requires identity · schema pinsbundle_versionto 1.1 · egress→IdentitySink→sealed-bundle round-trip.Regression: clean.
stressPASS (3000 runs — schema INV-1, hash INV-2, determinism INV-3 all hold);promote/twin/contract(14)PASS; PR-1 egress 29/29.Next: #5-core/ceiling (differential-first, PIV crypto) → CROSS-3 SEAL (fuses this identity + the signed authority ceiling).
Summary by Sourcery
Seal the generating model’s identity into evidence bundles and propagate model identity from LLM egress through the pipeline into the integrity-protected evidence schema.
New Features:
Enhancements:
Tests: