Skip to content

Commit a532db3

Browse files
Improve agent-memory evidence and benchmark integrity (#225)
* Expand benchmark evidence and add guarded Codex OAuth campaigns * Fix campaign ledger IDs and validate public result envelopes * Preserve interrupted coding runs and repair benchmark fixture validity * Record OAuth pilot outcomes and launch frozen local follow-up measurements * Finalize benchmark expansion and receipt integrity fixes * fix: address PR review findings * fix: mark campaign digests as non-security hashes * fix: exclude secret fields from campaign bindings * chore: document integrity digest CodeQL boundary * fix: place CodeQL suppression at digest sink * fix: allowlist non-security campaign integrity digest * fix: suppress audited campaign digest alert * fix: keep campaign digest compatible with CodeQL * fix: enable exact CodeQL alert suppressions * chore: refresh benchmark evidence and CodeQL suppression * fix: preserve CodeQL analysis configuration identity * fix: stabilize CodeQL analysis category * fix: match CodeQL baseline categories * fix: keep approved CodeQL digests out of uploaded SARIF * fix: preserve Python 3.9 write compatibility * test: refresh offline evidence after portability fix * fix: close remaining benchmark review gaps * fix: close remaining continuation review gaps * fix: close remaining review findings * chore: refresh public evidence after review fixes * fix: close remaining review and floor-gate gaps * fix: close remaining peer benchmark review gaps * fix: validate evidence actions and complete benchmark review fixes * chore: preserve existing line endings in evidence references * fix: reject incomplete campaigns and mismatched benchmark corpora * fix: preserve ownership of external container state directories * test: produce fresh source-bound queue analysis fixtures * fix: bind startup ownership marker to the managed data volume * fix: initialize private state for rootless container startup * fix: score invalid JSON candidate results deterministically * fix(container): own newly created private directory ancestors * fix(benchmarks): bind final artifacts and repair invalid exact metadata * fix(container): repair restored descendants despite ownership marker * fix(container): reject hard-linked privileged startup inputs * fix(benchmarks): retain bound literals and validate resumed evidence * fix(benchmarks): bind published results to evaluated snapshots * fix(context): preserve restrictions around multiline exact values * fix(benchmarks): preserve session and audit evidence across resumes * fix(evidence): bind source revisions and preserve benchmark populations * fix(memory): preserve relevant evidence through packing and correction * fix(benchmarks): bind evaluation to verified input snapshots * fix(memory): withhold incomplete exact evidence groups * fix(benchmarks): validate labels and bind adapter revision * fix(evidence): validate writes and campaign completion * fix(provenance): preserve campaign trust and effective recall depth * fix: preserve correction intent and peer evidence boundaries * Retain observed benchmark usage through failed attempts * Preserve campaign interruptions and synchronize integrity gate location * fix: release external checkpoint ownership after process death * fix: keep compact exact bindings with admitted evidence * Preserve complete restriction units and conservative budget ceilings * fix: bind benchmark digests to unique public source identities * fix: narrow captured repair digest before artifact verification * Preserve recall packing provenance and legacy binding safety * Require complete source for exact recall bindings * Recover local benchmark locks without replaying interrupted work * Bind producer lock probes to the inspected file identity * Validate benchmark eligibility and repair provenance * Validate benchmark rates, cell identities, and campaign locks * Validate public campaign metadata and observed usage * Preserve scope and atomic correction history in campaign replay * Prevent candidate output from forging coding oracle results
1 parent ca79026 commit a532db3

870 files changed

Lines changed: 134242 additions & 1312 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.claude-plugin/skill-assets.sha256‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,4 +3,4 @@ e2e08499a70d8d62ef22797cda0bb07d46cd1c6a79e5e18c1103b6c3c755b6d5 .claude-plugin
33
4bc8979b9ffeb97190960e551dbf4ddc6f7aeeb7b86894fd2298a59ff0001efa skills/engraphis-memory/SKILL.md
44
055655db84af07561d002f0c69744313d8413c39f3e873f941f0fa0b1e76dc66 skills/engraphis-memory/references/CONVENTIONS.md
55
62019760766ff472a76a0f81437898f39e3c1fe2631732b7b7733e50c1ad837f skills/engraphis-memory/references/SCOPING.md
6-
33874c7c7a1c0911b0e73c7d22addc9828963d5436cb315fe7c6c5587c6b911d skills/engraphis-memory/references/TOOLS.md
6+
9e5f1c8e91ca5697e828ab9e468504b8e1aa28e34f61307c9fcd850969126be9 skills/engraphis-memory/references/TOOLS.md

‎.github/codeql/codeql-config.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,6 @@
77
#
88
# Changing to SHA-256 would invalidate all existing local vectors and break
99
# the documented compatibility invariant in regression tests. The release SARIF
10-
# gate waives only the two exact call sites; the CodeQL query remains enabled.
10+
# gate waives only three exact call sites; the CodeQL query remains enabled.
1111

1212
name: "Engraphis CodeQL config"

‎.github/workflows/ci.yml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,8 @@ jobs:
5252
run: python -m eval.grounded
5353
- name: Code-agent arm gate
5454
run: python -m eval.code_arm
55+
- name: Evidence contract boundary gate
56+
run: python -m eval.evidence_contracts
5557

5658
typecheck:
5759
name: core + backends typecheck (Python 3.11)
@@ -125,6 +127,8 @@ jobs:
125127
run: python -m eval.reinforcement
126128
- name: Adversarial memory prompt-boundary gate
127129
run: python -m eval.adversarial_memory_security
130+
- name: Evidence contract boundary gate
131+
run: python -m eval.evidence_contracts
128132
- name: Build and smoke installed core artifacts
129133
shell: bash
130134
run: |

‎.github/workflows/codeql.yml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ on:
1010

1111
permissions:
1212
contents: read
13+
packages: read
1314
security-events: write
1415

1516
jobs:
@@ -33,10 +34,23 @@ jobs:
3334
languages: ${{ matrix.language }}
3435
build-mode: none
3536
config-file: ./.github/codeql/codeql-config.yml
37+
packs: ${{ matrix.language == 'python' && 'codeql/python-queries:AlertSuppression.ql' || 'codeql/javascript-queries:AlertSuppression.ql' }}
3638
- name: Analyze
3739
id: analyze
3840
uses: github/codeql-action/analyze@b96794f015dfd88f77b49b1c93e0fa7110f94c63 # v4
3941
with:
4042
output: codeql-results
43+
upload: never
44+
category: ".github/workflows/codeql.yml:analyze/language:${{ matrix.language }}"
4145
- name: Require clean CodeQL results
4246
run: python scripts/check_codeql_sarif.py "${{ steps.analyze.outputs.sarif-output }}"
47+
- name: Filter approved non-security digest results
48+
run: >-
49+
python scripts/check_codeql_sarif.py --filter-approved
50+
"${{ steps.analyze.outputs.sarif-output }}" codeql-results-filtered
51+
- name: Upload CodeQL results
52+
uses: github/codeql-action/upload-sarif@b96794f015dfd88f77b49b1c93e0fa7110f94c63 # v4
53+
with:
54+
sarif_file: codeql-results-filtered
55+
category: ".github/workflows/codeql.yml:analyze/language:${{ matrix.language }}"
56+
wait-for-processing: true

‎BENCHMARKS.md‎

Lines changed: 160 additions & 42 deletions
Large diffs are not rendered by default.

‎CHANGELOG.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,10 @@ All notable changes to Engraphis are documented here. Format loosely follows
55

66
## [Unreleased]
77

8+
- Receipt-chain structural corruption remains fail-closed at the Store boundary without
9+
bricking a completed service operation: affected responses now carry a content-free
10+
`receipt_warning`, and graph/import workers preserve their completed state.
11+
812
## [1.7.4] - 2026-09-13
913

1014
- Writable SQLite files now default to WAL plus FULL synchronization, with an explicit

‎Dockerfile‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,9 @@ ENV PYTHONUNBUFFERED=1 \
1818
# Customer-side cloud session and entitlement display cache. Keep it on /data rather
1919
# than the container's ephemeral home so reconnects do not lose rotated credentials.
2020
# License issuance, trial state, leases, and revocations remain private services.
21-
ENGRAPHIS_STATE_DIR=/data/.engraphis
21+
ENGRAPHIS_STATE_DIR=/data/.engraphis \
22+
# Dashboard-managed non-secret settings must survive a Railway redeploy with the volume.
23+
ENGRAPHIS_ENV_FILE=/data/.engraphis/config.env
2224

2325
WORKDIR /app
2426

@@ -33,6 +35,8 @@ RUN apt-get update \
3335
COPY pyproject.toml README.md LICENSE NOTICE ./
3436
COPY engraphis ./engraphis
3537
COPY scripts ./scripts
38+
# The declared distribution license assets are part of the package build metadata.
39+
COPY deploy ./deploy
3640

3741
# Railway runs CPU workloads. Install the CPU-only PyTorch wheel before the embedding
3842
# stack so pip cannot select PyPI's multi-gigabyte CUDA dependency chain. The public

‎README.md‎

Lines changed: 41 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@ by default, or accept an explicit workspace plus optional `from_ts`, `to_ts`, an
4444
`release_version` filters.
4545

4646
<p align="center">
47-
<img src="https://raw.githubusercontent.com/Coding-Dev-Tools/engraphis/main/docs/images/context-efficiency.svg" alt="Dark chart of local measurements and deterministic fixtures, including a local LoCoMo diagnostic marked with an asterisk. Cross-session handoff satisfaction rises from 3 of 15 queries with the last memories to 15 of 15 with proactive ranking or a consolidated summary. Intent-layered graph routing rises from 0 of 3 to 3 of 3 correct top-1 targets, and two-hop graph recall rises from 0 of 3 with one-hop expansion to 3 of 3 with Personalized PageRank. Consolidation-aware ranking selects the expected digest in 2 of 2 summary cases instead of 0 of 2 for the baseline. Structure-aware chunks reduce context from 740.3 to 214.3 tokens and the smallest evidence-holding memory from 162.2 to 42.4 tokens. A compact JSON-shape proxy uses 10,982 rather than 23,810 tokens. Grounded recall makes 11 of 11 correct decisions and packed context averages 85.38 tokens under a 1,500-token cap." width="100%">
47+
<img src="https://raw.githubusercontent.com/Coding-Dev-Tools/engraphis/main/docs/images/context-efficiency.svg" alt="Dark chart of registered deterministic fixtures. Structure-aware chunks reduce retrieved context from 740.3 to 214.3 tokens and the smallest evidence-holding memory from 162.2 to 42.4 tokens. A compact JSON-shape proxy uses 11,138 rather than 24,590 tokens. Retrieved-candidate quality is labeled separately from packed-context quality, both measured in the selected report with packed-quality fields. Actual MCP transport and provider billing are not measured." width="100%">
4848
<br>
4949
<sup>Less repeated history means more room for the task, tools, and useful evidence.</sup>
5050
</p>
@@ -73,17 +73,27 @@ its counting boundary explicit.
7373
|---|---|---|---|
7474
| Retrieved top-5 memory content, averaged per question | Whole documents: **740.3** tokens → structure-aware chunks: **214.3** tokens | **526.0 fewer tokens per question** (**71.1% lower**, about **3.5× smaller**) | Recall@5 **1.000** in both modes across 6 documents and 18 questions |
7575
| Smallest returned memory that contains the reference evidence | Whole documents: **162.2** tokens → chunks: **42.4** tokens | **119.8 fewer tokens to evidence** (**73.9% lower**, about **3.8× smaller**) | The same 18 questions had a returned evidence-holding memory in both modes |
76-
| Full versus compact recall payload proxy across one 26-question pass within a 260-timed-recall CodeMem run | Full proxy: **23,810** `engraphis.regex.v1` tokens → compact proxy: **10,982** tokens | **12,828 proxy tokens avoided** (**53.88% lower**) | 26 payload samples; 260 timed recalls; Recall@5, hit@5, and answer-token recall all **1.000** |
76+
| Full versus compact recall payload proxy across one 26-question pass within a 260-timed-recall CodeMem run | Full proxy: **24,590** `engraphis.regex.v1` tokens → compact proxy: **11,138** tokens | **13,452 proxy tokens avoided** (**54.71% lower**) | 26 payload samples; 260 timed recalls; Recall@5, hit@5, and answer-token recall all **1.000** |
7777
| Packed prompt-context usage in the same 26-question CodeMem sample pass | Hard budget: **1,500** tokens; observed mean: **85.38**; observed maximum: **108** | A hard cap prevents a recall from exceeding its configured context budget | This is usage accounting, not a before/after savings comparison |
7878

79-
These values are evidence IDs `offline-chunking` and `offline-performance` in
80-
[`offline-fixtures-v9.json`](https://github.com/Coding-Dev-Tools/engraphis/blob/main/docs/benchmark-evidence/offline-fixtures-v9.json),
81-
SHA-256
82-
`455fc9d32a236e582a49aaaf9b84f30cae2573dc6ed982f4dd7dd845afcaf24c`.
79+
The performance report keeps its legacy `quality` fields for all candidate chunks returned before
80+
context packing and adds `packed_quality` for evidence admitted to the reader context. The checked-in
81+
v19 artifact includes both quality views, with Recall@5, hit@5 and answer-token evidence coverage
82+
of 1.000 for the 26-question fixture in each view. Both views measure retrieved evidence;
83+
neither is an end-to-end question-answer score. Coding outcomes, external datasets, and staged
84+
operational capacity remain separate pending evaluation tracks until their artifacts are selected.
85+
86+
These values are evidence IDs `offline-chunking` and `offline-performance` in
87+
[`offline-fixtures-v70.json`](https://github.com/Coding-Dev-Tools/engraphis/blob/main/docs/benchmark-evidence/offline-fixtures-v70.json),
88+
SHA-256
89+
`a8a96cb4d096ee72131d60307cc95b95f5b09716e813d9ee07090578dc1439c3`.
8390
[`BENCHMARKS.md`](https://github.com/Coding-Dev-Tools/engraphis/blob/main/BENCHMARKS.md#public-numeric-evidence-registry)
84-
records the matching suite digest, exact commands, and per-command config digests. External,
85-
model-dependent, consolidation, productivity, and latency results remain unpublished until the
86-
same evidence exists for them.
91+
records the matching suite digest, exact commands, and per-command config digests. The offline
92+
fixture registry intentionally excludes external, model-dependent, consolidation, productivity,
93+
and latency results. Completed retrieval-only diagnostics are published separately in the
94+
[benchmark expansion results](https://github.com/Coding-Dev-Tools/engraphis/blob/main/docs/BENCHMARK_EXPANSION_RESULTS.md) with redacted immutable
95+
artifacts; no generated-answer, official leaderboard, hosted-latency, or paid result is claimed
96+
here.
8797

8898
The compact payload shape avoids duplicating full memory bodies when the packed context and source
8999
list are enough. The evaluator tokenizes JSON-shaped full and compact payload proxies built from
@@ -546,11 +556,27 @@ For an agent prompt, prefer `engraphis_recall_context`: it returns one hard-budg
546556
`source_tokens`, `saved_tokens`, `savings_ratio`, `packed_count`, `omitted_count`, and
547557
`token_counter`), and optional diagnostics. Accounting is exact for the named counter; inject the
548558
reader's tokenizer when reader-model token parity is required. `engraphis_recall` remains the compatible full-recall
549-
surface; use `response_mode="compact"` when the packed context is enough and full memory bodies
550-
would duplicate it. For advanced query-planning configuration, see the
551-
[architecture guide](https://github.com/Coding-Dev-Tools/engraphis/blob/main/docs/ARCHITECTURE_V3.md#query-planning).
552-
553-
For bi-temporal reads, `valid_at` selects what was true at a Unix timestamp and `known_at` selects
559+
surface; use `response_mode="compact"` when the packed context is enough and full memory bodies
560+
would duplicate it. For advanced query-planning configuration, see the
561+
[architecture guide](https://github.com/Coding-Dev-Tools/engraphis/blob/main/docs/ARCHITECTURE_V3.md#query-planning).
562+
563+
Benchmark-driven alternatives are opt-in: `packing_mode="coverage"` keeps complete evidence
564+
units from more source memories, while `retrieval_recipe="conversation"` and
565+
`retrieval_recipe="long_session"` select the measured depth/budget starting points. The
566+
historical `legacy`/`default` settings remain unchanged. For a value that must survive a file
567+
edit or tool call exactly, Smart and Classic `engraphis_remember` and the Python/service write
568+
APIs accept source-bound `exact_value` plus its `exact_value_type`. MCP remember requires a
569+
unique occurrence; Python/service writes can select a repeated occurrence with `exact_value_span`.
570+
Packed binding metadata requires the complete memory source, preserving conditions in any
571+
language. Boundary whitespace outside the bound value may be trimmed. Coverage withholds a
572+
bound group that cannot fit; legacy keeps its selected text but omits the incomplete binding.
573+
Corrections and content revisions clear the old binding when content changes;
574+
pass `exact_value` to explicitly bind the replacement, with `exact_value_span=[start,end]`
575+
for a repeated occurrence, or `clear_exact_value=true` to remove a binding. Unchanged content
576+
and title-only revisions preserve valid bindings. History preserves the original record.
577+
MCP response trimming removes binding metadata whenever its supporting context is omitted.
578+
579+
For bi-temporal reads, `valid_at` selects what was true at a Unix timestamp and `known_at` selects
554580
what Engraphis had learned then. `as_of` remains a compatibility alias for `valid_at`; supplying
555581
both is allowed only when they match.
556582

@@ -762,7 +788,7 @@ file. It never searches the working directory for `.env`, and explicit process v
762788
|---------|---------|-------------|
763789
| `ENGRAPHIS_ENV_FILE` | `~/.engraphis/config.env` | Optional trusted config leaf selected before trusted values load. Its bounded dependency-free parser performs no interpolation. An explicit value must be an absolute path to an owner-private regular file; arbitrary working-directory `.env` files are ignored. |
764790
| `ENGRAPHIS_DB_PATH` | Source: `<repo>/engraphis.db`; installed: platform user-data directory | SQLite database file. Installed defaults are `%LOCALAPPDATA%\engraphis\engraphis.db` (Windows), `~/Library/Application Support/engraphis/engraphis.db` (macOS), and `$XDG_DATA_HOME/engraphis/engraphis.db` or `~/.local/share/engraphis/engraphis.db` (Linux). The environment variable overrides every default; a relative value is resolved from the trusted `~/.engraphis/config.env` directory so launch CWD cannot select a different workspace database. |
765-
| `ENGRAPHIS_SQLITE_DURABILITY` | `durable` | Writable file databases use WAL and FULL commit synchronization. Explicit `balanced` selects NORMAL, which can lose recent acknowledged writes after OS/power failure. Effective settings appear in diagnostics; see [SQLite durability](https://github.com/Coding-Dev-Tools/engraphis/blob/main/docs/SQLITE_DURABILITY.md). |
791+
| `ENGRAPHIS_SQLITE_DURABILITY` | `durable` | Writable file databases use WAL and FULL commit synchronization. Explicit `balanced` selects NORMAL, which can lose recent acknowledged writes after OS/power failure. Effective settings appear in diagnostics; see [SQLite durability](https://github.com/Coding-Dev-Tools/engraphis/blob/main/docs/SQLITE_DURABILITY.md). |
766792
| `ENGRAPHIS_HOST` | `127.0.0.1` | Server bind address |
767793
| `ENGRAPHIS_PORT` | `8700` | Dashboard port. A platform-injected `$PORT` (Railway/Fly/Heroku) takes precedence over this value for the dashboard bind; Compose pins both to `ENGRAPHIS_COMPOSE_PORT` so the mapping stays in sync |
768794
| `ENGRAPHIS_SERVICE_MODE` | `customer` | The public package supports only `customer`; hosted vendor, relay, compute, and worker roles are not distributed here |

‎deploy/railway-template.json‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,11 @@
1515
}
1616
},
1717
"variables": {
18+
"ENGRAPHIS_HOST": {
19+
"value": "0.0.0.0",
20+
"prompt": "Bind the public Railway service on all IPv4 interfaces so the platform PORT and health probe are reachable.",
21+
"required": true
22+
},
1823
"ENGRAPHIS_SERVICE_MODE": {
1924
"value": "customer",
2025
"required": true
@@ -27,6 +32,10 @@
2732
"value": "/data/.engraphis",
2833
"required": true
2934
},
35+
"ENGRAPHIS_ENV_FILE": {
36+
"value": "/data/.engraphis/config.env",
37+
"required": true
38+
},
3039
"ENGRAPHIS_API_TOKEN": {
3140
"value": "${{ secret(48) }}",
3241
"secret": true,

0 commit comments

Comments
 (0)