Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/autonomousqa.yml
Original file line number Diff line number Diff line change
Expand Up @@ -96,4 +96,4 @@ jobs:
pip install -e .
bash ../.github/scripts/install-asdecided.sh 0.23.1
- name: Smoke — drive, compile, fidelity-gate, meter, re-score
run: RAC_BIN=decided python -m runner.cli smoke
run: DECIDED_BIN=decided python -m runner.cli smoke
19 changes: 14 additions & 5 deletions .github/workflows/decisiongrounding-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ name: decisiongrounding CI
# The decisiongrounding benchmark arrived from its standalone repository with
# its workflow at decisiongrounding/.github/workflows/ci.yml — a path GitHub
# does not read in a subdirectory. This root workflow re-homes those same jobs
# (test matrix, offline demo smoke, rac corpus gate) unchanged in substance,
# (test matrix, offline demo smoke, live AsDecided arm smoke, corpus gate)
# scoped to the subdirectory.

on:
Expand Down Expand Up @@ -48,10 +48,10 @@ jobs:
- name: Smoke the offline demo
run: python -m runner.cli demo

corpus:
# Dogfood RAC: enforce the benchmark's own knowledge corpus the same way
# the benchmark studies — validation, relationships, and review must pass.
name: rac corpus gate
asdecided-core:
# Exercise the actual Rust system-under-test boundary and enforce the
# benchmark's own knowledge corpus.
name: AsDecided Core integration
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
Expand All @@ -65,5 +65,14 @@ jobs:
- name: Install AsDecided
run: bash ../.github/scripts/install-asdecided.sh 0.23.1

- name: Install benchmark
run: pip install -e ".[dev,schema]"

- name: Smoke the AsDecided whole-artifact arm
run: python -m runner.cli run --arm rac --scenarios scenarios --out /tmp/dg-core-smoke

- name: Smoke the AsDecided snippet arm
run: python -m runner.cli run --arm rac_snippets --scenarios scenarios --out /tmp/dg-core-snippets-smoke

- name: Gate the RAC corpus (validate + relationships + review)
run: decided gate rac
13 changes: 7 additions & 6 deletions decisiongrounding/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -136,7 +136,8 @@ pip install -e ".[real,schema,chart]"
export ANTHROPIC_API_KEY=... # pinned answering model: claude-opus-4-8
export VOYAGE_API_KEY=... # real embeddings for naive_rag

# rac arm additionally needs the `rac` CLI on PATH (or set RAC_BIN)
# stable rac arm additionally needs AsDecided Core's `decided` CLI on PATH
# (or set DECIDED_BIN)
python -m runner.cli compare \
--arms context_dump,naive_rag,rac \
--answering claude \
Expand All @@ -154,15 +155,15 @@ so a run says exactly what produced it.

For repeat or large runs, `python -m runner.cli batch …` runs the same
comparison through the **Message Batches API at ~50% of standard token price**.
It assembles every arm's grounding locally (rac CLI, embeddings) up front, then
It assembles every arm's grounding locally (AsDecided Core, embeddings) up front, then
submits all answering calls as one batch and polls to completion (asynchronous —
usually under an hour). The trade vs `compare` is the live, abortable per-cell
feedback; for the first exploratory run prefer `compare`, for bulk runs prefer
`batch`. (`compare`/`run`/`demo` remain synchronous and streamed.)

A real run is expensive, so the runner protects your spend two ways. It
**preflights** the configuration before doing any work — a missing
`ANTHROPIC_API_KEY`, `VOYAGE_API_KEY`, backend package, or `rac` CLI fails fast
`ANTHROPIC_API_KEY`, `VOYAGE_API_KEY`, backend package, or Core CLI fails fast
with an actionable message instead of part-way through a paid sweep. And it
**streams every completed run** to a durable `results/run-<stamp>-<label>.partial.jsonl`
sidecar as it lands; a transient API error on one (arm, scenario) cell is
Expand Down Expand Up @@ -315,14 +316,14 @@ python -m runner.cli compare \
The `rac` arm is the grounding layer under test: it follows the typed
`supersedes` edge and supplies the live PEP 440, where `naive_rag` can surface
PEP 386's appealing `verlib` section without the header that marks it superseded.
It needs the `rac` CLI on PATH (`pip install -e .` from the repo root, or set
`RAC_BIN`); drop `,rac` to run the baselines alone.
It needs AsDecided Core's `decided` CLI on PATH (install `asdecided-core`, or
set `DECIDED_BIN`); drop `,rac` to run the baselines alone.

This produces the first genuine decision-adherence result (win, tie, or loss)
on a real corpus; like every run it is appended to `results/`. The build
environment for this pilot had no API keys, so the scenario is offline-validated
(loads, schema-validates, scores, and the `rac` arm's supersedes-following is
verified against the real `rac` CLI) and the real numbers are produced by whoever
verified against the real AsDecided Core CLI) and the real numbers are produced by whoever
holds the keys.

### Real adherence-vs-N curve (real distractors, not synthetic filler)
Expand Down
2 changes: 1 addition & 1 deletion decisiongrounding/docs/AGENTIC_BENCHMARK_RUN_HANDOFF.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ Either way:
Never push to main.
- Install: `pip install -e ".[real,schema,chart]"`. The `rac` arm needs the `rac`
CLI on PATH (`brew install asdecided/tap/asdecided-core`, then
`export RAC_BIN=decided`). Alternatively `./scripts/from_source.sh` does venv + install + loads
`export DECIDED_BIN=decided`). Alternatively `./scripts/from_source.sh` does venv + install + loads
keys from .env + probe + run in one go; the granular steps below are recommended
for the first real run so you can inspect each stage.

Expand Down
4 changes: 3 additions & 1 deletion decisiongrounding/providers/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,9 @@
from .rac_snippets import RacSnippetsProvider

# Real, runnable arms this pass: context_dump, naive_rag, no_grounding,
# naive_rag_full (offline); rac, rac_snippets (need the external rac CLI).
# naive_rag_full (offline); rac, rac_snippets (need AsDecided Core's external
# `decided` CLI). The rac identifiers are stable experimental arm IDs, not
# executable or product names.
# naive_rag_full and rac_snippets are the token-budget parity variants that
# complete the 2x2 (retrieval method x grounding granularity).
# memory_provider is a typed stub.
Expand Down
74 changes: 39 additions & 35 deletions decisiongrounding/providers/rac.py
Original file line number Diff line number Diff line change
@@ -1,18 +1,19 @@
"""`rac` arm — deterministic typed retrieval via the system under test.
"""`rac` arm — deterministic typed retrieval via AsDecided Core.

This arm assembles grounding by calling RAC's deterministic, typed retrieval
This arm assembles grounding by calling AsDecided Core's deterministic, typed retrieval
surface rather than embedding similarity. It is the layer whose value the whole
benchmark exists to test, so it gets NO special treatment: same answering model,
same scaffold, one symmetric grounding opportunity.

RAC is treated as an EXTERNAL TOOL, never a Python import (ADR-0001): the arm
shells out to the pinned `rac` CLI (`rac find … --json`, `rac relationships …
AsDecided Core is treated as an EXTERNAL TOOL, never a Python import (ADR-0001):
the arm shells out to the pinned `decided` CLI (`decided find … --json`,
`decided relationships …
--json`). The thesis is that typed retrieval + relationship traversal preserves
exactly what naive_rag severs — notably supersession — by FOLLOWING `supersedes`
edges instead of hoping the superseding artifact lands in top-k.

Requires the `rac` CLI on PATH (or set RAC_BIN). It does not run in the offline
demo; install `rac` to include this arm in a comparison.
Requires the `decided` CLI on PATH (or set DECIDED_BIN). It does not run in the
offline demo; install AsDecided Core to include this arm in a comparison.
"""

from __future__ import annotations
Expand All @@ -28,26 +29,26 @@
from pathlib import Path


def rac_version() -> str | None:
"""Best-effort version of the `rac` CLI — the system under test.
def core_version() -> str | None:
"""Best-effort version of AsDecided Core — the system under test.

The benchmark pins rac only by convention, so without this two runs
against different rac builds produce indistinguishable reports. Returns
the stripped `rac --version` output, or None when the binary is absent
Without this, two runs against different Core builds produce
indistinguishable reports. Returns the stripped `decided --version` output,
or None when the binary is absent
or the call fails: provenance must never break a run.
"""
return _rac_version(os.environ.get("RAC_BIN", "rac"))
return _core_version(os.environ.get("DECIDED_BIN", "decided"))


@lru_cache(maxsize=8)
def _rac_version(rac_bin: str) -> str | None:
def _core_version(decided_bin: str) -> str | None:
"""One subprocess per resolved binary per process (see the lru_cache);
tests vary RAC_BIN and call `_rac_version.cache_clear()`."""
if shutil.which(rac_bin) is None:
tests vary DECIDED_BIN and call `_core_version.cache_clear()`."""
if shutil.which(decided_bin) is None:
return None
try:
proc = subprocess.run(
[rac_bin, "--version"], capture_output=True, text=True, timeout=10
[decided_bin, "--version"], capture_output=True, text=True, timeout=10
)
except Exception: # noqa: BLE001 - absence/misbehaviour is expected, not fatal
return None
Expand All @@ -66,7 +67,7 @@ def _rac_version(rac_bin: str) -> str | None:


def _query_tokens(text: str) -> list[str]:
"""Salient single terms for `rac find`, which substring-matches ID/title and
"""Salient single terms for `decided find`, which matches ID/title and
narrows on multi-word queries. Querying one term at a time and unioning the
hits is the deterministic way to retrieve by topic over that surface."""
seen: list[str] = []
Expand Down Expand Up @@ -130,26 +131,29 @@ def __init__(self, answering_model, top_k: int = 4):

@staticmethod
def _bin() -> str:
return os.environ.get("RAC_BIN", "rac")
return os.environ.get("DECIDED_BIN", "decided")

def _run(self, *args: str) -> dict:
rac = self._bin()
if shutil.which(rac) is None:
decided = self._bin()
if shutil.which(decided) is None:
raise RuntimeError(
f"rac CLI not found ({rac!r}). Install `rac` or set RAC_BIN to "
"include the rac arm. It does not run in the offline demo."
f"AsDecided Core CLI not found ({decided!r}). Install "
"`asdecided-core` or set DECIDED_BIN to include the rac arm. "
"It does not run in the offline demo."
)
proc = subprocess.run([rac, *args], capture_output=True, text=True)
proc = subprocess.run([decided, *args], capture_output=True, text=True)
if proc.returncode != 0:
raise RuntimeError(
f"rac CLI failed ({rac} {' '.join(args)}): exit {proc.returncode}\n"
f"AsDecided Core CLI failed "
f"({decided} {' '.join(args)}): exit {proc.returncode}\n"
f"{proc.stderr.strip()}"
)
try:
return json.loads(proc.stdout)
except json.JSONDecodeError as exc:
raise RuntimeError(
f"rac CLI returned non-JSON for ({rac} {' '.join(args)}): {exc}"
f"AsDecided Core CLI returned non-JSON for "
f"({decided} {' '.join(args)}): {exc}"
) from exc

def _find_candidates(self, task: Task) -> list[str]:
Expand All @@ -170,15 +174,15 @@ def _find_candidates(self, task: Task) -> list[str]:
return sorted(order, key=lambda i: (-hits[i], order.index(i)))

def prepare(self, corpus: list[CorpusArtifact]) -> None:
# Write the corpus to a temp dir so the external `rac` CLI can index it.
# Write the corpus to a temp dir so the external Core CLI can index it.
self._by_id = {a.id: a for a in corpus}
tmp = Path(tempfile.mkdtemp(prefix="dg-rac-"))
tmp = Path(tempfile.mkdtemp(prefix="dg-asdecided-"))
for a in corpus:
(tmp / f"{a.id}.md").write_text(a.text, encoding="utf-8")
self._dir = tmp
# The corpus is written fresh every prepare(); clean up the previous temp
# dir and ensure this one is removed when the provider is collected, so a
# long sweep does not leak a dg-rac-* directory per run.
# long sweep does not leak a dg-asdecided-* directory per run.
self._cleanup()
self._finalizer = weakref.finalize(self, shutil.rmtree, tmp, ignore_errors=True)
self._grounding = GroundingContext(text="", artifacts_supplied=(), token_estimate=0)
Expand All @@ -195,10 +199,10 @@ def _resolve(self, task: Task) -> list[str]:
differ only in grounding granularity, never in retrieval."""
if self._dir is None:
raise RuntimeError("rac arm: prepare() must run before assemble()")
# 1. Typed candidate decisions. `rac find` substring-matches ID/title and
# 1. Typed candidate decisions. `decided find` matches ID/title and
# ANDs multi-word queries, so we query one salient task term at a time and
# union the hits, ranking by how many terms hit each decision (then by
# first appearance) — a deterministic topic search over rac's surface.
# first appearance) — a deterministic topic search over Core's surface.
matched = self._find_candidates(task)

# 2. Relationship graph → supersedes edges (source supersedes target).
Expand Down Expand Up @@ -226,12 +230,12 @@ def assemble(self, task: Task) -> GroundingContext:


def _extract_supersedes_edges(rels_json: dict) -> list[tuple[str, str]]:
"""Pull (source, target) supersedes pairs from `rac relationships --json`.
"""Pull (source, target) pairs from `decided relationships --json`.

Defensive against shape drift across rac versions: handles both a flat
Defensive against shape drift across Core versions: handles both a flat
relationships list (entries with relationship == "supersedes") and an
artifacts[] report carrying a per-artifact `relationships.supersedes` list.
TODO(rac-arm): pin and verify against the installed rac version's JSON.
TODO(rac-arm): pin and verify against the installed Core version's JSON.
"""
edges: list[tuple[str, str]] = []

Expand All @@ -242,8 +246,8 @@ def _extract_supersedes_edges(rels_json: dict) -> list[tuple[str, str]]:
edges.append((src, tgt))

for art in rels_json.get("artifacts", []) or []:
# `rac relationships --json` identifies each artifact by `path`, not `id`
# (this rac build emits no `id` here). The arm writes the corpus as
# `decided relationships --json` identifies each artifact by `path`, not
# `id` (this Core build emits no `id` here). The arm writes the corpus as
# `<id>.md`, so the file stem is the artifact id; prefer an explicit `id`
# if a future rac version adds one.
src = art.get("id") or Path(art.get("path", "")).stem or None
Expand Down
7 changes: 4 additions & 3 deletions decisiongrounding/providers/rac_snippets.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,15 +7,16 @@
naive_rag comparison: does typed, supersession-aware retrieval help, and
does dumping whole artifacts help or hurt?

This arm isolates the first. It runs rac's exact typed retrieval
This arm isolates the first. It runs AsDecided Core's exact typed retrieval
(`RacProvider._resolve` — same candidates, same supersedes traversal, same
top_k), then presents the resolved artifacts as SECTION SNIPPETS under a
token budget matched to naive_rag's typical grounding size — so `rac_snippets`
vs `naive_rag` compares retrieval method at an equal budget, and `rac_snippets`
vs `rac` compares snippet vs whole-artifact granularity at equal retrieval.

Requires the `rac` CLI (inherits `RacProvider`); does not run in the offline
demo. See the 2×2 with `naive_rag_full` (cosine retrieval, whole artifacts).
Requires the `decided` CLI (inherits `RacProvider`); does not run in the
offline demo. See the 2×2 with `naive_rag_full` (cosine retrieval, whole
artifacts).
"""

from __future__ import annotations
Expand Down
5 changes: 3 additions & 2 deletions decisiongrounding/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,9 @@ local-embeddings = ["sentence-transformers>=2.2"]
# Local web UI (`decisiongrounding ui`). Optional and lazily imported so the
# core stays dependency-free; the page needs no frontend build step.
ui = ["fastapi>=0.110", "uvicorn>=0.27"]
# The rac arm additionally needs the `rac` CLI on PATH (or RAC_BIN) — it is an
# external tool, not a Python dependency (ADR-0001), so it is not listed here.
# The stable rac arm additionally needs AsDecided Core's `decided` CLI on PATH
# (or DECIDED_BIN). It is the external system under test, not a Python
# dependency (ADR-0001), so it is not listed here.
# Test runner.
dev = ["pytest>=7"]

Expand Down
18 changes: 9 additions & 9 deletions decisiongrounding/runner/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@
check_context_window,
)
from providers.answering import error_kind # noqa: E402
from providers.rac import rac_version # noqa: E402
from providers.rac import core_version # noqa: E402
from scenarios.loader import Scenario, load_pool, load_scenarios # noqa: E402
from scoring import aggregate, score # noqa: E402
from scoring.crossover import ( # noqa: E402
Expand Down Expand Up @@ -161,11 +161,11 @@ def _preflight(arms: tuple[str, ...], answering: str, embedder: str) -> None:
)

if {"rac", "rac_snippets"} & set(arms) and shutil.which(
os.environ.get("RAC_BIN", "rac")
os.environ.get("DECIDED_BIN", "decided")
) is None:
problems.append(
"the rac/rac_snippets arms need the `rac` CLI on PATH (pip install -e "
"the rac repo, or set RAC_BIN)"
"the rac/rac_snippets arms need AsDecided Core's `decided` CLI on "
"PATH (install `asdecided-core`, or set DECIDED_BIN)"
)

if problems:
Expand Down Expand Up @@ -257,12 +257,12 @@ def _backend_versions() -> dict:
out[pkg] = md.version(pkg)
except Exception: # noqa: BLE001 - absence is expected offline
pass
# The system under test: without this, two runs against different rac
# The system under test: without this, two runs against different Core
# builds produce indistinguishable reports. Same omit-when-absent
# convention as the pip packages.
v = rac_version()
v = core_version()
if v is not None:
out["rac"] = v
out["asdecided_core"] = v
return out


Expand Down Expand Up @@ -456,7 +456,7 @@ def cmd_compare(args) -> int:

def cmd_batch(args) -> int:
"""Same comparison as `compare`, but the answering calls go through the
Message Batches API at 50% of standard price. Grounding assembly (rac CLI,
Message Batches API at 50% of standard price. Grounding assembly (AsDecided Core,
embeddings) still runs locally up front; only the held-constant answering
model is batched. Asynchronous: submit one batch, poll, then score."""
arms = tuple(a.strip() for a in args.arms.split(",") if a.strip())
Expand All @@ -472,7 +472,7 @@ def cmd_batch(args) -> int:
stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
label = "batch-" + "-".join(arms)

# 1. Assemble every cell's grounding + request locally (rac CLI / embeddings
# 1. Assemble every cell's grounding + request locally (Core CLI / embeddings
# happen here). The answering calls are submitted together below. A cell
# that would exceed the answering model's context window is skipped
# here — the same symmetric preflight as Provider.respond() and
Expand Down
2 changes: 1 addition & 1 deletion decisiongrounding/runner/dashboard.py
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ def curve_from_dataset(dataset) -> dict | None:
"naive_rag": "embeds the corpus, retrieves top-k chunks (classic RAG)",
"naive_rag_full": "cosine retrieval, whole artifact per top hit (whole-artifact granularity)",
"no_grounding": "supplies nothing — parametric memory only (control)",
"rac": "typed, supersession-aware grounding assembled by the rac CLI",
"rac": "typed, supersession-aware grounding assembled by AsDecided Core",
"rac_snippets": "rac's typed retrieval, snippets under naive_rag's token budget (equal budget)",
"memory_provider": "pluggable external memory provider (stub)",
}
Expand Down
Loading
Loading