feat(gitchameleon): ready the funded evidence run

gitchameleon/README.md

@@ -30,9 +30,10 @@ per-arm pass rate is SWE-DecisionBench's second co-primary outcome,
 ## Design (GCB-ADR-0001)
 
 - **Arms** (DG-ADR-0001 single-variable pattern; held-constant answering
-  model): `no_grounding` / `rac` (live-decision retrieval over the example's
-  corpus via the shared harness runner — rac strictly as an external CLI) /
-  `naive_rag` (refuses until its embedder is pinned at funded-run time).
+  model): `no_grounding` / `asdecided` (live-decision retrieval over the
+  example's corpus via the shared harness runner — As Decided strictly as an
+  external CLI) / `naive_rag` (embedding retrieval over the identical corpus, pinned to
+  `voyage-4-large`, with query/document input types and cosine ranking).
 - **Corpus**: `build_corpus.py` turns each problem into a RAC decision
   artifact — the version pin, its rationale, companion pins, and the
   dataset's documentation links; never the solution, function name, or
@@ -50,14 +51,14 @@ per-arm pass rate is SWE-DecisionBench's second co-primary outcome,
 ## Scaffold usage (offline, no model calls)
 
 ```
-python3 fetch_dataset.py                       # dataset/ (gitignored) + provenance pin
+python3 fetch_dataset.py --revision 799a6a33e572a07a8985914e7251f5dea54b0ac4
 python3 build_corpus.py                        # per-example corpora under corpus-build/
-python3 run.py --dry-run                       # per-example, per-arm prompt bundles
+python3 run.py --dry-run                       # no_grounding + asdecided bundles
 python3 run.py --dry-run \
   --dataset fixtures/sample_problems.json      # the same, offline from the fixtures
 ```
 
-`rac` must be on `PATH` for the rac arm (external CLI only — no engine
+`decided` must be on `PATH` for the `asdecided` arm (external CLI only — no engine
 imports, DG-ADR-0001).
 
 ## The funded run (GCB-ADR-0002 — the resolution co-primary pipeline)
@@ -66,36 +67,50 @@ The pre-registered analysis and falsifier for this outcome live in
 `../decisiongrounding/spec/analysis-plan-amendment-1.md` (H2). Each step is
 resumable in isolation:
 
-1. Pin the answering model (decisiongrounding pins `claude-opus-4-8`) and the
-   `naive_rag` embedder (`voyage:voyage-4-large` is the published strong
-   baseline there).
-2. Answer every dry-run bundle:
+1. Build all three arm bundles. The answering model is pinned to
+   `claude-opus-4-8`; the baseline embedder is pinned to
+   `voyage-4-large` (GCB-ADR-0003). Every frozen input is recorded in
+   [`run-config.json`](run-config.json):
 
    ```
+   VOYAGE_API_KEY=... python3 run.py --dry-run \
+     --arms no_grounding,asdecided,naive_rag --out out/bundles.jsonl
+   ```
+
+2. Run a one-example-per-arm shakedown, inspect it, then answer every bundle.
+   `--resume` appends only missing example IDs, flushes every completion, and
+   records hashes of the exact prompt and grounding injected into the model:
+
+   ```
+   python3 run.py solutions --bundles out/bundles.jsonl \
+     --answering claude --seed 0 --limit 1 --out out/shakedown
    python3 run.py solutions --bundles out/bundles.jsonl \
-     --answering claude --seed 0 --out out/solutions
+     --answering claude --seed 0 --resume --out out/solutions
    ```
 
    writes `out/solutions/solutions-<arm>.jsonl` in exactly the upstream
    `Solution` shape (`example_id` + `answer`; the provenance extras are
    ignored by upstream). `--answering offline-stub` exercises the plumbing
    keylessly; `litellm:<alias>` targets an OpenAI-compatible gateway.
 3. Score each arm's file with the upstream harness — its executable tests
-   are the scorer; we add nothing. Clone GitChameleonBenchmark at a recorded
-   commit and run `evaluate --solution-path out/solutions/solutions-<arm>.jsonl`
-   (Docker; budget the per-version dependency installs). It writes
+   are the scorer; we add nothing. Clone GitChameleonBenchmark at commit
+   `3a1b6045a6b2a276bd24d715589cb041f8eccb93`, build its Docker image locally
+   from that checkout, and run
+   `evaluate --solution-path out/solutions/solutions-<arm>.jsonl` (budget the
+   per-version dependency installs). It writes
    `solutions-<arm>_eval_results.csv` next to each solution file.
 4. Normalize the verdicts into paired resolution records
    (`schema/resolution_record.schema.json`) and run the pre-registered
    paired analysis:
 
    ```
-   python3 run.py score --arm rac \
-     --eval-results out/solutions/solutions-rac_eval_results.csv \
+   python3 run.py score --arm asdecided \
+     --eval-results out/solutions/solutions-asdecided_eval_results.csv \
      --answering-model claude-opus-4-8 --upstream-harness <commit> \
      --out out/resolution_records.jsonl
    python3 run.py score --arm no_grounding --eval-results … --append …
-   python3 run.py stats --records out/resolution_records.jsonl
+   python3 run.py stats --records out/resolution_records.jsonl \
+     --require-arms no_grounding,asdecided,naive_rag
    ```
 
 5. Publish the records, per-arm pass rates, and stats with the dataset
@@ -105,4 +120,5 @@ resumable in isolation:
 
 ## Local decisions
 
-See [`decisions/`](decisions/) for benchmark-local design records.
+See [`decisions/`](decisions/) for benchmark-local design records, including
+the funded-run execution contract and pinning rationale (GCB-ADR-0003).

gitchameleon/answer.py

@@ -21,6 +21,8 @@
 
 import json
 import os
+import time
+import urllib.error
 import urllib.request
 
 # The held-constant instruction every arm shares; the task prompt itself is
@@ -32,6 +34,7 @@
 
 PINNED_CLAUDE = "claude-opus-4-8"
 MAX_TOKENS = 2048
+MAX_ATTEMPTS = 5
 
 
 def _compose_user_prompt(prompt: str, grounding: list[str]) -> str:
@@ -67,7 +70,9 @@ class ClaudeModel:
     version = PINNED_CLAUDE
 
     def __init__(self, seed: int = 0):
-        self.seed = seed  # recorded for provenance; the pinned model rejects sampling seeds
+        self.seed = (
+            seed  # recorded for provenance; the pinned model rejects sampling seeds
+        )
         if not os.environ.get("ANTHROPIC_API_KEY"):
             raise SystemExit("the claude answering model needs ANTHROPIC_API_KEY")
         self._client = None
@@ -80,13 +85,36 @@ def _ensure_client(self):
         return self._client
 
     def complete(self, prompt: str, grounding: list[str]) -> str:
-        msg = self._ensure_client().messages.create(
-            model=self.version,
-            max_tokens=MAX_TOKENS,
-            system=SYSTEM,
-            messages=[{"role": "user", "content": _compose_user_prompt(prompt, grounding)}],
+        import anthropic
+
+        retryable = (
+            anthropic.RateLimitError,
+            anthropic.APIConnectionError,
+            anthropic.InternalServerError,
         )
-        return "".join(b.text for b in msg.content if getattr(b, "type", "") == "text")
+        for attempt in range(MAX_ATTEMPTS):
+            try:
+                msg = self._ensure_client().messages.create(
+                    model=self.version,
+                    max_tokens=MAX_TOKENS,
+                    system=SYSTEM,
+                    messages=[
+                        {
+                            "role": "user",
+                            "content": _compose_user_prompt(prompt, grounding),
+                        }
+                    ],
+                )
+                return "".join(
+                    block.text
+                    for block in msg.content
+                    if getattr(block, "type", "") == "text"
+                )
+            except retryable:
+                if attempt + 1 == MAX_ATTEMPTS:
+                    raise
+                time.sleep(min(2**attempt, 30))
+        raise AssertionError("unreachable")
 
 
 class LiteLLMModel:
@@ -112,7 +140,10 @@ def complete(self, prompt: str, grounding: list[str]) -> str:
                 "max_tokens": MAX_TOKENS,
                 "messages": [
                     {"role": "system", "content": SYSTEM},
-                    {"role": "user", "content": _compose_user_prompt(prompt, grounding)},
+                    {
+                        "role": "user",
+                        "content": _compose_user_prompt(prompt, grounding),
+                    },
                 ],
             }
         ).encode("utf-8")
@@ -124,9 +155,23 @@ def complete(self, prompt: str, grounding: list[str]) -> str:
                 "Authorization": f"Bearer {self._key}",
             },
         )
-        with urllib.request.urlopen(req, timeout=300) as resp:
-            payload = json.loads(resp.read().decode("utf-8"))
-        return payload["choices"][0]["message"]["content"]
+        for attempt in range(MAX_ATTEMPTS):
+            try:
+                with urllib.request.urlopen(req, timeout=300) as resp:
+                    payload = json.loads(resp.read().decode("utf-8"))
+                return payload["choices"][0]["message"]["content"]
+            except urllib.error.HTTPError as exc:
+                retryable = exc.code == 429 or 500 <= exc.code < 600
+                if not retryable or attempt + 1 == MAX_ATTEMPTS:
+                    raise
+                retry_after = exc.headers.get("Retry-After")
+                delay = float(retry_after) if retry_after else min(2**attempt, 30)
+                time.sleep(delay)
+            except urllib.error.URLError:
+                if attempt + 1 == MAX_ATTEMPTS:
+                    raise
+                time.sleep(min(2**attempt, 30))
+        raise AssertionError("unreachable")
 
 
 def make_answering_model(name: str, seed: int = 0):

gitchameleon/arms.py

@@ -5,8 +5,8 @@
 answering model (the DG-ADR-0001 single-variable design):
 
 - ``no_grounding`` — the problem alone; the model answers from its weights.
-- ``rac``          — the governing version-pin decision retrieved from the
-  example's corpus via the live-decision query (`rac find --decisions`),
+- ``asdecided``    — the governing version-pin decision retrieved from the
+  example's corpus via the live-decision query (`decided find --decisions`),
   driven strictly as an external CLI through the shared harness runner.
 - ``naive_rag``    — embedding retrieval over the same corpus. Deliberately a
   seam: the embedder is pinned at funded-run time (mirroring
@@ -16,48 +16,161 @@
 
 from __future__ import annotations
 
+import hashlib
+import json
+import math
+import os
+import time
+import urllib.error
+import urllib.request
 from pathlib import Path
 
 from harness.runner import RacRunner
 
-ARMS = ("no_grounding", "rac", "naive_rag")
-# How many retrieved decisions the rac arm feeds the answering model.
-RAC_TOP_K = 3
+ARMS = ("no_grounding", "asdecided", "naive_rag")
+# Both grounded arms supply the same number of artifacts to the answering model.
+GROUNDING_TOP_K = 3
+NAIVE_RAG_MODEL = "voyage-4-large"
+VOYAGE_EMBEDDINGS_URL = "https://api.voyageai.com/v1/embeddings"
 
 
-def rac_grounding(runner: RacRunner, corpus_dir: Path, row: dict) -> list[str]:
-    """The rac arm: live-decision retrieval, retrieved artifacts verbatim.
+class VoyageEmbedder:
+    """Pinned, retrieval-only baseline used by the funded naive_rag arm.
+
+    Embeddings are cached by exact text within one run. This matters because
+    deterministic distractors recur across per-example corpora; repeated text
+    must not create repeated spend or a subtly different baseline.
+    """
+
+    model = NAIVE_RAG_MODEL
+
+    def __init__(self, api_key: str | None = None, *, max_attempts: int = 5):
+        self.api_key = api_key or os.environ.get("VOYAGE_API_KEY")
+        if not self.api_key:
+            raise ValueError("the naive_rag arm needs VOYAGE_API_KEY")
+        self.max_attempts = max_attempts
+        self._cache: dict[tuple[str, str], list[float]] = {}
+
+    def _request(self, texts: list[str], input_type: str) -> list[list[float]]:
+        body = json.dumps(
+            {
+                "input": texts,
+                "model": self.model,
+                "input_type": input_type,
+                "truncation": False,
+            }
+        ).encode("utf-8")
+        request = urllib.request.Request(
+            VOYAGE_EMBEDDINGS_URL,
+            data=body,
+            headers={
+                "Content-Type": "application/json",
+                "Authorization": f"Bearer {self.api_key}",
+            },
+        )
+        for attempt in range(self.max_attempts):
+            try:
+                with urllib.request.urlopen(request, timeout=300) as response:
+                    payload = json.loads(response.read().decode("utf-8"))
+                vectors = [entry["embedding"] for entry in payload["data"]]
+                if len(vectors) != len(texts):
+                    raise RuntimeError(
+                        f"Voyage returned {len(vectors)} vectors for {len(texts)} texts"
+                    )
+                return vectors
+            except urllib.error.HTTPError as exc:
+                retryable = exc.code == 429 or 500 <= exc.code < 600
+                if not retryable or attempt + 1 == self.max_attempts:
+                    raise
+                retry_after = exc.headers.get("Retry-After")
+                delay = float(retry_after) if retry_after else min(2**attempt, 30)
+                time.sleep(delay)
+            except urllib.error.URLError:
+                if attempt + 1 == self.max_attempts:
+                    raise
+                time.sleep(min(2**attempt, 30))
+        raise AssertionError("unreachable")
+
+    def embed(self, texts: list[str], input_type: str) -> list[list[float]]:
+        keys = [
+            (input_type, hashlib.sha256(text.encode("utf-8")).hexdigest())
+            for text in texts
+        ]
+        missing: list[tuple[tuple[str, str], str]] = [
+            (key, text) for key, text in zip(keys, texts) if key not in self._cache
+        ]
+        if missing:
+            vectors = self._request([text for _, text in missing], input_type)
+            for (key, _), vector in zip(missing, vectors):
+                self._cache[key] = vector
+        return [self._cache[key] for key in keys]
+
+
+def _cosine(left: list[float], right: list[float]) -> float:
+    if len(left) != len(right):
+        raise ValueError("embedding dimensions differ")
+    numerator = sum(a * b for a, b in zip(left, right))
+    left_norm = math.sqrt(sum(value * value for value in left))
+    right_norm = math.sqrt(sum(value * value for value in right))
+    return numerator / (left_norm * right_norm) if left_norm and right_norm else 0.0
+
+
+def asdecided_grounding(runner: RacRunner, corpus_dir: Path, row: dict) -> list[str]:
+    """The As Decided arm: live-decision retrieval, artifacts verbatim.
 
     The query is what an agent grounding itself would ask before writing code
     against a library: which live decisions bind this dependency?
     """
     query = f"{row['library']} version pin"
     returned = runner.find_ids(query, str(corpus_dir), decisions=True)
     grounding: list[str] = []
-    for artifact_id in returned[:RAC_TOP_K]:
+    for artifact_id in returned[:GROUNDING_TOP_K]:
         resolved = runner.resolve(artifact_id, str(corpus_dir))
         payload = resolved.payload()
         if resolved.exit_code == 0 and "path" in payload:
             grounding.append(Path(payload["path"]).read_text(encoding="utf-8"))
     return grounding
 
 
+def naive_rag_grounding(
+    embedder: VoyageEmbedder, corpus_dir: Path, row: dict
+) -> list[str]:
+    """Embedding retrieval over the exact corpus presented to RAC.
+
+    Files are ranked by cosine similarity, then path for deterministic ties.
+    Query and document input types follow Voyage's retrieval contract.
+    """
+    paths = sorted(corpus_dir.glob("*.md"))
+    documents = [path.read_text(encoding="utf-8") for path in paths]
+    if not documents:
+        return []
+    document_vectors = embedder.embed(documents, "document")
+    query_vector = embedder.embed([f"{row['library']} version pin"], "query")[0]
+    ranked = sorted(
+        zip(paths, documents, document_vectors),
+        key=lambda item: (-_cosine(query_vector, item[2]), str(item[0])),
+    )
+    return [document for _, document, _ in ranked[:GROUNDING_TOP_K]]
+
+
 def assemble_grounding(
-    arm: str, runner: RacRunner | None, corpus_dir: Path, row: dict
+    arm: str,
+    runner: RacRunner | None,
+    corpus_dir: Path,
+    row: dict,
+    embedder: VoyageEmbedder | None = None,
 ) -> list[str]:
     """The grounding context one arm supplies for one example."""
     if arm == "no_grounding":
         return []
-    if arm == "rac":
+    if arm == "asdecided":
         if runner is None:
-            raise ValueError("the rac arm needs a RacRunner (rac CLI on PATH)")
-        return rac_grounding(runner, corpus_dir, row)
+            raise ValueError("the asdecided arm needs the decided CLI on PATH")
+        return asdecided_grounding(runner, corpus_dir, row)
     if arm == "naive_rag":
-        raise NotImplementedError(
-            "naive_rag is a funded-run seam: pin the embedder there "
-            "(decisiongrounding pins voyage:voyage-4-large as the strong "
-            "published baseline) rather than shipping a weak lexical stand-in"
-        )
+        if embedder is None:
+            raise ValueError("the naive_rag arm needs the pinned Voyage embedder")
+        return naive_rag_grounding(embedder, corpus_dir, row)
     raise ValueError(f"unknown arm {arm!r}; expected one of {ARMS}")