PolicyC

PolicyC is a polyglot research system for testing whether a task-specific slice of a large policy prompt can preserve the behaviorally relevant obligations of the full policy while using less context.

It deliberately separates two concerns:

• TypeScript is the single authority for policy loading, validation, matching, graph closure, candidate construction, prompt emission, hashing, and artifact serialization.
• Python owns repeated model trials, bounded concurrency, rate limiting, retries, timeouts, cancellation, streamed events, persistence, evaluation, statistics, and baseline comparison.

PolicyC does not currently prove behavioral equivalence. Its included end-to-end demo uses a deterministic fake provider to verify runtime mechanics. A real OpenAI adapter is available only when explicitly configured with credentials.

Architecture

flowchart LR
    P["YAML policy packs"] --> V["Zod runtime validation"]
    V --> G["Graph validation"]
    X["Request + artifact context"] --> S["Deterministic selector"]
    G --> S
    S --> C["Five candidate strategies"]
    C --> A["Versioned JSON artifacts + hashes"]
    A --> M["Run manifest"]
    M --> R["Python asyncio runtime"]
    R --> Q["Bounded worker queue + rate limiter"]
    Q --> F["Fake or OpenAI provider"]
    F --> E["Rule-based evaluator"]
    E --> O["Atomic trials + JSONL/SSE events + report"]

Loading

The protocol boundary is defined by JSON Schemas under protocol/. Python validates TypeScript artifacts and verifies candidate and compiled-prompt hashes before execution.

Compiler pipeline

Compiler 0.7 loads 43 manually structured policy nodes from six YAML packs. PolicyC does not yet extract those nodes from arbitrary natural-language prompts.

1. Zod validates required fields, enums, priorities, triggers, and unknown fields.
2. Graph validation rejects duplicate IDs/edges, missing references, self-dependencies, cycles, unreachable structural nodes, unknown validators, and invalid always-active configurations.
3. Regex intent detection and structured artifact context select seed policies.
4. Queue-based traversal adds the transitive requires closure.
5. Policies are ordered deterministically and emitted as a runtime prompt.
6. The compiler serializes five experimental candidates:
   • full_policy
   • compiler_slice
   • kernel_only
   • direct_matches
   • conservative_expanded

Candidate names describe construction strategies, not behavioral equivalence.

Python runtime

The Python package lives in runtime/python and targets Python 3.12+.

The scheduler creates a fixed number of worker tasks and feeds them through a bounded queue. A second semaphore enforces provider-specific concurrency. An in-process sliding-window limiter can constrain requests and estimated prompt tokens. Each invocation has a timeout; retryable failures use exponential backoff with deterministic jitter and optional provider Retry-After values.

Trial IDs are hashes of stable run, provider, model, candidate, and sample inputs. Completed trials are written atomically and reused after restart when their provenance hash matches. Incomplete and retryable trials execute again. Cancellation propagates to active provider tasks.

Events receive a per-run monotonic sequence number, persist as JSONL, and serialize as SSE. FastAPI provides:

• POST /runs
• GET /runs/{run_id}
• GET /runs/{run_id}/events
• POST /runs/{run_id}/cancel
• GET /runs/{run_id}/report

Raw response retention is explicit in each manifest: none, text, or full. Secrets are never written by the runtime.

Research question and evaluation

Given a large system prompt P and a user request x, can we compile P into a much smaller active policy subset Pₓ such that a model using Pₓ preserves the same critical obligations as a model using the full prompt P?

Structural evidence and behavioral evidence are kept separate.

Structural evaluation covers selected-policy precision/recall, critical-policy recall, dependency completeness, graph validity, deterministic artifact construction, and exact or estimated token counts.

The paired path evaluates the full-policy and compiler-slice responses for each case against the same independently authored case obligations, prohibitions, refusal expectation, tool expectation, and rubric. Obligations never come from the candidate being judged. Reports expose both-pass, both-fail, full-pass/compiled-fail, and compiled-pass/full-fail pairs, exact obligation regressions, severe failures, Wilson intervals, and descriptive McNemar analysis. Selector metrics are structural metrics, not behavioral metrics.

The built-in evaluator is intentionally small and deterministic. Model graders can be added as adapters, but should not be treated as ground truth.

Latest frozen result

Compiler 0.7 did not pass its preregistered held-out-v3 test. In 360 GPT-5 mini executions over 60 independently authored cases, exhaustive strategy-blind semantic review measured 130 both-pass, 33 full-only, 11 compiler-only, and six both-fail pairs. Conditional critical-obligation preservation was 130/163 = 79.75% (Wilson 95%: 72.93%--85.21%), versus the 95% target. The compiler reduced mean actual input tokens by 93.75% and uncached-equivalent cost by 59.46%, but total billed cost fell only 12.14% because the compiled condition made more paid searches. The run cost $0.9333. See eval/audits/held-out-v3-execution.md for the immutable hashes, incident accounting, blind-review provenance, gate decisions, and regression breakdown.

Setup

pnpm install
pnpm build

python3.12 -m venv .venv
.venv/bin/pip install -e 'runtime/python[dev]'

The local environment may use a newer Python, but CI verifies Python 3.12.

CLI

Existing commands remain available:

pnpm policyc select --input "what's the latest OpenAI news?"
pnpm policyc compile --input "rewrite this email professionally"
pnpm policyc inspect --policy current_info_requires_web
pnpm policyc eval

Generate cross-language experiment artifacts:

node dist/cli.js compile-candidates \
  --input "what's the latest OpenAI news?" \
  --output experiment \
  --model fake-v1

Run and stream an offline experiment:

.venv/bin/policyc-runtime run experiment/manifest.json

Or run the complete demo:

pnpm demo

The demo compiles five candidates, executes three samples per candidate with maximum concurrency four, streams SSE-compatible events, atomically persists 15 trial results, and produces report.json. Its compliance results are fake-provider evidence only.

Safe paired OpenAI experiments

The supported entry point compiles only the requested strategies, validates the cases and fixed model pricing, creates one full-policy baseline per case, and delegates scheduling to Python. The default provider is fake; unknown providers fail closed. The pinned smoke model is gpt-5-mini-2025-08-07, using the versioned registry at pricing/openai-v1.json.

Dry-run the one-case experiment without a key or network access:

pnpm policyc experiment \
  --cases eval/behavioral/smoke-v1.jsonl \
  --strategies full_policy,compiler_slice \
  --provider openai \
  --model gpt-5-mini-2025-08-07 \
  --samples 1 --concurrency 1 \
  --max-output-tokens 1024 --max-calls 2 \
  --max-cost-usd 0.02 --retries 0 \
  --run-label smoke-1 \
  --output runs/openai-smoke --dry-run

The dry run validates every provider payload and prints the exact paid command. To execute later, export OPENAI_API_KEY, remove --dry-run, and either type the displayed RUN <run-id> confirmation or deliberately add --yes for non-interactive execution. Keys are read from the environment and are never persisted.

Hard limits cover logical trials, provider attempts, cumulative input tokens, cumulative output tokens, dollar exposure, fixed model, and per-call output tokens. Checks run before the experiment, before every attempt, after returned usage, after retries, and on resume. Pricing includes standard input, cached input, and output; missing usage fails closed rather than becoming zero. seed is not sent.

Raw HTTP responses are atomically persisted before parsing and evaluation. Completed provider calls resume without another call. The manifest embeds the Git commit and dirty state used to construct the run, and the SQLite catalog retains the same provenance. A timeout or transport disconnect is recorded as an ambiguous paid attempt, consumes call and worst-case cost exposure, and is not retried unless --retry-ambiguous was explicitly selected. This mitigates duplicate billing but cannot provide exactly-once semantics across an uncertain network boundary.

Pricing registry v2 also reserves and records OpenAI built-in web-search fees. Web-enabled responses set max_tool_calls: 1 and search_context_size: low; reports separate built-in call counts and tool cost. OpenAI documents that search context size does not guarantee an exact token count, so retrieved search-content tokens are charged from actual provider usage and can overshoot the pre-call token estimate for the in-flight request. Treat the configured ceiling as a scheduler ceiling with bounded per-search fees, not an external billing-account guarantee, whenever built-in search is enabled.

The 20-case pilot shape is:

pnpm policyc experiment \
  --cases eval/behavioral/pilot-v2.jsonl \
  --strategies full_policy,compiler_slice \
  --provider openai \
  --model gpt-5-mini-2025-08-07 \
  --samples 3 --concurrency 2 \
  --max-output-tokens 1024 --max-calls 120 \
  --max-cost-usd 0.55 --retries 0 \
  --run-label pilot-v2 \
  --output runs/openai-pilot-v2 --dry-run

Dataset discipline

Use development-v1.jsonl for iteration, freeze compiler and cases, then run a versioned evaluation set once. pilot-v2.jsonl is a development-informed pilot, not a held-out set; its preregistration and the rejected v1 audit live under eval/. Every manifest persists the split, version, and canonical hash; edits or relabeling cause validation failure. Export discovered failures to a separate development file rather than rewriting evaluated inputs. adversarial-template-v1.jsonl is intentionally non-executable until independently authored cases replace its template row.

Confirmed compiler 0.5 held-out failures were copied—not moved or edited—into eval/behavioral/compiler-v0.6-regression-v1.jsonl as a nine-case development-only regression set. Its 18-call compiler 0.6 smoke produced eight both-pass pairs and one compiler-only pass under strategy-blind grading, with no compiler regressions. This is targeted development evidence, never fresh held-out evidence; the complete audit is in eval/audits/compiler-v0.6-development-smoke.md.

Blinded grading

Every v2 run writes blind/grading-packets.json with opaque answer IDs and deterministically randomized answer order. It omits strategy names and token counts. The private mapping is stored separately as blind/answer-map.private.json. Manual grading requires no paid grader; any later model-grader result is evidence, not ground truth.

Create a smaller reviewer-safe adjudication bundle containing every automated critical regression plus deterministic both-pass and both-fail controls:

.venv/bin/policyc-runtime adjudication-bundle runs/<run-directory> \
  --agreements-per-class 10

The reviewer folder contains enriched critical obligations, opaque answer IDs, and a fillable grade template, but no strategy names, token counts, or selection labels. A hash-linked private selection file stays beside the existing private answer map and must not be shown to the reviewer until grading is final.

To create an exhaustive reviewer-safe bundle containing every complete pair, add --all-complete. Lock and commit the completed anonymous grade hashes before opening the private strategy map.

If targeted controls show that the automated agreement strata are unreliable, create a second bundle containing every remaining complete pair without duplicating reviewed packets:

.venv/bin/policyc-runtime adjudication-completion-bundle runs/<run-directory> \
  --prior-bundle runs/<run-directory>/blind/adjudication-v1 \
  --completed-grades /path/to/grades.completed.json

Persistent run catalog

PolicyC maintains a local SQLite catalog at .policyc/catalog.sqlite. It records run status, Git commit and dirty state, manifest and dataset hashes, model, trial counts, calls, token usage, costs, failures, and paths to the authoritative run files.

pnpm policyc runs list
pnpm policyc runs list --json
pnpm policyc runs show <run-id>
pnpm policyc runs rebuild --root runs

The database persists across process restarts and is ignored by Git. Back up .policyc/catalog.sqlite together with runs/ if you want the history on another machine. The run directories remain the scientific source of truth; if the SQLite file is lost, runs rebuild recreates the catalog from their manifests, trials, budgets, and reports.

Repeated invocations with the same experiment configuration and output directory reuse the same run ID and completed trials. Reusing an output directory with an incompatible configuration is rejected instead of silently starting a different run.

Completed responses can be regraded offline after a versioned evaluator correction without changing source evidence or making provider calls:

.venv/bin/python -m policyc_runtime.regrade runs/<run-directory>

Evaluator 2.4 recognizes semantically equivalent unavailable-source disclosures and distinguishes visual references to an image “background” from first-person promises of asynchronous work. Derived regrades remain separate and hash-linked; original manifests, trials, raw responses, evaluations, and reports are immutable.

The command writes a hash-linked addendum, derived report, and derived evaluations under derived/evaluator-<version>/. Original manifests, raw responses, trials, evaluations, and reports remain unchanged. Incomplete trials remain failures.

Behavioral evaluator 2.3 applies independently testable universal obligations to every case, including no background-work claims, no hidden-reasoning disclosure, no raw tool traces, and no claimed search/tool action without a recorded tool call. Compiled artifacts also carry explicit tool availability: unavailable required tools become a brief limitation and may never be simulated.

Terminal provider failures are preserved and never silently replayed. After correcting an external problem such as unavailable quota, start a distinct attempt with a new output directory and explicit identity, for example --run-label smoke-2. This retains the failed run and creates a separate catalog record.

Verification

pnpm typecheck
pnpm test
pnpm graph:validate

pnpm py:format
pnpm py:lint
pnpm py:typecheck
pnpm py:test

pnpm test:all

CI installs both environments, builds TypeScript, validates the graph and cross-language artifacts, runs all offline tests, and executes the fake-provider demo without credentials.

Token accounting

Modern OpenAI-family models use exact o200k_base BPE counts through gpt-tokenizer. Unsupported model names use the documented whitespace estimator ceil(words / 0.75). Every artifact records tokens, method, tokenizer, and model; reports never silently mix estimated and exact values.

Persistence layout

experiment/
  manifest.json                 # input manifest
  artifacts/*.json              # compiler candidates
  manifest.canonical.json       # normalized run configuration
  environment.json              # environment and reproduction command
  events.jsonl                  # ordered run events
  trials/<trial-id>.json        # atomic individual results
  report.json                   # aggregate comparison

V2 runs additionally contain raw/<trial>/attempt-*.json, parsed provider/*.json, independent evaluations/*.json, budget.json, and the two blind-review files.

Known limitations

• Policy nodes and dependency edges are manually authored; prompt-to-IR extraction is not implemented.
• Intent detection is lexical and can miss paraphrases.
• The existing 157-case selector evaluation is repository-authored, not held out.
• The deterministic evaluator uses regular expressions and is not a complete judge of model behavior.
• The 20-case pilot set is small and repository-authored; it is not held out and does not provide broad external validity.
• The fake provider validates scheduling and reproducibility, not policy preservation.
• The OpenAI adapter is contract-tested against mocked documented shapes but has not yet been verified by a live call.
• Network ambiguity prevents guaranteed exactly-once billing; PolicyC stops by default and surfaces the exposure.
• Real-provider results require repeated trials and careful interpretation; no single run proves equivalence.
• The API is an in-process research service, not a distributed production control plane.