diff --git a/.github/ISSUE_TEMPLATE/provider_request.md b/.github/ISSUE_TEMPLATE/provider_request.md new file mode 100644 index 0000000..75fbcb5 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/provider_request.md @@ -0,0 +1,25 @@ +--- +name: New model provider +about: Request or propose a backend for `context-report run` / `judge` +labels: enhancement +--- + +**Provider name** (as it would appear in `PROVIDERS`, e.g. `bedrock`): + +**Does it speak the OpenAI chat-completions shape?** + + +**How is a prompt sent and the answer read back?** + + +**Auth** (API key env var, local login, none): + +**Does it report token usage?** (input/output/cached -- `last_usage`, if so, in what shape) + +**Does it report which model actually answered?** (`last_model`, e.g. an alias resolving to a +pinned version) + +**Any field `ModelRef` doesn't have yet** (like `openai-compatible`'s `baseUrl`/`apiKeyEnv`): + +**Documentation link:** diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index b4a4704..cde1247 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -14,6 +14,37 @@ You do **not** need to write Python to make a real difference here: ambiguous `basis` case, a `NotApplicable` reason that isn't actionable. - **Ideas & feedback** — start a [Discussion](https://github.com/open-coder-ai/context-report/discussions). Telling us what's confusing *is* a contribution. +## The ladder + +Contributions here get larger in one direction, and you can stop at any rung: + +1. **An evidence report** — run a probe, paste what actually happened. No code, and it is the + most useful thing a newcomer can do, because a claim nobody re-ran is just a claim. In this + repo that rung is not an analogy: an evidence report *is* the product. Run + `context-report produce --subject --kind plugin --target claude_code --n 20 --out + report.json` against any public plugin, then `context-report verify report.json --subject + ` — the README's whole 30-second quickstart — and open the resulting `report.json` as an + issue or a PR. That is rung 1, done. +2. **An eval case** — an input that should be caught, or should not be, with the expected + verdict. This is how a guard stops regressing. +3. **A policy** — a rule plus the mechanism that enforces it, honestly labelled as enforced or + advisory. +4. **An adapter** — support for one more agent, matched to what that agent's hooks can really do. + Here that means one more target entry in `src/context_report/data/payloads-v0.1.json` and + `bundle-layout-v0.1.json` (a target agent's plugin-bundle layout), or one more `Asker` backend + registered in `PROVIDERS` in `src/context_report/run/runner.py` (a model `run` and `judge` can + actually reach). +5. **Review** — reading someone else's evidence and saying whether it holds. + +**Becoming a maintainer:** three merged pull requests earns triage rights — labelling, closing +duplicates, and asking for the evidence a report is missing. Nobody is asked to commit to more +than they want to. + +This project runs its own chock policies (`.chock/`, the same `protect-main-branch` rule +dogfooded below) plus DCO sign-off, on every pull request, and that pair is the filter for +low-effort machine-generated contributions — not a human gatekeeper. A PR that cannot say what it +checked will not pass, whoever or whatever wrote it. + ## Development setup ```bash @@ -164,6 +195,11 @@ Browse issues by label to find your entry point: - `bug` — confirmed defects. - `documentation` — no code required. +The [open, seeded good-first-issue +list](https://github.com/open-coder-ai/context-report/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) +is maintained separately from this file — expect the kind of thing in "Good first contributions" +below: a missing target's payload shape, a citation to source, a `NotAvailable` row worth closing. + Comment on an issue to claim it — we'll assign it to you so no one double-works. ## Good first contributions diff --git a/llms.txt b/llms.txt new file mode 100644 index 0000000..1023345 --- /dev/null +++ b/llms.txt @@ -0,0 +1,58 @@ +# context-report + +> An open, signed report format for whether an agent context artifact — a plugin, an +> `AGENTS.md`, a skill, a hook, an MCP server — actually works. + +`context-report` is a predicate an author's CI produces and a catalog verifies at submission: +one row per fact (`reachability`, `cost`, `fault.*`, `interference`, `efficacy`, ...), each +declaring its `basis` as `re-derivable` (anyone can recompute it from the subject) or `claimed` +(the author asserts it) — never a single pass/fail for the artifact as a whole. `produce` runs the +measurement, `verify` reprints the report and checks its hashes, and `run`/`compare` drive whole +manifests of subjects × models × tasks over time. + +License: Apache-2.0. Status: pre-1.0 (`v0.1`) — the schema, CLI surface, and row set can change +between MINOR releases; see the Compatibility note in the spec READMEs below. + +## Docs + +- [README](https://github.com/open-coder-ai/context-report/blob/main/README.md): the problem, + the 30-second quickstart, a real report's anatomy, and the three headline findings from the + measurement paper. +- [CONTRIBUTING](https://github.com/open-coder-ai/context-report/blob/main/CONTRIBUTING.md): the + contribution ladder from an evidence report to review, development setup, branch/PR workflow, + and the DCO + chock policy pair every PR must pass. +- [CONVENTIONS](https://github.com/open-coder-ai/context-report/blob/main/CONVENTIONS.md): house + style — docstrings, templating (`__TOKEN__` placeholders, never f-strings), where facts about + the world live versus code. +- [AGENTS](https://github.com/open-coder-ai/context-report/blob/main/AGENTS.md): the same + conventions distilled for an agent working in this repo. +- [SECURITY](https://github.com/open-coder-ai/context-report/blob/main/SECURITY.md): supported + versions and how to report a vulnerability. +- [GOVERNANCE](https://github.com/open-coder-ai/context-report/blob/main/GOVERNANCE.md): solo + maintainer, decision-making, and the succession plan if the maintainer disappears. +- [CHANGELOG](https://github.com/open-coder-ai/context-report/blob/main/CHANGELOG.md): every + user-visible behavior change, one entry each. + +## Spec + +- [attestation/v0.1 README](https://github.com/open-coder-ai/context-report/blob/main/spec/attestation/v0.1/README.md): + normative spec for the report predicate itself — what each field means, what a producer or + verifier MUST/SHOULD/MAY do; the schema at `schema.json` is authoritative over this prose. +- [run/v0.1 README](https://github.com/open-coder-ai/context-report/blob/main/spec/run/v0.1/README.md): + normative spec for the `run` manifest — subjects, target agent, subject models, tasks, arms. + +## CLI and library + +- [docs/cli.md](https://github.com/open-coder-ai/context-report/blob/main/docs/cli.md): `run` and + `compare` — manifest shape, `--dry-run`/`--n`/`--resume`, and the three model providers + (`anthropic`, `claude-cli`, `openai-compatible`) a manifest can reach. +- [docs/library.md](https://github.com/open-coder-ai/context-report/blob/main/docs/library.md): + using `context-report` as a library — the public function surface a catalog or CI job can call + directly instead of shelling out. + +## Method + +- [paper/context-report.md](https://github.com/open-coder-ai/context-report/blob/main/paper/context-report.md): + the measurement paper — method, the three findings from running the reference producer over + chock's bundles, 18 public plugins, and seven instruction files/skills, and what v0.1 still + cannot measure.