Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions .github/ISSUE_TEMPLATE/provider_request.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
---
name: New model provider
about: Request or propose a backend for `context-report run` / `judge`
labels: enhancement
---

**Provider name** (as it would appear in `PROVIDERS`, e.g. `bedrock`):

**Does it speak the OpenAI chat-completions shape?**
<!-- If yes, `openai-compatible` + `--base-url` may already cover it -- say why it doesn't. -->

**How is a prompt sent and the answer read back?**
<!-- HTTP endpoint + request/response shape, or a local CLI's argv and stdout, as
`OpenAICompatibleAsker`/`CliAsker` show in `src/context_report/efficacy/`. -->

**Auth** (API key env var, local login, none):

**Does it report token usage?** (input/output/cached -- `last_usage`, if so, in what shape)

**Does it report which model actually answered?** (`last_model`, e.g. an alias resolving to a
pinned version)

**Any field `ModelRef` doesn't have yet** (like `openai-compatible`'s `baseUrl`/`apiKeyEnv`):

**Documentation link:**
36 changes: 36 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,37 @@ You do **not** need to write Python to make a real difference here:
ambiguous `basis` case, a `NotApplicable` reason that isn't actionable.
- **Ideas & feedback** — start a [Discussion](https://github.com/open-coder-ai/context-report/discussions). Telling us what's confusing *is* a contribution.

## The ladder

Contributions here get larger in one direction, and you can stop at any rung:

1. **An evidence report** — run a probe, paste what actually happened. No code, and it is the
most useful thing a newcomer can do, because a claim nobody re-ran is just a claim. In this
repo that rung is not an analogy: an evidence report *is* the product. Run
`context-report produce --subject <path> --kind plugin --target claude_code --n 20 --out
report.json` against any public plugin, then `context-report verify report.json --subject
<path>` — the README's whole 30-second quickstart — and open the resulting `report.json` as an
issue or a PR. That is rung 1, done.
2. **An eval case** — an input that should be caught, or should not be, with the expected
verdict. This is how a guard stops regressing.
3. **A policy** — a rule plus the mechanism that enforces it, honestly labelled as enforced or
advisory.
4. **An adapter** — support for one more agent, matched to what that agent's hooks can really do.
Here that means one more target entry in `src/context_report/data/payloads-v0.1.json` and
`bundle-layout-v0.1.json` (a target agent's plugin-bundle layout), or one more `Asker` backend
registered in `PROVIDERS` in `src/context_report/run/runner.py` (a model `run` and `judge` can
actually reach).
5. **Review** — reading someone else's evidence and saying whether it holds.

**Becoming a maintainer:** three merged pull requests earns triage rights — labelling, closing
duplicates, and asking for the evidence a report is missing. Nobody is asked to commit to more
than they want to.

This project runs its own chock policies (`.chock/`, the same `protect-main-branch` rule
dogfooded below) plus DCO sign-off, on every pull request, and that pair is the filter for
low-effort machine-generated contributions — not a human gatekeeper. A PR that cannot say what it
checked will not pass, whoever or whatever wrote it.

## Development setup

```bash
Expand Down Expand Up @@ -164,6 +195,11 @@ Browse issues by label to find your entry point:
- `bug` — confirmed defects.
- `documentation` — no code required.

The [open, seeded good-first-issue
list](https://github.com/open-coder-ai/context-report/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22)
is maintained separately from this file — expect the kind of thing in "Good first contributions"
below: a missing target's payload shape, a citation to source, a `NotAvailable` row worth closing.

Comment on an issue to claim it — we'll assign it to you so no one double-works.

## Good first contributions
Expand Down
58 changes: 58 additions & 0 deletions llms.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
# context-report

> An open, signed report format for whether an agent context artifact — a plugin, an
> `AGENTS.md`, a skill, a hook, an MCP server — actually works.

`context-report` is a predicate an author's CI produces and a catalog verifies at submission:
one row per fact (`reachability`, `cost`, `fault.*`, `interference`, `efficacy`, ...), each
declaring its `basis` as `re-derivable` (anyone can recompute it from the subject) or `claimed`
(the author asserts it) — never a single pass/fail for the artifact as a whole. `produce` runs the
measurement, `verify` reprints the report and checks its hashes, and `run`/`compare` drive whole
manifests of subjects × models × tasks over time.

License: Apache-2.0. Status: pre-1.0 (`v0.1`) — the schema, CLI surface, and row set can change
between MINOR releases; see the Compatibility note in the spec READMEs below.

## Docs

- [README](https://github.com/open-coder-ai/context-report/blob/main/README.md): the problem,
the 30-second quickstart, a real report's anatomy, and the three headline findings from the
measurement paper.
- [CONTRIBUTING](https://github.com/open-coder-ai/context-report/blob/main/CONTRIBUTING.md): the
contribution ladder from an evidence report to review, development setup, branch/PR workflow,
and the DCO + chock policy pair every PR must pass.
- [CONVENTIONS](https://github.com/open-coder-ai/context-report/blob/main/CONVENTIONS.md): house
style — docstrings, templating (`__TOKEN__` placeholders, never f-strings), where facts about
the world live versus code.
- [AGENTS](https://github.com/open-coder-ai/context-report/blob/main/AGENTS.md): the same
conventions distilled for an agent working in this repo.
- [SECURITY](https://github.com/open-coder-ai/context-report/blob/main/SECURITY.md): supported
versions and how to report a vulnerability.
- [GOVERNANCE](https://github.com/open-coder-ai/context-report/blob/main/GOVERNANCE.md): solo
maintainer, decision-making, and the succession plan if the maintainer disappears.
- [CHANGELOG](https://github.com/open-coder-ai/context-report/blob/main/CHANGELOG.md): every
user-visible behavior change, one entry each.

## Spec

- [attestation/v0.1 README](https://github.com/open-coder-ai/context-report/blob/main/spec/attestation/v0.1/README.md):
normative spec for the report predicate itself — what each field means, what a producer or
verifier MUST/SHOULD/MAY do; the schema at `schema.json` is authoritative over this prose.
- [run/v0.1 README](https://github.com/open-coder-ai/context-report/blob/main/spec/run/v0.1/README.md):
normative spec for the `run` manifest — subjects, target agent, subject models, tasks, arms.

## CLI and library

- [docs/cli.md](https://github.com/open-coder-ai/context-report/blob/main/docs/cli.md): `run` and
`compare` — manifest shape, `--dry-run`/`--n`/`--resume`, and the three model providers
(`anthropic`, `claude-cli`, `openai-compatible`) a manifest can reach.
- [docs/library.md](https://github.com/open-coder-ai/context-report/blob/main/docs/library.md):
using `context-report` as a library — the public function surface a catalog or CI job can call
directly instead of shelling out.

## Method

- [paper/context-report.md](https://github.com/open-coder-ai/context-report/blob/main/paper/context-report.md):
the measurement paper — method, the three findings from running the reference producer over
chock's bundles, 18 public plugins, and seven instruction files/skills, and what v0.1 still
cannot measure.