Dispatch Triage

Programmable triage for GitHub.

AI provides the signal. Your rules decide the action.

Documentation · Quick start · Configuration · Cookbook · Contributing

---

Dispatch turns probabilistic model output into controlled, auditable GitHub automation.

It asks a model typed, atomic questions about an issue or pull request — is this a bug, does it have a reproduction, how risky is this diff — and gets back answers with calibrated probabilities. What happens next is decided by rules you write, in a file you own, in your repository.

The model never picks an action. It only answers questions.

AI should provide the signal. Maintainers should own the policy.

---

How it works

flowchart TD
    A[GitHub event] --> B[Atomic questions]
    B --> C[Model: probabilities]
    C --> D[Your rules: policy]
    D --> E[Action plan]
    E --> F[Execute + decision record]

    style C fill:#2d333b,stroke:#539bf5,color:#adbac7
    style D fill:#2d333b,stroke:#57ab5a,color:#adbac7

Loading

Three systems, three responsibilities, and a hard line between them:

	Responsibility
Model	observations and probabilities
Maintainer	policy — the rules and thresholds
Dispatch	execution and the audit trail

---

Why Dispatch

Most GitHub automation sits at one of two extremes.

Hardcoded rules are predictable but blind. A regex cannot tell you whether an issue contains a usable reproduction, or whether a diff touches something security-sensitive.

A model wired directly to actions can tell you those things, but it decides what to do with them. That brings problems that are hard to engineer away:

• generated prose is difficult to review before it is posted
• model-to-action systems are difficult to audit after the fact
• behaviour shifts when the prompt, the model, or the input distribution changes
• the maintainer's policy lives inside a prompt instead of in the repository

Dispatch puts a boundary in the middle. The model is asked narrow questions and returns probabilities. Your rules read those probabilities and decide. Dispatch executes the result and writes down everything it considered.

That boundary is what makes the interesting properties possible: a shadow mode that decides without writing, a replay that re-triages two hundred historical issues offline, and a calibration report that checks whether a claimed 80% confidence is actually right 80% of the time.

---

Example

A real recorded decision — a pull request from jspdf-md-renderer, captured by dispatch record and shipped as a test fixture.

Input — title, body, changed paths, a truncated diff, and the repository brief.

Add opt-in security controls and update documentation

Ten questions, one request — the default pull-request set, unmodified:

Question	Type	Answer	Confidence
change_type	choice	feat	100%
semver_bump	choice	minor	91%
label_area	choice	docs	82%
risk	score	3.00	100%
description_quality	score	1.49	50%
touches_security_surface	noul	0.90	80%
needs_tests	noul	0.83	66%
breaking_change	noul	0.37	26%
matches_linked_issue	noul	0.06	88%
low_effort	noul	0.04	92%

Your rule reads them:

- id: security-review
  when: [pull_request.opened, pull_request.synchronize]
  if: 'answers.touches_security_surface > 0.7 && answers.risk >= 3'
  then:
    - { op: label, add: [security-review] }
    - { op: request_review, codeowners: true }

Result: 8,167 input tokens, one provider request, $0.000343.

Note description_quality at 50% confidence. A rule gated on it would be suggested rather than applied, because a rule is only as trustworthy as its least certain input. That is not a special case — it is the gate ladder doing its job.

---

Core concepts

Concept	What it means
Atomic question	One typed question with one answer: yes/no (noul), pick-one (choice), or a rubric score. Never "analyse this PR".
Probabilistic answer	Every answer carries a calibrated probability. Across many predictions at 0.8, about 80% should be right.
Rule	Your policy. A when trigger, an if expression over answers.*, and a then list of actions.
triage()	Pure planning. Produces a decision. Performs no platform writes, ever.
execute()	The side-effect boundary. Every write to GitHub happens here and nowhere else.
Decision record	The persisted result: answers, probabilities, which rules fired, what was applied, what was suppressed and why. Written before execution.
Gate ladder	Resolves each planned action to apply, suggest, or suppress. First match wins, twelve rungs, fully deterministic.
Owned labels	Dispatch tracks the labels it added and removes only those. A label you applied by hand is never touched.
Shadow mode	Decide, log, write nothing. The default for a new install.
Provider	The port the model sits behind. The engine has no model SDK in it.
Replay / eval	Re-triage history offline from recorded fixtures, then measure calibration and accuracy against it.

---

Modes

Dispatch is meant to earn permission, not ask for it up front. The modes are enforced in the engine — rung 10 and 11 of the gate ladder — not merely documented.

Mode	What it does	What earns the next step
shadow	Decides and logs. Zero comments, zero labels.	You read the decision log and agree with it.
suggest	One bot comment proposing labels and flags. Applies nothing.	You stop disagreeing with the proposals.
auto	Applies labels, asks for reproductions, routes reviews.	The eval report shows calibration holds.

A new install starts in shadow. Destructive operations — close, lock, minimize, convert-to-discussion — additionally require naming them in allowDestructive, and no shipped default includes one.

---

Quick start

npx @dispatch-triage/cli init

This scaffolds .github/dispatch.yml and a workflow. Then re-triage your own history and read what it would have done:

npx @dispatch-triage/cli replay --repo your-org/your-repo --last 200

replay never writes to GitHub. There is no code path in it that could.

Each decision prints as a plan you can argue with:

acme/widgets#412  issue.opened  mode=shadow  key=8f2a1c04

  question                      answer      confidence
  has_repro                     0.12        ########..  76%
  is_spam                       0.02        ##########  96%
  severity                      1.60        #######...  72%
  type                          bug         ########..  84%

  Will apply
    (nothing)

  Suppressed
    x label: add needs-repro    [needs-repro] shadow_mode
        Shadow mode decides and logs but writes nothing.
    x comment: needs-repro.md   [needs-repro] shadow_mode
        Shadow mode decides and logs but writes nothing.

The suppressed list is the product, not diagnostics. It is what you read in shadow mode to decide whether you trust this, and what dispatch eval measures later.

---

Configuration

Everything lives in .github/dispatch.yml. Nothing Dispatch does should surprise someone who has read it.

version: 1
mode: shadow

confidence:
  auto: 0.85 # apply at or above this
  suggest: 0.60 # propose at or above this; below it, say nothing

# Label dimensions become questions. These descriptions are the rubric the model reads,
# so write them for someone who has never seen your repository.
labels:
  area:
    parser: 'Markdown tokenization and AST construction'
    renderer: 'PDF drawing, layout, pagination, fonts'

rules:
  - id: needs-repro
    when: [issue.opened, issue.edited]
    if: "answers.type == 'bug' && answers.has_repro < 0.3"
    then:
      - { op: label, add: [needs-repro] }
      - { op: comment, template: needs-repro.md, key: needs-repro }

dispatch validate reports every problem at once, with a file, line and column:

.github/dispatch.yml:14:3  error  References `has_reproduction`, which is not a question. (rules[1].if)
    Define it under `questions:`, or check the spelling.

Full schema: Configuration reference.

---

Safety by design

These are enforced by tests and by an import-boundary check in CI, not by convention.

1. triage() performs no platform writes. execute() performs all of them.
2. One provider request per event. Never a loop over questions.
3. The bot removes only labels it added, tracked cumulatively across decisions.
4. Destructive operations require explicit opt-in. No shipped default enables one.
5. All comment text comes from editable templates. The model cannot produce prose.
6. Every decision is persisted with full probabilities before anything executes.
7. At most one Dispatch comment per item, edited in place on re-runs.
8. Duplicates are never auto-closed — at any confidence, in any mode.

The model cannot invent an action. It answers the questions it was given, and the set of operations Dispatch can perform is fixed in code.

---

Provider architecture

The engine talks to a Provider port. It contains no model SDK, no HTTP client, and no vendor types — a CI check fails the build if that stops being true.

The initial provider is TypeSafe Jev / System One, a model that returns typed answers with calibrated probabilities and cannot generate text. Three things follow from that:

• Schema violations are impossible. The answer is always one of the options the question defined. No JSON parsing, no retry-on-malformed, no validation layer.
• It cannot write a comment. Every word Dispatch says comes from a template.
• The probabilities are calibrated, which is what makes a confidence threshold defensible rather than decorative.

Questions are evaluated in parallel and in isolation, so asking twenty costs barely more than asking one. Measured, not assumed:

Questions per request	p50	Input tokens
1	455 ms	466
12	478 ms	721
30	480 ms	1,153

Across the 20 pull requests and issues recorded as test fixtures, the median item used 7,550 input tokens and cost $0.00032; all twenty together came to $0.0053. Round trips take 479 ms at the median and 2.1 s at p99, measured from India. See the provider README for method.

Dispatch is not conceptually locked to one provider. @dispatch-triage/provider-llm is a placeholder for an ordinary-LLM adapter — not implemented, not published, and deliberately so: a provider whose probabilities are not calibrated would quietly undermine every threshold in every config.

---

Architecture

flowchart LR
    subgraph surfaces [Surfaces]
        CLI[cli]
        ACT[action]
    end
    subgraph engine [Engine]
        CORE[core]
    end
    subgraph adapters [Adapters]
        CFG[config]
        JEV[provider-jev]
        GH[github]
        TPL[templates]
        ST[store]
        RT[runtime]
    end
    surfaces --> CORE
    surfaces --> adapters
    CORE -.ports.-> adapters

Loading

core is a leaf: it imports no GitHub client, no model SDK, no filesystem and no network. Everything outside reaches it through a port. Deeper detail lives in the documentation, not here.

---

Packages

Package	Purpose
@dispatch-triage/core	The engine: questions, rules, the gate ladder, decisions. A leaf with no I/O.
@dispatch-triage/config	Schema, layered loader, and validation for dispatch.yml.
@dispatch-triage/provider-jev	The TypeSafe Jev provider, plus record/replay for offline tests.
@dispatch-triage/github	Payload normalizer, Octokit executor, CODEOWNERS resolution.
@dispatch-triage/templates	The Handlebars renderer and the default template pack.
@dispatch-triage/store	SQLite and JSON stores, held to one shared contract suite.
@dispatch-triage/runtime	The jexl evaluator, a redacting logger, terminal styling.
@dispatch-triage/dedup	Duplicate detection: retrieval, two-stage judging, reporting.
@dispatch-triage/eval	Calibration and accuracy: reliability diagrams, ECE, threshold recommendations.
@dispatch-triage/cli	init, validate, run, replay, record, eval, sync-labels.

---

Surfaces

Surface	Status
CLI — @dispatch-triage/cli	Available. Seven commands. Only record contacts the model provider.
GitHub Action — apps/action	Available. Runs from a committed bundle; CI enforces that the bundle matches its sources.
GitHub App — apps/app	Not yet available. The webhook, worker and Postgres store are written and tested, but there is no bootstrap or container image, so it cannot be deployed today.

# .github/workflows/triage.yml
- uses: JeelGajera/Dispatch/apps/action@v0.1.0
  with:
    jev-key: ${{ secrets.TYPESAFE_API_KEY }}

More: Action guide · CLI reference · Workflow recipes

---

Development

pnpm install
pnpm check

check runs formatting, the import-boundary check, build, lint, typecheck and the full test suite.

No test in this repository reaches the network or needs an API key — every one runs offline against recorded fixtures. The only live calls are a nightly smoke test that catches API drift, and scripts/measure-latency.mjs, which you run yourself.

---

Contributing

Bug reports, questions and pull requests are all welcome. CONTRIBUTING.md covers setup, the invariants that are load-bearing, and how to change a rule without surprising anyone.

You do not need an API key to work on Dispatch.

• Code of conduct
• Security policy — report vulnerabilities privately, never as an issue

---

Licence

MIT