Siftguard

A quality gate for open-source maintainers.

Problem

AI-assisted tooling has made it trivial to open issues and pull requests, and just as trivial to open bad ones — vague bug reports with no reproduction steps, hallucinated stack traces, templated "AI slop," and duplicates of things already reported. Maintainers absorb that cost one submission at a time: reading, guessing at what's missing, asking for clarification, waiting, often getting nothing back. The bottleneck isn't a lack of contributions; it's a lack of reviewable ones.

Solution

Siftguard is a GitHub App that sits between contributors and maintainers. It scores every incoming issue and pull request against a transparent, explainable rubric — reproducibility, environment completeness, stack-trace validity, AI-slop signal, duplicate similarity — and only asks a maintainer to look at submissions that clear the bar. Everything else gets labeled, explained, and sent back to the contributor with exactly what's missing, automatically.

Nothing happens silently. Every score comes with a one-line reason per check, every label and comment is logged to an audit trail, and a maintainer can always override the assessment.

What it does

• Scores every submission (0–100) on reproducibility, environment completeness, stack-trace validity against the actual repo, AI-slop signal, and duplicate similarity against prior issues/PRs.
• Posts a transparent breakdown as a comment, so both maintainer and contributor can see exactly why something was or wasn't flagged.
• Auto-labels and requests missing information, creating labels on demand so a repo works without any manual setup.
• Detects duplicates using embedding similarity, cached per repo so re-edits don't recompute the whole index.
• Dispatches reproduction attempts and git bisects into the target repo's own GitHub Actions — Actions provides the sandbox isolation; Siftguard never executes contributor code itself.
• Generates minimal reproducible examples and estimates security severity on demand, gated behind cheap heuristics so AI calls only happen when they're actually needed.
• Tracks contributor history and repo-level analytics — submission volume, pass/flag rates, duplicate catch rate — surfaced on a review dashboard.
• Classifies issue type (bug, feature request, question, documentation) and groups merged PRs into a categorized changelog on release.
• Brings your own Claude key per installation through a settings page tied to the GitHub App installation itself — no separate accounts, no OAuth.

How it's put together

src/
  config.ts               Loads and merges .siftguard.yml per repo
  crypto.ts                AES-256-GCM encryption for stored API keys
  keys.ts                   Resolves which Claude key to use per installation

  scoring/
    heuristics.ts            Reproducibility, environment, stack-trace checks
    embeddings.ts             Real or hashed-vector embeddings for similarity
    duplicates.ts              Duplicate detection against prior submissions
    claude.ts                   AI-slop classification and maintainer summaries
    severity.ts                  Security severity estimation
    triage.ts                     Issue type classification
    mre.ts                          Minimal reproducible example generation
    index.ts                         Composite scorer

  db/
    types.ts                 Storage backend interface
    sqlite.ts                  Zero-setup local backend
    postgres.ts                  Postgres + pgvector backend
    index.ts                       Backend selection

  github/
    comment.ts               Score-breakdown comment formatter
    labels.ts                  Label creation and application
    actions.ts                   Applies labels/comments, writes the audit log

  repro/
    dispatch.ts               Dispatches reproduction and bisect workflows

  changelog/
    generate.ts               Groups merged PRs into a categorized changelog

  handlers/
    issues.ts                 issues.opened / issues.edited
    pullRequests.ts             pull_request.opened / synchronize
    comments.ts                   issue_comment.created (slash commands)
    workflowRun.ts                  workflow_run.completed
    release.ts                        release.published

  settings/
    router.ts                 Per-installation Claude key settings page

  dashboard/
    router.ts                 Review queue, analytics, contributor history

  index.ts                   App entrypoint, wires handlers and routers

templates/
  siftguard-repro.yml.example   Reference Actions workflow for reproduction

sql/
  schema.sql                  Postgres + pgvector reference schema

app.yml                      GitHub App manifest

Configuration

Behavior is controlled per repo through a .siftguard.yml file — threshold, label names, which checks run automatically, and which contributors to ignore. See .siftguard.yml.example for the full reference.

Security

• Webhook signatures are verified on every request.
• Contributor-supplied code is never executed by Siftguard itself; reproduction and bisect runs happen inside the target repo's own GitHub Actions.
• Per-installation API keys are encrypted at rest.
• Only what's needed for scoring and duplicate detection is retained.

License

MIT