This is a staged target architecture. It is not implemented.
RunSift should grow only when product evidence requires it. The first useful system is a local CLI and optional read-only collector, not a distributed platform.
offline fixture | downloaded bundle | read-only GitHub collector
→ input validation and normalization
→ secret redaction
→ failure-window and structured-evidence extraction
→ deterministic classification rules
→ decision and abstention policy
→ terminal, Markdown, and JSON reports
Required modules:
- configuration and input schema validation;
- bounded GitHub Actions read-only collection;
- untrusted-input and path-safety controls;
- secret redaction;
- step and error-window extraction;
- structured evidence model;
- deterministic category rules;
- evidence ranking;
- rule-score and abstention policy;
- human-readable and machine-readable reports;
- fixture and golden-test harness.
No database, server, webhook, queue, or GitHub App is required.
The normative command, input, output, fixture, and security contracts are defined in the CLI MVP contract.
reviewed failures
→ append-only label ledger
→ validated dataset snapshot
→ repository-grouped and temporal splits
→ rules and classical ML candidates
→ calibration and selective policy
→ reproducible evaluation report
Use simple local artifacts first:
- JSONL label ledger;
- Parquet or JSONL snapshots;
- snapshot manifest with hashes;
- reproducible training CLI;
- versioned evaluation reports and model cards.
Do not add a model registry before multiple credible model versions exist.
GitHub webhook
→ signature verification and idempotency
→ bounded job ingestion
→ redact and parse
→ rules/model policy
→ GitHub Check summary
→ explicit feedback and rerun reconciliation
This stage begins only after the CLI workflow demonstrates repeated value.
A practical beta may use:
- FastAPI ingress;
- one worker process;
- PostgreSQL for events, predictions, labels, idempotency, and work leasing;
- local or S3-compatible redacted artifact storage.
The webhook request path must acknowledge quickly and keep log processing outside the request handler.
champion prediction
+ candidate shadow prediction
→ delayed reviewed outcomes
→ comparison report
→ manual promotion or rejection
Promotion must be blocked when:
- schema compatibility fails;
- high-risk precision falls below policy;
- calibration or selective coverage regresses materially;
- repository, runner, or workflow segments degrade;
- evidence quality regresses;
- system reliability guardrails fail.
Rollback must restore both the application version and the approved model reference.
Fetches only selected metadata, jobs, annotations, and logs. Enforces bounded retries, rate-limit behavior, least privilege, and idempotent local storage.
Rejects unsupported encodings, oversized inputs, path traversal, invalid manifests, and inconsistent hashes. Produces deterministic normalized text while preserving source references.
Runs before any RunSift-owned cache, report, dataset, or telemetry write. Produces a redaction report without retaining detected secret values.
Creates versioned failure windows and structured evidence. Parser failure must degrade to generic evidence extraction or unknown, not silently produce an empty successful result.
Handles deterministic signatures such as dependency resolution, test assertion, lint/type-check, build/compile, permissions/environment, workflow configuration, network, registry, and rate-limit failures.
Rules produce documented match strength, not calibrated probability.
Handles ambiguous combinations that rules do not adequately cover. The initial candidate should be a calibrated linear model, not an LLM. It remains outside the user-visible path until it improves reviewed natural-data results.
Combines rule matches or model probabilities with evidence quality and risk thresholds. Policy owns abstention, decision source, temporal-assessment constraints, and recommended actions.
Keeps explicit maintainer corrections separate from heuristic or rerun-derived labels. Corrections supersede previous labels without deleting history.
- verify webhook signatures when webhooks eventually exist;
- use least-privilege GitHub credentials;
- never persist or print tokens;
- treat logs, annotations, paths, and metadata as untrusted input;
- redact before RunSift-owned persistence;
- apply compressed and uncompressed size limits;
- never execute commands found in logs;
- escape active content in Markdown and terminal renderers;
- prevent symlink and path traversal outside configured roots;
- isolate data by repository installation in service mode;
- support retention and deletion;
- keep private logs out of telemetry and public artifacts.
- GitHub App and webhooks before CLI validation;
- Kubernetes;
- Kafka;
- microservices;
- feature store;
- distributed training;
- LLM decision path;
- automatic remediation;
- multiple CI providers;
- custom dashboard before a GitHub Check proves useful.