CLI MVP Contract

Status: contracted, not implemented.

This document is the implementation contract for the first production-quality RunSift slice. Implementation issues should reference this file when deciding scope or acceptance criteria.

Goals

The CLI must let a maintainer:

reproduce analysis without network access;
analyze a downloaded GitHub Actions job log;
optionally collect selected run and job evidence through the GitHub API;
receive an evidence-backed result in terminal, Markdown, or JSON form;
distinguish a recognized rule match from an abstention;
run the same fixtures in local development and CI.

Supported environment

The initial support boundary is intentionally narrow:

GitHub Actions;
Ubuntu-hosted jobs;
Python-oriented workflows;
pytest, package installation, lint/type-check, and Python build failures;
UTF-8 text logs up to a configured size limit;
one failed job attempt per analysis request;
optional related-attempt metadata when temporal evidence is available.

Unsupported inputs must return a safe validation error or unknown; they must not be silently coerced into a supported category.

Input modes

1. Offline fixture mode

runsift analyze --fixture dependency-resolution

This mode is mandatory for development, CI, demos, and deterministic regression tests. It performs no network access.

2. Local file mode

runsift analyze \
  --log job.log \
  --metadata job.json \
  --format terminal

The user supplies files already present on the local machine. RunSift must not copy the raw log outside the selected working/output directory.

3. Read-only GitHub mode

runsift collect --repo owner/name --run-id 123456789
runsift analyze --input .runsift/runs/123456789 --format json

The collector may fetch workflow-run metadata, jobs, steps, annotations, and selected job logs. It must not rerun workflows, dispatch workflows, modify checks, comment on pull requests, or write repository content.

Authentication and permissions

Read-only GitHub mode accepts a token through an environment variable such as GITHUB_TOKEN or an explicit token-command integration added later.

Requirements:

never accept tokens as ordinary positional arguments;
never print or persist token values;
public repository access should work without broader permissions where GitHub permits it;
private repository access requires only the minimum metadata/actions read permissions needed for the selected repository;
authentication failures must identify the missing capability without echoing credentials;
offline and local file modes must work without authentication.

A GitHub App is out of scope for this contract.

Planned command surface

runsift analyze     Analyze a fixture, local input bundle, or collected run
runsift collect     Fetch selected read-only GitHub Actions evidence
runsift fixtures    List and validate bundled fixtures
runsift version     Print CLI and output-schema versions

The CLI must provide --help, deterministic exit codes, structured errors, and a documented configuration precedence order.

Input bundle

A normalized analysis bundle contains:

input/
├── manifest.json
├── job.log
└── annotations.json        # optional

Minimum manifest.json fields:

{
  "schema_version": 1,
  "source": "fixture",
  "repository": "example/project",
  "workflow_name": "ci",
  "run_id": "123456789",
  "run_attempt": 1,
  "job_id": "987654321",
  "job_name": "tests-py311",
  "runner_os": "Linux",
  "conclusion": "failure",
  "started_at": "2026-07-17T10:00:00Z",
  "completed_at": "2026-07-17T10:04:12Z",
  "log_sha256": "hex-encoded-sha256"
}

Fields unavailable in local mode may be null, but schema version, source, conclusion, and log hash are required.

Fixture strategy

Each fixture represents one known scenario:

tests/fixtures/<fixture-id>/
├── manifest.json
├── job.log
├── expected.json
├── README.md
└── provenance.json

Fixture requirements:

sanitized and free of live secrets, tokens, private URLs, and personal data;
deterministic expected evidence offsets after normalization;
explicit provenance: synthetic, public-derived, or consented-private-derived;
license and redistribution status recorded;
raw private logs must never be committed;
separate positive, ambiguous, unsupported, and redaction cases;
mutation or negative cases for patterns that could cause false positives.

Initial fixture families:

dependency resolution or installation;
pytest assertion/test collection;
lint or type-check;
build or compile;
missing environment, permissions, or secret configuration;
network, registry, API, or rate-limit failure;
malformed workflow/configuration evidence;
ambiguous and unsupported logs;
secret-redaction cases.

Classification contract

Primary categories describe the most likely immediate failure cause:

dependency
test
lint_typecheck
build_compile
environment_permissions
network_rate_limit
workflow_configuration
unknown

Temporal behavior is represented separately:

not_assessed
suspected_transient
confirmed_recurring
maintainer_confirmed_flaky

A single failed attempt must not produce maintainer_confirmed_flaky. A transient or flaky assessment requires comparable prior attempts, rerun evidence, a known infrastructure signature, or explicit maintainer confirmation.

Decision and score semantics

Deterministic rules do not produce calibrated probabilities.

decision_source: "rule" uses a documented rule priority and evidence-quality score.
decision_source: "model" may expose a calibrated probability only after a validated model exists.
decision_source: "abstain" must include an abstention reason.
UI labels must not call a rule score “confidence.”

Every non-unknown result requires at least one redacted evidence item and the rule or model version that produced it.

JSON output

{
  "schema_version": 1,
  "tool_version": "0.1.0",
  "input": {
    "source": "fixture",
    "repository": "example/project",
    "run_id": "123456789",
    "job_id": "987654321",
    "log_sha256": "hex-encoded-sha256"
  },
  "result": {
    "category": "dependency",
    "temporal_assessment": "not_assessed",
    "decision": "review_required",
    "decision_source": "rule",
    "rule_id": "python.pip.no-matching-distribution",
    "rule_version": "1",
    "score": {
      "type": "rule_strength",
      "value": 0.9
    },
    "abstention_reason": null,
    "recommended_actions": [
      "Check package version constraints and the selected Python version."
    ]
  },
  "evidence": [
    {
      "source": "job.log",
      "line_start": 184,
      "line_end": 185,
      "step_name": "Install dependencies",
      "excerpt": "ERROR: No matching distribution found for example==9.9.9",
      "redacted": true
    }
  ],
  "warnings": []
}

The exact score scale must be documented and tested. Consumers must branch on decision, decision_source, and category—not on undocumented numeric thresholds.

Human-readable outputs

Terminal and Markdown reports must include:

category and decision;
clear uncertainty or abstention language;
failed workflow/job/step context when available;
the smallest useful redacted evidence snippets;
safe recommended actions;
links only when supplied by trusted metadata;
tool and schema versions for reproducibility.

Exit codes

Proposed stable exit codes:

0: analysis completed, including an unknown result;
2: invalid CLI usage or configuration;
3: unsupported or invalid input bundle;
4: authentication or authorization failure;
5: collection/network failure after bounded retries;
6: internal processing error.

A classified CI failure is data, not a CLI process failure; therefore it returns 0.

Data handling and security

Treat every log as untrusted input.
Never execute, interpolate into a shell, or render unescaped active content from logs.
Apply input-size, line-length, output-size, and decompression limits.
Redact before any RunSift-owned cache, report, dataset, or telemetry write.
Preserve source offsets through deterministic normalization.
Store only the minimum local artifact needed for the selected command.
Default to no telemetry.
Provide a documented cleanup command or directory-removal procedure.
Reject symlink/path traversal outside the configured input and output roots.

Non-goals

The MVP does not include:

automatic reruns or workflow dispatch;
test quarantine;
code or workflow modification;
generated patches;
a GitHub App or webhook service;
cloud persistence;
multi-repository analytics;
CI reliability dashboards;
machine-learning inference;
LLM-based decisions;
non-GitHub CI providers.

Acceptance criteria

The contract is implementation-ready when:

all supported input and authentication modes are documented;
at least one complete input and output fixture is specified;
categories and temporal assessments cannot be confused;
rule score semantics cannot be mistaken for calibrated probability;
safe error and exit behavior is explicit;
privacy, redaction, and untrusted-input boundaries are explicit;
implementation issues can reference stable sections of this document.

FilesExpand file tree

cli-contract.md

Latest commit

History