Skip to content

feat(reconciliation): cross-source failure injection, chaos timeline DSL, and escalation runbook (C03) - #21

Closed
kapustazh wants to merge 2 commits into
developfrom
milestone/c03-failure-injection
Closed

kapustazh wants to merge 2 commits into
developfrom
milestone/c03-failure-injection

Conversation

@kapustazh

Copy link
Copy Markdown
Collaborator

Summary

Implements Coder C Milestone C03: Cross-Source Failure Injection.
Builds a deterministic chaos harness, failure timeline DSL (BEFORE_SUBMISSION, POSSIBLY_SUBMITTED, CONFIRMED), 16-scenario catalog, Subgraph MCP degradation suite, provider/RPC contradiction matrix, restart replay idempotency, UNKNOWN aging evaluation, and an operator escalation runbook.

Key guarantees:

  • In 100% of failure, crash, lost-response, degradation, and contradiction scenarios: externalSubmissionCount is exactly 0 and settlementPermission is 'NEVER'.
  • Escalation runbook strictly enforces that operators must NEVER "just retry" or treat lease expiration as permission to submit payments.
  • All scenarios execute deterministically with recorded seeds and zero external network/credentials.

Scope and acceptance criteria

  • The change is limited to the stated milestone or issue.
  • Acceptance criteria are listed and satisfied:
    • C03.1 — Failure timeline DSL: injection points (BEFORE_SUBMISSION, POSSIBLY_SUBMITTED, CONFIRMED), process kills, timeouts, disconnects, response loss, delayed evidence, and restarts with deterministic seeds.
    • C03.2 — Graph & Subgraph MCP degradation: empty results, lagging chain heads, indexing errors, wrong tool/deployment, oversized results, prompt injection. Zero permission change (NEVER).
    • C03.3 — Provider/RPC contradiction suite: Privy status vs Arc status matrix; recipient, token, and amount mismatches hold/escalate safely.
    • C03.4 — Restart/evidence replay: identical commands and view disposition on replay with recorded seed.
    • C03.5 — Agent failure, UNKNOWN aging, and escalation: fail-closed to WAIT on unsupported actions or fabricated bindings; age buckets (<5m FRESH, 5-60m STALE, >60m CRITICAL); strict no-blind-retry runbook.
  • No unrelated cleanup is included.

Product and security invariants

  • Tenant isolation remains fail-closed.
  • Sponsor authorization, auditability, and daily caps remain enforced where applicable.
  • Recipients cannot modify sponsor controls or access sponsor-only data.
  • No secret, token, production identifier, or personal data is committed or pasted into review prompts.
  • Invariant notes: 1 business intent -> at most 1 committed settlement. UNKNOWN state reconciles without blind retries. Uncertainty never turns into settlement permission.

Validation

Commands and results:

pnpm typecheck: PASS (TypeScript 6.0.3 strict build)
pnpm lint: PASS (ESLint 10.10.0 strict rules)
pnpm format:check: PASS (Prettier code style)
pnpm test: PASS (10 test suites, 101 tests passing)
npx markdownlint-cli2: PASS (0 issues across 64 files)

Independent review evidence

Gate A — exact candidate tree before push

  • Base commit SHA: 6dd2e37

  • Candidate tree SHA: 47cb9bc35f9b269008d1e5abc6a8119d3af1128d

  • Candidate commit SHA: 7b6e112

  • Reviewer tool: free-pi-cli

  • Reviewer model: glm 5.3

  • Verdict: VERDICT: PASS

  • Findings or residual risks: None.

  • The reviewed tree equals the committed tree.

Gate B — exact remote PR head

  • Pull request URL/number: Pending PR creation

  • Remote head commit SHA: 7b6e112

  • Remote head tree SHA: 47cb9bc35f9b269008d1e5abc6a8119d3af1128d

  • Reviewer tool: free-pi-cli

  • Reviewer model: glm 5.3

  • Verdict: Pending required CI and Gate B review

  • Findings or residual risks: Pending Gate B

  • Gate B reviewed the current remote head and matches Gate A's approved tree, or a fresh Gate A was run for the changed tree.

  • Agent policy / repository-policy and all applicable CI checks pass.

Risk and rollback

  • Residual risks: Live credentials and production Subgraph/model endpoints remain unadmitted in C03 as planned; qualification occurs under C04/C06.
  • Rollback or recovery plan: Revert PR commit 7b6e112 on develop.

Human merge

  • A human owner has reviewed the evidence and will perform the merge.

@kapustazh

Copy link
Copy Markdown
Collaborator Author

Closed: opened in error; this lane belongs to Coder C.

@kapustazh kapustazh closed this Sep 7, 2026
@kapustazh
kapustazh deleted the milestone/c03-failure-injection branch September 7, 2026 16:28
@kapustazh
kapustazh restored the milestone/c03-failure-injection branch September 7, 2026 16:30
@kapustazh kapustazh reopened this Sep 7, 2026
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Sep 7, 2026 •

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
❌ Deployment failed
View logs
oneshot 7b6e112 Sep 07 2026, 04:32 PM

@SuPuHe SuPuHe closed this Sep 7, 2026
@SuPuHe
SuPuHe deleted the milestone/c03-failure-injection branch September 7, 2026 21:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants