Give mirror delivery a retry path, and record the creates it cannot confirm

packages/plugins/paperclip-plugin-github-mirror/README.md

@@ -35,12 +35,54 @@ plugin state.
 ## Behaviour under failure
 
 GitHub being down, rate-limiting, or rejecting the token must not stop Paperclip from
-processing events. Every handler is wrapped: failures are logged with a `retryable` flag
-(429, rate-limited 403, and 5xx are retryable; other 4xx are not) and the worker keeps
+processing events. Every handler is wrapped: failures are logged and the worker keeps
 running.
 
-Mirroring is idempotent — the GitHub issue number is stored in plugin state, so repeated
-events never create duplicates.
+**Retry.** Each write is attempted up to three times, with a jittered 1s/4s backoff, when
+GitHub asked us to back off (429, rate-limited 403), fell over (5xx), or the worker→host
+call timed out without an answer. Everything else — 401, 404, 422 — fails on the first
+attempt, because it will fail identically on the second. If GitHub named a wait via
+`retry-after` or `x-ratelimit-reset`, that wait is used instead of the backoff.
+
+**Timeouts.** The plugin does not set one, and cannot: `ctx.http.fetch` serializes only
+method, headers and body, so an `AbortSignal` never reaches the host. It does not need to.
+Each call is already bounded at 30s twice over — by the SDK's worker→host call timer and by
+the host's own `AbortController` — and the retry budget is capped at that same 30s so three
+attempts cannot occupy a handler for a minute and a half.
+
+**Duplicate creates.** Mirroring is idempotent: the GitHub issue number is stored in plugin
+state, so repeated events never create duplicates. Creating the issue is the one step that
+cannot simply be repeated, so it is written down first — see below.
+
+## The create outbox
+
+Event delivery is fire-and-forget. The host pushes events as a JSON-RPC notification and
+drops them outright when the worker is down; there is no replay. So if a create is
+interrupted between the POST and the write that records its number, nothing would ever
+mention it again — and the next event for that task would happily create a second GitHub
+issue.
+
+Before posting, the plugin upserts a `mirror-create` entity keyed `<companyId>:<issueId>`
+with status `pending`. On success it stores the number in plugin state and flips the record
+to `done`. A `pending` record therefore means exactly one thing: a create was attempted and
+we do not know how it ended.
+
+A scheduled job (`*/5 * * * *`) resolves those:
+
+| Record | Becomes | Why |
+|---|---|---|
+| `pending`, number known (in the record or in plugin state) | `done` | The create demonstrably succeeded; only the closing write was lost |
+| `pending`, no number, started under 2 minutes ago | unchanged | Could still be in flight |
+| `pending`, no number, older than that | `uncertain` | Logged once, and never attempted again |
+
+`uncertain` is terminal on purpose. The issue may or may not exist on GitHub, and finding
+out would mean reading GitHub back — which this plugin does not do, at all, by design. So
+it records the ambiguity where a human can see it instead of guessing. That trades a silent
+duplicate for a task that is visibly not mirrored, which is the lesser of the two.
+
+The outbox lives in `ctx.entities` rather than `ctx.state` because entities can be
+enumerated. Plugin state cannot: the host's state store has a `list`, but it is not exposed
+over the worker→host RPC, so a queue kept there could never find its own pending work.
 
 ## Development
 

packages/plugins/paperclip-plugin-github-mirror/src/constants.ts

@@ -9,6 +9,54 @@ export const STATE_KEYS = {
   lastStatus: "last-mirrored-status",
 } as const;
 
+/**
+ * Entity type for the create outbox — one record per Paperclip issue we have
+ * tried to mirror, written before the POST so an interrupted create leaves a
+ * trace instead of nothing.
+ *
+ * It lives in `ctx.entities` rather than `ctx.state` for one reason: entities
+ * can be enumerated (`ctx.entities.list`), and plugin state cannot. The state
+ * store does have a `list`, but it is not exposed over the worker→host RPC, so
+ * an outbox kept there could never find its own pending work.
+ */
+export const OUTBOX_ENTITY_TYPE = "mirror-create";
+
+/** Job key for the outbox drain, declared in the manifest. */
+export const OUTBOX_DRAIN_JOB = "drain-mirror-outbox";
+
+/**
+ * Statuses a create record moves through.
+ *
+ * `uncertain` is terminal and deliberately final: it means a create may or may
+ * not have reached GitHub and we have no way to find out — the mirror is
+ * write-only, so it will not go and look. Refusing forever turns what used to
+ * be a silent duplicate issue into one recorded fact a human can act on.
+ *
+ * `failed` is the case `uncertain` must not swallow. When GitHub answers with a
+ * status code — a 500, or a 429 that outlived the retry budget — the create
+ * definitively did not happen, so there is nothing to be uncertain about and no
+ * duplicate to fear. Those records stay retryable: a later event creates the
+ * issue. Collapsing them into `uncertain` would refuse forever on the strength
+ * of an answer that said "no", which is worse than the behaviour this file
+ * replaced, where a failed create was simply retried on the next event.
+ *
+ * The distinction is exactly whether GitHub replied. It did — `failed`. We
+ * never found out — `pending`, and `uncertain` once the grace window passes.
+ */
+export const OUTBOX_STATUS = {
+  pending: "pending",
+  done: "done",
+  uncertain: "uncertain",
+  failed: "failed",
+} as const;
+
+/**
+ * How long a `pending` record is left alone before the drain will call it
+ * `uncertain`. Comfortably past the 30s a single call can take, so the drain
+ * never condemns a create that is still in flight.
+ */
+export const OUTBOX_PENDING_GRACE_MS = 120_000;
+
 /**
  * Emitted by the escalation plugin when a task is handed to a human. Subscribed to
  * rather than reimplemented, so escalation policy stays in one place.

packages/plugins/paperclip-plugin-github-mirror/src/github.ts

@@ -3,16 +3,40 @@
  *
  * Deliberately write-only: the mirror never reads GitHub state back into
  * Paperclip, so there is no `get`/`list` here. Paperclip stays the store of
- * record; GitHub is a viewing surface.
+ * record; GitHub is a viewing surface. Retry does not change that: a failed
+ * write is tried again, never read back to find out what happened.
+ *
+ * ## The 30s budget
+ *
+ * A plugin cannot cancel its own request. `ctx.http.fetch` is a JSON-RPC call
+ * whose `init` is serialized down to method, headers and body, so an
+ * `AbortSignal` passed here would be silently dropped and a `Promise.race`
+ * around the await would only shorten the plugin's wait while the host socket
+ * kept running.
+ *
+ * It does not need one. Every call is already bounded twice at 30 seconds:
+ *
+ * - worker side, the SDK's `callHost` timer (`DEFAULT_RPC_TIMEOUT_MS` in
+ *   `worker-rpc-host.ts`; `runWorker` never passes `rpcTimeoutMs`), which
+ *   rejects with a JSON-RPC timeout;
+ * - host side, an `AbortController` armed with `PLUGIN_FETCH_TIMEOUT_MS`
+ *   (`plugin-host-services.ts`), which aborts the socket itself.
+ *
+ * So the ceiling is the runtime's, not ours, and the only thing this file owes
+ * it is that retrying stays inside it — see `RETRY_BUDGET_MS` in `retry.ts`.
  */
 
+import { withRetry, type RetryDeps } from "./retry.js";
+
 export interface GithubClientOptions {
   /** `owner/repo`. */
   repository: string;
   /** Resolved at call time by the caller — never cached or logged here. */
   token: string;
   fetchImpl: (url: string, init?: RequestInit) => Promise<Response>;
   apiBaseUrl?: string;
+  /** Retry timing seams. Tests inject them; production uses the defaults. */
+  retryDeps?: Partial<RetryDeps>;
 }
 
 export interface GithubIssueRef {
@@ -29,12 +53,42 @@ export class GithubApiError extends Error {
     readonly status: number,
     /** True when GitHub asked us to back off rather than rejecting the request outright. */
     readonly retryable: boolean,
+    /**
+     * How long GitHub asked us to wait, in ms, when it said so via
+     * `retry-after` or `x-ratelimit-reset`. `null` when it did not.
+     */
+    readonly retryAfterMs: number | null = null,
   ) {
     super(message);
     this.name = "GithubApiError";
   }
 }
 
+/**
+ * GitHub names a wait in one of two ways: `retry-after` (seconds, on secondary
+ * rate limits and abuse detection) or `x-ratelimit-reset` (epoch seconds, on
+ * primary rate limits). Anything absent, unparseable, or in the past yields
+ * `null`, and the caller falls back to its own backoff.
+ */
+function parseRetryAfterMs(headers: Headers, nowMs: number): number | null {
+  const retryAfter = headers.get("retry-after");
+  if (retryAfter) {
+    const seconds = Number(retryAfter);
+    if (Number.isFinite(seconds) && seconds >= 0) return Math.round(seconds * 1000);
+  }
+
+  const reset = headers.get("x-ratelimit-reset");
+  if (reset) {
+    const resetSeconds = Number(reset);
+    if (Number.isFinite(resetSeconds)) {
+      const waitMs = resetSeconds * 1000 - nowMs;
+      if (waitMs > 0) return Math.round(waitMs);
+    }
+  }
+
+  return null;
+}
+
 /** `owner/repo` → validated parts. Throws on anything else so a typo fails loudly at setup. */
 export function parseRepository(repository: string): { owner: string; repo: string } {
   const match = /^([A-Za-z0-9._-]+)\/([A-Za-z0-9._-]+)$/.exec(repository.trim());
@@ -56,7 +110,17 @@ export class GithubClient {
     this.apiBaseUrl = options.apiBaseUrl ?? DEFAULT_API_BASE;
   }
 
+  /**
+   * One write, retried on the failures that can succeed on a second try. The
+   * retry lives here rather than in the handlers so every call gets it, and so
+   * a handler that ends up in `guard` has genuinely exhausted its options
+   * rather than given up on the first 502.
+   */
   private async request<T>(path: string, init: RequestInit): Promise<T> {
+    return withRetry(() => this.attempt<T>(path, init), this.options.retryDeps);
+  }
+
+  private async attempt<T>(path: string, init: RequestInit): Promise<T> {
     const response = await this.options.fetchImpl(`${this.apiBaseUrl}${path}`, {
       ...init,
       headers: {
@@ -83,6 +147,7 @@ export class GithubClient {
         `GitHub ${init.method ?? "GET"} ${path} failed: ${response.status} ${detail.slice(0, 300)}`,
         response.status,
         rateLimited || response.status >= 500,
+        parseRetryAfterMs(response.headers, Date.now()),
       );
     }
 

packages/plugins/paperclip-plugin-github-mirror/src/manifest.ts

@@ -1,5 +1,5 @@
 import type { PaperclipPluginManifestV1 } from "@paperclipai/plugin-sdk";
-import { PLUGIN_ID, PLUGIN_VERSION } from "./constants.js";
+import { OUTBOX_DRAIN_JOB, PLUGIN_ID, PLUGIN_VERSION } from "./constants.js";
 
 const manifest: PaperclipPluginManifestV1 = {
   id: PLUGIN_ID,
@@ -17,10 +17,20 @@ const manifest: PaperclipPluginManifestV1 = {
     "plugin.state.write",
     "http.outbound",
     "secrets.read-ref",
+    "jobs.schedule",
   ],
   entrypoints: {
     worker: "./dist/worker.js",
   },
+  jobs: [
+    {
+      jobKey: OUTBOX_DRAIN_JOB,
+      displayName: "Resolve interrupted mirrors",
+      description:
+        "Closes create records left open by an interrupted mirror, and marks the ones that cannot be confirmed. Reads plugin state only — it never calls GitHub.",
+      schedule: "*/5 * * * *",
+    },
+  ],
   instanceConfigSchema: {
     type: "object",
     properties: {