|
| 1 | +# Hooks Framework |
| 2 | + |
| 3 | +Fire-and-forget side effects for pipeline lifecycle events: one shared event contract, a durable hook topic per domain, pluggable hooks. |
| 4 | + |
| 5 | +## Problem |
| 6 | + |
| 7 | +The pipelines emit lifecycle transitions — a request lands or fails, a batch merges, a build finishes — but nothing can react outside pipeline state: no warehouse export, no PR comments or closes on merge events, no notifications or audit trails. The log topic is not this seam: SubmitQueue request statuses only, consumed solely to build gateway read models. |
| 8 | + |
| 9 | +Two requirements: side effects must never stall or fail the pipeline, and "fire and forget" must not mean lossy — a merge-failure comment that silently never posts is a support ticket. |
| 10 | + |
| 11 | +## Proposal |
| 12 | + |
| 13 | +When a controller performs a transition, it also publishes a **hook event** to a durable `hook` topic. A thin per-domain dispatcher stage consumes it and hands each event to the **hooks** the host wired — no-op by default, real integrations as they arrive. |
| 14 | + |
| 15 | +``` |
| 16 | +pipeline controller dispatcher stage (per domain) |
| 17 | +state write → hook publish → downstream ──▶ [hook topic] ──▶ decode → validate → hook.Handle |
| 18 | + │ ├─ noop (default) |
| 19 | + │ retries exhausted └─ composite ─▶ warehouse, code host, … |
| 20 | + ▼ |
| 21 | + [hook_dlq] ──▶ log full event + page; manual republish |
| 22 | +``` |
| 23 | + |
| 24 | +One event shape for every domain — the CloudEvents core plus a version ordinal: |
| 25 | + |
| 26 | +| Field | What it is | Why it is on the envelope | |
| 27 | +|---|---|---| |
| 28 | +| `id` | Opaque occurrence identity, publisher-minted (`source/type/subject-id/version`) | Queue dedupe key and hook idempotency key | |
| 29 | +| `source` | Producing domain (`submitqueue`) | Keeps multi-domain sinks unambiguous | |
| 30 | +| `type` | What happened, one dotted open string (`batch.failed`) | The single filter dimension | |
| 31 | +| `timestamp_ms` | Occurrence time, ms since epoch | Uniform time axis | |
| 32 | +| `version` | Subject's ordinal at the transition; 0 = not a versioned write | Staleness detection under at-least-once — wall clocks can't | |
| 33 | +| `payload` | Per-type facts, always including the subject's id | Everything else; open, additive vocabulary | |
| 34 | + |
| 35 | +Delivery promise: |
| 36 | + |
| 37 | +- At least once, deduped by `id` within the queue's retention window; hooks are idempotent by `id`. |
| 38 | +- The hook publish rides the delivery that causes the transition — state write, hook publish, downstream publishes, ack — so a crash replays everything. No outbox. |
| 39 | +- A failed publish fails the stage (the hook topic shares the pipeline's queue backend); loss is never silent. |
| 40 | +- Past the retry budget, the DLQ logs the complete event and pages; manual republish recovers it. |
| 41 | +- Ordering per subject only. Hook outcomes never write pipeline state. |
| 42 | + |
| 43 | +## Decisions |
| 44 | + |
| 45 | +### Identity and replay |
| 46 | + |
| 47 | +- `id` = source / type / subject id / post-transition version; the causal message id stands in when unversioned, plus an ordinal for multiple same-typed events per cause. Components are separator-free. Consumers never parse it. |
| 48 | +- Partition key = subject id; per-subject order only. |
| 49 | +- Replay finding the target state already written → republish (idempotent). Beyond it → superseded; the event may be lost. Named, accepted gap. |
| 50 | +- RPC-caused transitions are at-most-once; needing the guarantee means publishing from the first queue-driven stage. |
| 51 | +- Opt-in per host via topic-key registration; a registered host never skips, so off and loss are distinguishable. |
| 52 | + |
| 53 | +### Contract |
| 54 | + |
| 55 | +- `api/base/hook/`: no owning domain, so the message-queue location rule extends — platform-owned contracts live under `api/base/`. |
| 56 | +- Envelope = only fields every consumer keys on uniformly; subject, queue, and error are occurrence facts → payload. `source`/`type` are strings, not enums, for additive evolution. |
| 57 | +- Payload (`Struct`): shaped per type, add-only, documented by its domain; must carry the subject's id and transient facts (merge step outcomes, build failure detail) — the event is their only durable record. Never entity snapshots; hooks resolve entities from stores. |
| 58 | + |
| 59 | +### Hooks and dispatch |
| 60 | + |
| 61 | +- Extension at `platform/extension/hook/`, singleton shape (counter precedent), wired once per host; no per-queue factory. |
| 62 | +- Hook contract: at-least-once, idempotent by `id`, plain errors, never writes pipeline state; ignore an event by returning nil (no filter API). |
| 63 | +- Ships `noop` (default) and `composite` (runs all children, joins failures, names failing children). A cross-domain sink is the same impl wired into each domain. |
| 64 | +- Dispatcher: decode, validate (`id`/`source`/`type` non-empty), invoke. Malformed events dead-letter, never silently acked; hook errors retry then dead-letter, with errs classifiers fast-pathing permanent failures. |
| 65 | +- DLQ reconciler: log the full event with its failure attribution, page (new metric — the log DLQ only warns), then ack. Manual republish recovers; pipeline state is never touched. |
| 66 | +- Per-hook retry isolation later: consumer groups on the same `hook` topic key, once the registry supports multiple groups per key and rejection becomes group-local (today it moves the shared row). Until then the composite's shared budget is accepted. |
| 67 | + |
| 68 | +## Example |
| 69 | + |
| 70 | +```proto |
| 71 | +syntax = "proto3"; |
| 72 | +
|
| 73 | +package uber.base.hook; |
| 74 | +
|
| 75 | +import "google/protobuf/struct.proto"; |
| 76 | +
|
| 77 | +import "api/base/messagequeue/proto/messagequeue.proto"; |
| 78 | +
|
| 79 | +// HookEvent is one fire-and-forget lifecycle event. Every domain publishes |
| 80 | +// this same shape to its own hook topic; hook implementations consume it. |
| 81 | +message HookEvent { |
| 82 | + option (uber.base.messagequeue.topic_keys) = "hook"; |
| 83 | +
|
| 84 | + string id = 1; // Opaque occurrence identity; queue message id and hook idempotency key. |
| 85 | + string source = 2; // Producing domain: "submitqueue", "stovepipe", ... |
| 86 | + string type = 3; // What happened: "request.landed", "batch.failed", ... |
| 87 | + int64 timestamp_ms = 4; // Occurrence time, ms since the Unix epoch. |
| 88 | + int32 version = 5; // Subject's version at the transition; 0 when not tied to a state write. |
| 89 | + google.protobuf.Struct payload = 6; // Publisher-defined facts, including the subject's id; never a snapshot. |
| 90 | +} |
| 91 | +``` |
| 92 | + |
| 93 | +A failed batch, carrying merge-result facts persisted nowhere else (protojson: int64 as string, empty fields omitted): |
| 94 | + |
| 95 | +```json |
| 96 | +{ |
| 97 | + "id": "submitqueue/batch.failed/batch-778/4", |
| 98 | + "source": "submitqueue", |
| 99 | + "type": "batch.failed", |
| 100 | + "timestamp_ms": "1722800012345", |
| 101 | + "version": 4, |
| 102 | + "payload": { |
| 103 | + "batch_id": "batch-778", |
| 104 | + "queue": "go-monorepo", |
| 105 | + "error": "merge conflict", |
| 106 | + "failed_step": "sq-12346", |
| 107 | + "conflict_paths": ["foo/bar.go"] |
| 108 | + } |
| 109 | +} |
| 110 | +``` |
| 111 | + |
| 112 | +## Rejected |
| 113 | + |
| 114 | +- **A contract per domain.** N schemas, N hook shapes, N warehouse tables; one envelope absorbs differences additively. |
| 115 | +- **Inline hook calls.** Couples pipeline latency to integrations; a crash between write and call silently drops the notification. |
| 116 | +- **A second consumer group on the log topic.** Request statuses only; no path to batch, build, merge, or other domains. |
| 117 | +- **Enums for source/type.** protojson rejects unknown enum values; every addition would break consumers. |
| 118 | +- **Subject, queue, or error on the envelope.** Occurrence facts; they live in the payload. No major event platform carries a top-level error. |
| 119 | +- **Entity snapshots as payload.** Stale on redelivery; competes with the store; drags domain schemas into the shared contract. |
| 120 | +- **Typed per-event payloads (`oneof`).** Every new type becomes a wire-contract change. |
| 121 | +- **A filter/subscription API.** Returning nil costs nothing; routing can be a wiring decorator later. |
| 122 | +- **Best-effort publishing.** Silent loss; failing the stage is safe because dedupe makes the retry idempotent. |
| 123 | +- **A transactional outbox.** The publish rides the triggering delivery before ack; a crash replays both. RPC-edge transitions are scoped out instead. |
0 commit comments