fix(pipeline): bound embed batches by count times longest input, not by total [pipeline, poller, docs, tests]

docs/0-requirements.md

@@ -195,7 +195,7 @@ The commit-diff poller runs in two phases:
 
 Each phase is bounded on two axes per repo per run: at most 5 commits, and at most `diffFileBudgetPerPhase` files. Upserts through `processAndUpsertCommitDiff` are idempotent on `(repo, commit_sha, file_path)`, so overlap between webhook and either phase is safe.
 
-A commit count does not bound what a phase spends. Each indexed file costs 3 subrequests worst case — the D1 FTS mirror write, the store row, and the amortised share of its embed batch (a batch spends 2 and holds at least 2 files, since `MAX_EMBEDDING_INPUT_CHARS` caps an input at 8000 characters, which is at most 24000 UTF-8 bytes, against the 60000-byte batch budget) — and one commit carries up to 300 files, so 5 commits span two orders of magnitude of cost. The measured consequence: a 44-file commit in the last repo of `POLL_REPOS` had all three of its embed batches rejected with `Too many subrequests by single Worker invocation` while the repos ahead of it in the loop indexed normally, and the invariant below then held its watermark on that commit — the same deterministic, cron-after-cron stall the token axis produced, on the file axis (issue #238).
+A commit count does not bound what a phase spends. Each indexed file costs 3 subrequests worst case — the D1 FTS mirror write, the store row, and the amortised share of its embed batch (a batch spends 2 and holds at least 2 files, since `MAX_EMBEDDING_INPUT_CHARS` caps an input at 8000 characters, which is at most 24000 UTF-8 bytes, and the endpoint charges a batch its input count times its longest input: 2 × 24003 = 48006 against a ceiling of 60000) — and one commit carries up to 300 files, so 5 commits span two orders of magnitude of cost. The measured consequence: a 44-file commit in the last repo of `POLL_REPOS` had all three of its embed batches rejected with `Too many subrequests by single Worker invocation` while the repos ahead of it in the loop indexed normally, and the invariant below then held its watermark on that commit — the same deterministic, cron-after-cron stall the token axis produced, on the file axis (issue #238).
 
 The diff surface therefore declares its share of the invocation ceiling explicitly: `DIFF_SUBREQUEST_BUDGET_PER_RUN` = 900 of the 1000 Cloudflare allows. The diffs cron is the surface's own invocation, so that ceiling is not shared with docs / wiki / issues / releases — but it *is* shared across every repo in `POLL_REPOS` and both phases, which is the sharing that produced the failure above. `diffFileBudgetPerPhase` divides the declared share by `repoCount × 2` and converts the remainder to files. Deriving it from the repo list rather than fixing a literal is deliberate: `POLL_REPOS` grows by ordinary config commits (issue #233 appended the sixth repo), and a literal sized against the list of the day overruns the ceiling on the next append, surfacing as failures on whichever repo the loop reaches last rather than on the change that caused them. A floor of 5 files holds under a repo list long enough to divide the budget below it — an overrun is retried by the next cron, whereas a budget of zero stalls every diff watermark permanently, which is the failure being removed.
 
@@ -260,7 +260,7 @@ Responsibilities:
 - on the metadata-only path (state / labels / milestone / assignees changed, body did not) a failed mirror write is **not** best-effort: the diff baseline is held so the next poll or webhook delivery retries. The baseline is the IssueStore record itself, so advancing it past a failed mirror makes the miss permanent — a state-only change never brings the body change the embed path waits for (issue #209)
 - the dense and sparse mirrors on that path are written independently: a row with no vector (issue #210) still gets its sparse state updated
 - for commit diffs: batch-embed a commit's file list through Workers AI (`text: string[]`) and upsert each batch's N vectors in one `VECTORIZE.upsert` call
-- batches are cut on a UTF-8 byte budget (`MAX_EMBEDDING_BATCH_BYTES`), not on a file count, not on an estimated token count, and not on a character count. The ceiling is the endpoint's aggregate context across a call's inputs, not bge-m3's documented 8192-token per-input maximum: the batch is summed, and the rejection names the sum (`3030: Max context reached 68736 tokens but model supports only 60000`). It is unpublished, so `WORKERS_AI_BATCH_CONTEXT_LIMIT` records it from that error, and the byte budget is that ceiling. What makes bytes the right unit is that they *dominate* the token count rather than approximating it: byte fallback is the finest split the tokenizer can make — a character the vocabulary lacks is decomposed into the bytes of its UTF-8 encoding — so no token spans less than one byte of the input, and charging each input its byte length plus `TOKEN_OVERHEAD_PER_INPUT` puts the batch inside the ceiling. That overhead is 3: the two sentinels the model wraps around each input (`<s>` … `</s>`), plus the SentencePiece word-boundary marker, which is emitted as a token of its own where it does not merge into the first piece and so consumes no byte. It is charged per input rather than as one flat reserve because nothing bounds a batch's input count, and many tiny inputs — each carrying its own three — is where a flat reserve comes up short. A file count cannot express any of this, and the two budgets before this one each rested on a premise that measurement broke. The first estimated tokens, reading ASCII at 3 characters per token while bge-m3 splits a diff patch nearer 1.4 — `+`/`-` prefixes, indentation, punctuation and short identifiers all tokenize small — so it ran about 2.1x optimistic and passed 60678- and 64413-token batches as fitting a 30000-token budget (issue #241). The second charged characters, on the premise that a token spans at least one character; byte fallback is exactly the case that breaks it, since one 3-byte Japanese character can cost 3 tokens, and production answered a batch charged at most 60000 characters with 68736 tokens — 1.146 tokens per character (issue #244). Bytes sit below both: nothing splits finer, so no payload can be the one this underestimates. What is left assumed is that the tokenizer's NFKC normalization does not expand the input in bytes, which holds except for compatibility characters that decompose into several (Arabic ligatures, CJK square abbreviations) — a different class of exposure from the two retired premises, which broke on ordinary diff text, where this one needs an input made predominantly of one rare block. A commit that goes over fails its whole chunk, and a commit whose vectors never landed is one the diff watermark holds on (issue #178), so the surface stalls there permanently instead of passing it by — deterministically, on the same commit every cron tick, which is what separates it from the transient subrequest overruns sharing the log (issue #236). No margin is held under the ceiling: a bound that holds by construction has nothing left for a margin to buy, and every extra batch spends two subrequests on an invocation budget this worker already overruns. An input whose own charge exceeds the budget is sent alone — cutting it down further is the truncation axis (`MAX_EMBEDDING_INPUT_CHARS`), and dropping it would lose the file from the index. That same truncation caps one input at 8000 characters, which is at most 24000 bytes, so a batch holds at least 2 files whatever the payload is made of, which is the floor the poller's per-file subrequest estimate rests on. That floor is a third of what the character budget gave, so a Japanese commit now splits about three ways where it used to be one call; the extra subrequests are what the bound costs, and the file budget above already covers them at exactly 3 per file
+- batches are cut on what the endpoint charges a call — its input count times its longest input — not on the sum of the inputs, not on a file count, not on an estimated token count, and not on a character count. The ceiling is the endpoint's per-call context, not bge-m3's documented 8192-token per-input maximum, and it is unpublished, so `WORKERS_AI_BATCH_CONTEXT_LIMIT` records it from the rejections (`3030: Max context reached 60678 tokens but model supports only 60000`). What those rejections count is `count × longest`, not a sum: four of them divide exactly by their input counts — 18 × 3371 = 60678, 17 × 3789 = 64413, 20 × 4296 = 85920, 16 × 4296 = 68736 — and the last two are the same commit sent at different counts with the per-input quotient unchanged, which a true sum could only produce if the 4 dropped inputs had been identical in length. The endpoint pads every slot of a batch to its longest member and bills the padded width across the whole count. Bytes remain the unit each input is measured in, because they *dominate* the token count rather than approximating it: byte fallback is the finest split the tokenizer can make — a character the vocabulary lacks is decomposed into the bytes of its UTF-8 encoding — so no token spans less than one byte of the input, and one input's cost is its byte length plus `TOKEN_OVERHEAD_PER_INPUT`. That overhead is 3: the two sentinels the model wraps around each input (`<s>` … `</s>`), plus the SentencePiece word-boundary marker, which is emitted as a token of its own where it does not merge into the first piece and so consumes no byte. It is charged per input rather than as one flat reserve because nothing bounds a batch's input count, and many tiny inputs — each carrying its own three — is where a flat reserve comes up short. The three budgets before this one all bounded the *sum* of the inputs, in three successive units, and the endpoint kept rejecting the same batches at the same figure. The first estimated tokens, reading ASCII at 3 characters per token while bge-m3 splits a diff patch nearer 1.4 — `+`/`-` prefixes, indentation, punctuation and short identifiers all tokenize small — so it ran about 2.1x optimistic and passed 60678- and 64413-token batches as fitting a 30000-token budget (issue #241). The second charged characters, on the premise that a token spans at least one character; byte fallback is exactly the case that breaks it, since one 3-byte Japanese character can cost 3 tokens, and production answered a batch charged at most 60000 characters with 68736 tokens — 1.146 tokens per character (issue #244). The third charged UTF-8 bytes, which are below every split the tokenizer can make and so were not optimistic at all — and the reported figure did not move by a single token, because the sum never appears in what the endpoint bills, whatever unit it is summed in (issue #246). What is left assumed on the per-input axis is that the tokenizer's NFKC normalization does not expand the input in bytes, which holds except for compatibility characters that decompose into several (Arabic ligatures, CJK square abbreviations) — a different class of exposure from the retired premises, which broke on ordinary diff text, where this one needs an input made predominantly of one rare block. A commit that goes over fails its whole chunk, and a commit whose vectors never landed is one the diff watermark holds on (issue #178), so the surface stalls there permanently instead of passing it by — deterministically, on the same commit every cron tick, which is what separates it from the transient subrequest overruns sharing the log (issue #236). Two properties follow from the padding that no sum axis has: the largest input sets the unit price for the whole batch, so a single large input among small ones is charged as if all of them were large, and grouping inputs of similar size is therefore what makes a batch efficient. Inputs are nonetheless taken in the order given, without sorting — the caller slices its parallel arrays (files, metadata) on the returned boundaries, and the position-for-position correspondence between inputs and returned vectors is what the upsert depends on, so sorting would buy fewer calls at the cost of an index-set return type and a permutation the caller has to carry, and it buys nothing against the worst case the file budget above is sized on. No margin is held under the ceiling: a bound that holds by construction has nothing left for a margin to buy, and every extra batch spends two subrequests on an invocation budget this worker already overruns. Equality with the ceiling is admissible, since the rejections name charges strictly above it. An input whose own cost exceeds the budget is sent alone — cutting it down further is the truncation axis (`MAX_EMBEDDING_INPUT_CHARS`), and dropping it would lose the file from the index. That same truncation caps one input at 8000 characters, which is at most 24000 bytes, so `2 × 24003 = 48006` fits and a batch holds at least 2 files whatever the payload is made of, which is the floor the poller's per-file subrequest estimate rests on. That floor is unchanged by the move off the sum: the two formulas agree exactly on a batch of equal inputs, which is the case it is read off, and the file budget above still covers it at exactly 3 subrequests per file
 
 **Missing-entry repair.** The watermark fix stops the leak but does not fill the hole: a stranded item is only re-fetched when its `updated_at` moves, and closed history never moves again. `POST /admin/backfill-issue-index?repo=owner/repo` (see the installation guide) walks the gap directly — the repository's issue-number space is dense and bounded, so the numbers with no `search_docs` issue / PR row are exactly the missing set, and a numeric cursor states how far the sweep has reached. A timestamp cursor over the same set would reintroduce the ordering the defect exploited. Numbers GitHub no longer has (deleted or transferred) answer 404 and are counted rather than retried. The ingest is forced past the body-hash check: every candidate is known to be missing a retrieval surface, and a matching hash — which an embed whose FTS5 mirror failed leaves behind — would otherwise skip it permanently. Unlike the state repair this one embeds, so every candidate carries the full ingest fan-out and the caller drives the sweep one batch at a time; `dry_run=true` measures the gap without spending it. The sweep obeys the same invariant as the poller's watermark: **the cursor never advances past the first candidate a call failed to ingest**, so a per-call limit set too high costs a wasted call rather than a missed item, and the result does not depend on how accurately that limit models the subrequest budget (issue #216). A number GitHub no longer has is exempt — nothing will ever ingest it, so holding there would stall the sweep instead of bounding a retry. The tradeoff is the poller's: a candidate that fails on every attempt stops the sweep. Here that is visible rather than silent (`nextCursor` comes back equal to the `cursor` passed in), and because a human or an AI drives this endpoint rather than cron, stepping over the blocking number is a matter of passing the next `cursor` by hand.