diff --git a/docs/0-requirements.ja.md b/docs/0-requirements.ja.md index 1b8e06c..3cafcba 100644 --- a/docs/0-requirements.ja.md +++ b/docs/0-requirements.ja.md @@ -190,7 +190,7 @@ commit diff poller は 2-phase 構成: 1 repo 1 run あたりの上限は、各 phase が 2 軸で持つ: commit 数 5 件と、file 数 `diffFileBudgetPerPhase`。`processAndUpsertCommitDiff` の upsert は `(repo, commit_sha, file_path)` で idempotent なので、webhook / 両 phase 間で overlap しても副作用はない。 -commit 数は phase の消費量を縛れない。1 file の index は worst case で 3 subrequest(D1 FTS mirror 書き込み、store row、および embed batch の按分——1 batch は 2 subrequest を使い、`MAX_EMBEDDING_INPUT_CHARS` が 1 input を 8000 文字=最大 24000 UTF-8 バイトに切る一方 batch 予算は 60000 バイトなので、1 batch は最低 2 file を載せる)であり、1 commit は最大 300 file を運ぶため、5 commits のコストは 2 桁の幅を持つ。実測された帰結: `POLL_REPOS` の末尾 repo にある 44 file の commit で、embed batch 3 本すべてが `Too many subrequests by single Worker invocation` で拒否された。同じ loop の手前にある repo は正常に index されていた。そして後述の不変条件がその commit で watermark を止める——token 軸が起こしたのとまったく同じ、毎 cron 決定論的に再現する停止が、file 軸で起きた(issue #238)。 +commit 数は phase の消費量を縛れない。1 file の index は worst case で 3 subrequest(D1 FTS mirror 書き込み、store row、および embed batch の按分——1 batch は 2 subrequest を使い、`MAX_EMBEDDING_INPUT_CHARS` が 1 input を 8000 文字=最大 24000 UTF-8 バイトに切り、endpoint は batch を「input 件数 × 最長 input」で課金するので 2 × 24003 = 48006 が天井 60000 に収まり、1 batch は最低 2 file を載せる)であり、1 commit は最大 300 file を運ぶため、5 commits のコストは 2 桁の幅を持つ。実測された帰結: `POLL_REPOS` の末尾 repo にある 44 file の commit で、embed batch 3 本すべてが `Too many subrequests by single Worker invocation` で拒否された。同じ loop の手前にある repo は正常に index されていた。そして後述の不変条件がその commit で watermark を止める——token 軸が起こしたのとまったく同じ、毎 cron 決定論的に再現する停止が、file 軸で起きた(issue #238)。 そこで diff surface は invocation 予算のうち自分の取り分を明示する: `DIFF_SUBREQUEST_BUDGET_PER_RUN` = Cloudflare が許す 1000 のうち 900。diffs は専用 cron を持つので、この天井を docs / wiki / issue / release と共有はしていない。しかし `POLL_REPOS` の全 repo と両 phase では共有しており、上記の失敗を生んだのはそちらの共有である。`diffFileBudgetPerPhase` は宣言した取り分を `repoCount × 2` で割り、残りを file 数に換算する。literal を固定せず repo list から導出するのは意図的である: `POLL_REPOS` は通常の config commit で増える(issue #233 が 6 番目の repo を追加した)ため、その日の list に合わせた literal は次の追加で天井を超え、しかもその超過は原因となった変更ではなく loop が最後に到達した repo の失敗として現れる。repo list が長く予算を下回るところまで割った場合は 5 file の下限が効く——超過は次の cron が再試行するが、予算 0 は全 diff watermark を恒久的に止める。それはいま取り除こうとしている失敗そのものである。 @@ -255,7 +255,7 @@ Responsibilities: - metadata のみの経路(body は変わらず state / labels / milestone / assignees が変わった場合)では、mirror 書き込みの失敗を best-effort 扱いに**しない**。差分検出の基準を進めずに保持し、次の poll / webhook 配信で再試行させる。基準は IssueStore の record そのものなので、失敗した mirror を追い越して基準を進めると取り残しが恒久化する — state だけの変更は、embed 経路が待っている body 変更を二度と連れてこない(issue #209) - この経路の dense / sparse mirror は互いに独立して書く。vector が欠けている行(issue #210)でも sparse 側の state は更新される - commit diff は 1 commit 分の file リストを batch embed(Workers AI の `text: string[]` 対応を利用)し、batch ごとに 1 回の Vectorize upsert で N vector を書き込む -- batch の切れ目は file 件数でも token 数の推定でも文字数でもなく、UTF-8 バイト数予算(`MAX_EMBEDDING_BATCH_BYTES`)で決める。天井は bge-m3 の documented な per-input 上限 8192 token ではなく、1 call の input 全体を合算した endpoint 側の上限である。batch は合算され、拒否応答はその合計値を名指しする(`3030: Max context reached 68736 tokens but model supports only 60000`)。この値は非公開なので `WORKERS_AI_BATCH_CONTEXT_LIMIT` としてそのエラーから記録し、バイト数予算はその天井そのものに置く。バイト数が正しい単位である理由は、それが token 数の近似ではなく**上界**だからである。tokenizer が作れる最も細かい分解が byte_fallback — 語彙に無い文字はその UTF-8 バイト列へ分解される — であり、それ以下には割れないので、1 token は入力の 1 バイト以上を必ず消費する。したがって各 input にその UTF-8 バイト数と `TOKEN_OVERHEAD_PER_INPUT` を課金すれば、batch は天井の内側に収まる。この overhead が 3 なのは、model が各 input を挟む sentinel 2 個(`` … ``)に加え、SentencePiece の語境界マーカーが先頭 piece に併合されない場合に単独の token として現れ、入力バイトを 1 つも消費しないためである。1 batch 固定の引当てではなく input ごとに課金するのは、batch の input 件数に上限が無く、極小 input が多数並ぶ場合(そのそれぞれが自分の 3 を持つ)に固定引当てでは足りなくなるためである。file 件数ではこれを表現できず、これ以前の 2 つの予算はいずれも実測に破られた前提の上に立っていた。1 つ目は token 数の推定で、ASCII を 3 文字/token として読むが bge-m3 が diff patch を割る密度は約 1.4 文字/token である — `+` / `-` 接頭、インデント、記号、短い識別子がいずれも細かく分割される — ため約 2.1x 楽観に振れ、実測 60678 / 64413 token の batch を「30000 token 予算に収まる」と判定して通していた(issue #241)。2 つ目は文字数で、「1 token は 1 文字以上に対応する」という前提に立っていたが、byte_fallback がまさにその前提を破る経路であり、3 バイトの日本語 1 文字が 3 token になりうる。実測では、文字数で高々 60000 と課金した batch に 68736 token が返った — 1 文字あたり 1.146 token である(issue #244)。バイト数はその両方の下にある。これより細かい分解が無いので、この軸が過小評価する payload は存在しない。残る前提は「tokenizer の NFKC 正規化が入力をバイト数で膨らませない」ことだけであり、これは互換分解を持つ文字(アラビア語の合字、CJK の組文字)を除いて成立する — 退けた 2 つの前提が通常の diff テキストで破れたのとは性質が異なり、こちらは入力の大半がその稀なブロックで占められている必要がある。天井を超えた commit は chunk 全体を失敗させ、vector が載らなかった commit は diff watermark が留まる対象(issue #178)なので、その commit を飛ばすのではなく surface がそこで恒久的に停止する。しかも毎 cron 同じ commit で同じ結果になる決定論的な失敗で、同じログに同居する一過性の subrequest 超過とはそこで性質が分かれる(issue #236)。天井の下に margin は取らない — 構成上成立する上界に margin が買えるものは残っておらず、batch が 1 つ増えるごとに subrequest を 2 つ消費し、この worker は既に invocation 予算を超過しているためである。単独で予算を超える input はそのまま単独で送る — さらに削るのは truncate 軸(`MAX_EMBEDDING_INPUT_CHARS`)の仕事であり、落とせばその file が索引から消えるため。その truncate が 1 input を 8000 文字=最大 24000 バイトに抑えるので、payload の中身によらず 1 batch は最低 2 file を保持する。これが poller の per-file subrequest 見積りが立っている床である。この床は文字数予算のときの 1/3 であり、日本語主体の commit はこれまで 1 call だったところが約 3 分割になる。増える subrequest は上界の代金であり、上の file 予算が 1 file あたりちょうど 3 subrequest としてすでに賄っている +- batch の切れ目は、endpoint が 1 call に課金する量 — **input 件数 × その batch の最長 input** — で決める。input の合計でも file 件数でも token 数の推定でも文字数でもない。天井は bge-m3 の documented な per-input 上限 8192 token ではなく 1 call 単位の上限であり、非公開なので `WORKERS_AI_BATCH_CONTEXT_LIMIT` として拒否応答から記録する(`3030: Max context reached 60678 tokens but model supports only 60000`)。その拒否が数えているのは合計ではなく `件数 × 最長` である。4 件の観測がいずれも input 件数で割り切れ — 18 × 3371 = 60678、17 × 3789 = 64413、20 × 4296 = 85920、16 × 4296 = 68736 — しかも後ろの 2 件は同一 commit を異なる件数で投げたもので、1 件あたりの商が 4296 のまま動かない。これが真の合計であれば、除外された 4 件が全て同一長でない限り起こりえない。endpoint は batch の各スロットを最長の input に padding し、その幅を件数分だけ課金している。1 input を測る単位はこれまで通り UTF-8 バイト数である。バイト数は token 数の近似ではなく**上界**だからで、tokenizer が作れる最も細かい分解が byte_fallback — 語彙に無い文字はその UTF-8 バイト列へ分解される — であり、1 token は入力の 1 バイト以上を必ず消費する。したがって 1 input のコストはそのバイト数に `TOKEN_OVERHEAD_PER_INPUT` を加えた値になる。この overhead が 3 なのは、model が各 input を挟む sentinel 2 個(`` … ``)に加え、SentencePiece の語境界マーカーが先頭 piece に併合されない場合に単独の token として現れ、入力バイトを 1 つも消費しないためである。1 batch 固定の引当てではなく input ごとに課金するのは、batch の input 件数に上限が無く、極小 input が多数並ぶ場合(そのそれぞれが自分の 3 を持つ)に固定引当てでは足りなくなるためである。これ以前の 3 つの予算はいずれも input の**合計**を縛っており、単位を 3 度替えても endpoint は同じ batch を同じ数値で拒否し続けた。1 つ目は token 数の推定で、ASCII を 3 文字/token として読むが bge-m3 が diff patch を割る密度は約 1.4 文字/token である — `+` / `-` 接頭、インデント、記号、短い識別子がいずれも細かく分割される — ため約 2.1x 楽観に振れ、実測 60678 / 64413 token の batch を「30000 token 予算に収まる」と判定して通していた(issue #241)。2 つ目は文字数で、「1 token は 1 文字以上に対応する」という前提に立っていたが、byte_fallback がまさにその前提を破る経路であり、3 バイトの日本語 1 文字が 3 token になりうる。実測では、文字数で高々 60000 と課金した batch に 68736 token が返った — 1 文字あたり 1.146 token である(issue #244)。3 つ目は UTF-8 バイト数で、これは tokenizer のどの分解よりも下にあるので楽観ではまったくなかった — それでも報告値は 1 token も動かなかった。どの単位で合計しようと、合計は endpoint の課金式に現れないからである(issue #246)。per-input 軸に残る前提は「tokenizer の NFKC 正規化が入力をバイト数で膨らませない」ことだけであり、これは互換分解を持つ文字(アラビア語の合字、CJK の組文字)を除いて成立する — 退けた前提が通常の diff テキストで破れたのとは性質が異なり、こちらは入力の大半がその稀なブロックで占められている必要がある。天井を超えた commit は chunk 全体を失敗させ、vector が載らなかった commit は diff watermark が留まる対象(issue #178)なので、その commit を飛ばすのではなく surface がそこで恒久的に停止する。しかも毎 cron 同じ commit で同じ結果になる決定論的な失敗で、同じログに同居する一過性の subrequest 超過とはそこで性質が分かれる(issue #236)。padding から、合計軸には無かった 2 つの性質が従う: 最大の input が batch 全体の単価を決めるので、小さい input の中に大きいものが 1 件混ざれば全件がその長さで課金される。したがってサイズの近いものをまとめるほど batch の効率は上がる。それでも input は与えられた順のまま扱い、並べ替えは行わない — 呼び出し側は返された境界で自分の並列配列(files / metadata)を切っており、input と返り vector の位置対応こそが upsert の前提だからである。並べ替えは call 数を減らす代わりに index 集合の返り値型と呼び出し側が持ち回る置換を要求し、しかも上の file 予算が想定している最悪ケースには何も効かない。天井の下に margin は取らない — 構成上成立する上界に margin が買えるものは残っておらず、batch が 1 つ増えるごとに subrequest を 2 つ消費し、この worker は既に invocation 予算を超過しているためである。天井ちょうどは許容する。拒否応答が名指すのは天井を厳密に超えた課金額だからである。単独で予算を超える input はそのまま単独で送る — さらに削るのは truncate 軸(`MAX_EMBEDDING_INPUT_CHARS`)の仕事であり、落とせばその file が索引から消えるため。その truncate が 1 input を 8000 文字=最大 24000 バイトに抑えるので `2 × 24003 = 48006` は天井に収まり、payload の中身によらず 1 batch は最低 2 file を保持する。これが poller の per-file subrequest 見積りが立っている床である。合計軸から移っても この床は動かない: 2 つの式は input が等長の batch では厳密に一致し、床を読み取るのはまさにその場合であり、上の file 予算が 1 file あたりちょうど 3 subrequest として引き続き賄っている **索引欠落の修復.** watermark の修正は漏れを止めるだけで、既に空いた穴は埋まらない — 取り残された項目が再 fetch されるのは `updated_at` が動いたときだけで、閉じた履歴はもう動かない。`POST /admin/backfill-issue-index?repo=owner/repo`(installation guide 参照)が欠落そのものを走査する。repository の issue 番号空間は密かつ有界なので、`search_docs` に issue / PR 行が無い番号がそのまま欠落集合であり、数値 cursor が「どこまで走査したか」を厳密に表せる。同じ集合を時刻 cursor で辿ると、欠陥が突いた順序をそのまま持ち込むことになる。GitHub 側に既に無い番号(削除・transfer 済み)は 404 を返すので、retry せず計上のみ。取り込みは body-hash 判定を強制的に飛ばす: 候補はいずれも retrieval surface が欠けていると分かっている項目であり、hash が一致していると(embed 成功後に FTS5 mirror が失敗した行がこの状態になる)そのまま恒久的に skip されてしまうため。state 修復と違いこちらは embed を伴い、1 候補ごとに取り込み fan-out の全額がかかるので、呼び出し側が batch を跨いで sweep を進める。`dry_run=true` は予算を使わずに欠落量だけを測る。sweep は poller の watermark と同じ不変条件に従う: **取り込みそこねた最初の候補を cursor が追い越さない。** これにより 1 call の上限を大きく取りすぎたときの代償は 1 回分の無駄な呼び出しであって取りこぼしではなく、結果が「上限値が subrequest 予算をどれだけ正確に写しているか」に依存しなくなる(issue #216)。GitHub 側に既に無い番号は例外 — 誰が何回試しても取り込めないので、そこで止めると retry の境界にならず sweep が停止する。トレードオフは poller と同じで、恒久的に失敗する候補があると sweep は止まる。ただしそれは応答に現れる(`nextCursor` が渡した `cursor` と同じ値で返る)し、この endpoint は cron ではなく人間 / AI が駆動するので、詰まった番号を手動の `cursor` 指定で越えられる。 diff --git a/docs/0-requirements.md b/docs/0-requirements.md index 8b8d099..831f5b7 100644 --- a/docs/0-requirements.md +++ b/docs/0-requirements.md @@ -195,7 +195,7 @@ The commit-diff poller runs in two phases: Each phase is bounded on two axes per repo per run: at most 5 commits, and at most `diffFileBudgetPerPhase` files. Upserts through `processAndUpsertCommitDiff` are idempotent on `(repo, commit_sha, file_path)`, so overlap between webhook and either phase is safe. -A commit count does not bound what a phase spends. Each indexed file costs 3 subrequests worst case — the D1 FTS mirror write, the store row, and the amortised share of its embed batch (a batch spends 2 and holds at least 2 files, since `MAX_EMBEDDING_INPUT_CHARS` caps an input at 8000 characters, which is at most 24000 UTF-8 bytes, against the 60000-byte batch budget) — and one commit carries up to 300 files, so 5 commits span two orders of magnitude of cost. The measured consequence: a 44-file commit in the last repo of `POLL_REPOS` had all three of its embed batches rejected with `Too many subrequests by single Worker invocation` while the repos ahead of it in the loop indexed normally, and the invariant below then held its watermark on that commit — the same deterministic, cron-after-cron stall the token axis produced, on the file axis (issue #238). +A commit count does not bound what a phase spends. Each indexed file costs 3 subrequests worst case — the D1 FTS mirror write, the store row, and the amortised share of its embed batch (a batch spends 2 and holds at least 2 files, since `MAX_EMBEDDING_INPUT_CHARS` caps an input at 8000 characters, which is at most 24000 UTF-8 bytes, and the endpoint charges a batch its input count times its longest input: 2 × 24003 = 48006 against a ceiling of 60000) — and one commit carries up to 300 files, so 5 commits span two orders of magnitude of cost. The measured consequence: a 44-file commit in the last repo of `POLL_REPOS` had all three of its embed batches rejected with `Too many subrequests by single Worker invocation` while the repos ahead of it in the loop indexed normally, and the invariant below then held its watermark on that commit — the same deterministic, cron-after-cron stall the token axis produced, on the file axis (issue #238). The diff surface therefore declares its share of the invocation ceiling explicitly: `DIFF_SUBREQUEST_BUDGET_PER_RUN` = 900 of the 1000 Cloudflare allows. The diffs cron is the surface's own invocation, so that ceiling is not shared with docs / wiki / issues / releases — but it *is* shared across every repo in `POLL_REPOS` and both phases, which is the sharing that produced the failure above. `diffFileBudgetPerPhase` divides the declared share by `repoCount × 2` and converts the remainder to files. Deriving it from the repo list rather than fixing a literal is deliberate: `POLL_REPOS` grows by ordinary config commits (issue #233 appended the sixth repo), and a literal sized against the list of the day overruns the ceiling on the next append, surfacing as failures on whichever repo the loop reaches last rather than on the change that caused them. A floor of 5 files holds under a repo list long enough to divide the budget below it — an overrun is retried by the next cron, whereas a budget of zero stalls every diff watermark permanently, which is the failure being removed. @@ -260,7 +260,7 @@ Responsibilities: - on the metadata-only path (state / labels / milestone / assignees changed, body did not) a failed mirror write is **not** best-effort: the diff baseline is held so the next poll or webhook delivery retries. The baseline is the IssueStore record itself, so advancing it past a failed mirror makes the miss permanent — a state-only change never brings the body change the embed path waits for (issue #209) - the dense and sparse mirrors on that path are written independently: a row with no vector (issue #210) still gets its sparse state updated - for commit diffs: batch-embed a commit's file list through Workers AI (`text: string[]`) and upsert each batch's N vectors in one `VECTORIZE.upsert` call -- batches are cut on a UTF-8 byte budget (`MAX_EMBEDDING_BATCH_BYTES`), not on a file count, not on an estimated token count, and not on a character count. The ceiling is the endpoint's aggregate context across a call's inputs, not bge-m3's documented 8192-token per-input maximum: the batch is summed, and the rejection names the sum (`3030: Max context reached 68736 tokens but model supports only 60000`). It is unpublished, so `WORKERS_AI_BATCH_CONTEXT_LIMIT` records it from that error, and the byte budget is that ceiling. What makes bytes the right unit is that they *dominate* the token count rather than approximating it: byte fallback is the finest split the tokenizer can make — a character the vocabulary lacks is decomposed into the bytes of its UTF-8 encoding — so no token spans less than one byte of the input, and charging each input its byte length plus `TOKEN_OVERHEAD_PER_INPUT` puts the batch inside the ceiling. That overhead is 3: the two sentinels the model wraps around each input (`` … ``), plus the SentencePiece word-boundary marker, which is emitted as a token of its own where it does not merge into the first piece and so consumes no byte. It is charged per input rather than as one flat reserve because nothing bounds a batch's input count, and many tiny inputs — each carrying its own three — is where a flat reserve comes up short. A file count cannot express any of this, and the two budgets before this one each rested on a premise that measurement broke. The first estimated tokens, reading ASCII at 3 characters per token while bge-m3 splits a diff patch nearer 1.4 — `+`/`-` prefixes, indentation, punctuation and short identifiers all tokenize small — so it ran about 2.1x optimistic and passed 60678- and 64413-token batches as fitting a 30000-token budget (issue #241). The second charged characters, on the premise that a token spans at least one character; byte fallback is exactly the case that breaks it, since one 3-byte Japanese character can cost 3 tokens, and production answered a batch charged at most 60000 characters with 68736 tokens — 1.146 tokens per character (issue #244). Bytes sit below both: nothing splits finer, so no payload can be the one this underestimates. What is left assumed is that the tokenizer's NFKC normalization does not expand the input in bytes, which holds except for compatibility characters that decompose into several (Arabic ligatures, CJK square abbreviations) — a different class of exposure from the two retired premises, which broke on ordinary diff text, where this one needs an input made predominantly of one rare block. A commit that goes over fails its whole chunk, and a commit whose vectors never landed is one the diff watermark holds on (issue #178), so the surface stalls there permanently instead of passing it by — deterministically, on the same commit every cron tick, which is what separates it from the transient subrequest overruns sharing the log (issue #236). No margin is held under the ceiling: a bound that holds by construction has nothing left for a margin to buy, and every extra batch spends two subrequests on an invocation budget this worker already overruns. An input whose own charge exceeds the budget is sent alone — cutting it down further is the truncation axis (`MAX_EMBEDDING_INPUT_CHARS`), and dropping it would lose the file from the index. That same truncation caps one input at 8000 characters, which is at most 24000 bytes, so a batch holds at least 2 files whatever the payload is made of, which is the floor the poller's per-file subrequest estimate rests on. That floor is a third of what the character budget gave, so a Japanese commit now splits about three ways where it used to be one call; the extra subrequests are what the bound costs, and the file budget above already covers them at exactly 3 per file +- batches are cut on what the endpoint charges a call — its input count times its longest input — not on the sum of the inputs, not on a file count, not on an estimated token count, and not on a character count. The ceiling is the endpoint's per-call context, not bge-m3's documented 8192-token per-input maximum, and it is unpublished, so `WORKERS_AI_BATCH_CONTEXT_LIMIT` records it from the rejections (`3030: Max context reached 60678 tokens but model supports only 60000`). What those rejections count is `count × longest`, not a sum: four of them divide exactly by their input counts — 18 × 3371 = 60678, 17 × 3789 = 64413, 20 × 4296 = 85920, 16 × 4296 = 68736 — and the last two are the same commit sent at different counts with the per-input quotient unchanged, which a true sum could only produce if the 4 dropped inputs had been identical in length. The endpoint pads every slot of a batch to its longest member and bills the padded width across the whole count. Bytes remain the unit each input is measured in, because they *dominate* the token count rather than approximating it: byte fallback is the finest split the tokenizer can make — a character the vocabulary lacks is decomposed into the bytes of its UTF-8 encoding — so no token spans less than one byte of the input, and one input's cost is its byte length plus `TOKEN_OVERHEAD_PER_INPUT`. That overhead is 3: the two sentinels the model wraps around each input (`` … ``), plus the SentencePiece word-boundary marker, which is emitted as a token of its own where it does not merge into the first piece and so consumes no byte. It is charged per input rather than as one flat reserve because nothing bounds a batch's input count, and many tiny inputs — each carrying its own three — is where a flat reserve comes up short. The three budgets before this one all bounded the *sum* of the inputs, in three successive units, and the endpoint kept rejecting the same batches at the same figure. The first estimated tokens, reading ASCII at 3 characters per token while bge-m3 splits a diff patch nearer 1.4 — `+`/`-` prefixes, indentation, punctuation and short identifiers all tokenize small — so it ran about 2.1x optimistic and passed 60678- and 64413-token batches as fitting a 30000-token budget (issue #241). The second charged characters, on the premise that a token spans at least one character; byte fallback is exactly the case that breaks it, since one 3-byte Japanese character can cost 3 tokens, and production answered a batch charged at most 60000 characters with 68736 tokens — 1.146 tokens per character (issue #244). The third charged UTF-8 bytes, which are below every split the tokenizer can make and so were not optimistic at all — and the reported figure did not move by a single token, because the sum never appears in what the endpoint bills, whatever unit it is summed in (issue #246). What is left assumed on the per-input axis is that the tokenizer's NFKC normalization does not expand the input in bytes, which holds except for compatibility characters that decompose into several (Arabic ligatures, CJK square abbreviations) — a different class of exposure from the retired premises, which broke on ordinary diff text, where this one needs an input made predominantly of one rare block. A commit that goes over fails its whole chunk, and a commit whose vectors never landed is one the diff watermark holds on (issue #178), so the surface stalls there permanently instead of passing it by — deterministically, on the same commit every cron tick, which is what separates it from the transient subrequest overruns sharing the log (issue #236). Two properties follow from the padding that no sum axis has: the largest input sets the unit price for the whole batch, so a single large input among small ones is charged as if all of them were large, and grouping inputs of similar size is therefore what makes a batch efficient. Inputs are nonetheless taken in the order given, without sorting — the caller slices its parallel arrays (files, metadata) on the returned boundaries, and the position-for-position correspondence between inputs and returned vectors is what the upsert depends on, so sorting would buy fewer calls at the cost of an index-set return type and a permutation the caller has to carry, and it buys nothing against the worst case the file budget above is sized on. No margin is held under the ceiling: a bound that holds by construction has nothing left for a margin to buy, and every extra batch spends two subrequests on an invocation budget this worker already overruns. Equality with the ceiling is admissible, since the rejections name charges strictly above it. An input whose own cost exceeds the budget is sent alone — cutting it down further is the truncation axis (`MAX_EMBEDDING_INPUT_CHARS`), and dropping it would lose the file from the index. That same truncation caps one input at 8000 characters, which is at most 24000 bytes, so `2 × 24003 = 48006` fits and a batch holds at least 2 files whatever the payload is made of, which is the floor the poller's per-file subrequest estimate rests on. That floor is unchanged by the move off the sum: the two formulas agree exactly on a batch of equal inputs, which is the case it is read off, and the file budget above still covers it at exactly 3 subrequests per file **Missing-entry repair.** The watermark fix stops the leak but does not fill the hole: a stranded item is only re-fetched when its `updated_at` moves, and closed history never moves again. `POST /admin/backfill-issue-index?repo=owner/repo` (see the installation guide) walks the gap directly — the repository's issue-number space is dense and bounded, so the numbers with no `search_docs` issue / PR row are exactly the missing set, and a numeric cursor states how far the sweep has reached. A timestamp cursor over the same set would reintroduce the ordering the defect exploited. Numbers GitHub no longer has (deleted or transferred) answer 404 and are counted rather than retried. The ingest is forced past the body-hash check: every candidate is known to be missing a retrieval surface, and a matching hash — which an embed whose FTS5 mirror failed leaves behind — would otherwise skip it permanently. Unlike the state repair this one embeds, so every candidate carries the full ingest fan-out and the caller drives the sweep one batch at a time; `dry_run=true` measures the gap without spending it. The sweep obeys the same invariant as the poller's watermark: **the cursor never advances past the first candidate a call failed to ingest**, so a per-call limit set too high costs a wasted call rather than a missed item, and the result does not depend on how accurately that limit models the subrequest budget (issue #216). A number GitHub no longer has is exempt — nothing will ever ingest it, so holding there would stall the sweep instead of bounding a retry. The tradeoff is the poller's: a candidate that fails on every attempt stops the sweep. Here that is visible rather than silent (`nextCursor` comes back equal to the `cursor` passed in), and because a human or an AI drives this endpoint rather than cron, stepping over the blocking number is a matter of passing the next `cursor` by hand. diff --git a/src/pipeline/embed-diff.test.ts b/src/pipeline/embed-diff.test.ts index b7a5cd9..f0b653d 100644 --- a/src/pipeline/embed-diff.test.ts +++ b/src/pipeline/embed-diff.test.ts @@ -2,7 +2,6 @@ import { describe, it, expect, vi } from "vitest"; import type { Env } from "../types.js"; import { processAndUpsertCommitDiff, type GitHubCommitDetail } from "./embed-diff.js"; import { - MAX_EMBEDDING_BATCH_BYTES, TOKEN_OVERHEAD_PER_INPUT, WORKERS_AI_BATCH_CONTEXT_LIMIT, utf8ByteLength, @@ -78,21 +77,32 @@ function mkDiffPatch(chars: number): string { return line.repeat(Math.ceil(chars / line.length)).slice(0, chars); } -/** What the planner charges one Workers AI call. Upper-bounds its token count. */ +/** What the endpoint charges one Workers AI call: its input count times its longest + * input, measured in UTF-8 bytes plus the per-input token overhead. The endpoint + * pads every slot of a batch to the longest member (#246). */ function callCharge(texts: string[]): number { + if (texts.length === 0) return 0; + const longest = Math.max( + ...texts.map((text) => utf8ByteLength(text) + TOKEN_OVERHEAD_PER_INPUT), + ); + return texts.length * longest; +} + +/** What the retired byte-sum budget charged the same call (#244). Kept in the tests + * only, to hold the regression fixtures inside the shape it passed. */ +function retiredSumCharge(texts: string[]): number { return texts.reduce( (sum, text) => sum + utf8ByteLength(text) + TOKEN_OVERHEAD_PER_INPUT, 0, ); } -/** What the retired character budget charged the same call (#242). Kept in the - * tests only, to hold the regression fixtures inside the shape it passed. */ +/** What the retired character budget charged the same call (#242). */ function retiredCharCharge(texts: string[]): number { return texts.reduce((sum, text) => sum + text.length + 2, 0); } -describe("embed-diff: the batch axis is the UTF-8 byte budget, not the file count", () => { +describe("embed-diff: the batch axis is count times longest input, not the file count", () => { it("still sends an ordinary commit as one call", async () => { // 30 files well past the retired count cap of 20, each a small patch. const { env, aiCalls } = mkEnv(); @@ -130,7 +140,7 @@ describe("embed-diff: the batch axis is the UTF-8 byte budget, not the file coun // Every call carrying more than one input stays inside the budget. for (const call of aiCalls) { if (call.length > 1) { - expect(callCharge(call)).toBeLessThanOrEqual(MAX_EMBEDDING_BATCH_BYTES); + expect(callCharge(call)).toBeLessThanOrEqual(WORKERS_AI_BATCH_CONTEXT_LIMIT); } } @@ -166,7 +176,7 @@ describe("embed-diff: the batch axis is the UTF-8 byte budget, not the file coun expect(aiCalls.length).toBeGreaterThan(1); for (const call of aiCalls) { - expect(callCharge(call)).toBeLessThanOrEqual(MAX_EMBEDDING_BATCH_BYTES); + expect(callCharge(call)).toBeLessThanOrEqual(WORKERS_AI_BATCH_CONTEXT_LIMIT); } // Splitting is only the fix if every file still lands. @@ -197,7 +207,7 @@ describe("embed-diff: the batch axis is the UTF-8 byte budget, not the file coun expect(aiCalls.length).toBeGreaterThan(1); for (const call of aiCalls) { - expect(callCharge(call)).toBeLessThanOrEqual(MAX_EMBEDDING_BATCH_BYTES); + expect(callCharge(call)).toBeLessThanOrEqual(WORKERS_AI_BATCH_CONTEXT_LIMIT); } expect(result.embedded).toBe(16); @@ -205,6 +215,65 @@ describe("embed-diff: the batch axis is the UTF-8 byte budget, not the file coun expect(new Set(upsertedIds.flat()).size).toBe(16); }); + it("splits the 18-file commit that a byte-sum budget let through", async () => { + // `neuron-graph-rag@1fb0f6b`, the commit that still failed 8 hours after the byte + // budget went to 100% traffic: 20 files, 18 of them with a patch, and the + // rejection unchanged at `Max context reached 60678 tokens but model supports + // only 60000` across all three sum-axis budgets — 18 × 3371, the longest input + // charged once per slot. The sum of these inputs is a fifth of the ceiling, so no + // sum-axis budget of any unit sees this batch. + const { env, aiCalls, upsertedIds } = mkEnv(); + const commit = mkCommit([ + mkDiffPatch(3600), + ...Array.from({ length: 17 }, (_, i) => `@@ -1 +1 @@\n+line ${i}`), + ]); + commit.files!.push( + { filename: "assets/a.png", status: "modified", sha: "blobA" }, + { filename: "assets/b.png", status: "added", sha: "blobB" }, + ); + + const result = await processAndUpsertCommitDiff(env, mkStore(), REPO, commit); + + const allInputs = aiCalls.flat(); + // The fixture reproduces the shape only while it stays in the failing zone: well + // inside the retired byte-sum budget, over the ceiling once padded to 18 slots. + expect(retiredSumCharge(allInputs)).toBeLessThanOrEqual(WORKERS_AI_BATCH_CONTEXT_LIMIT); + expect(callCharge(allInputs)).toBeGreaterThan(WORKERS_AI_BATCH_CONTEXT_LIMIT); + + expect(aiCalls.length).toBeGreaterThan(1); + for (const call of aiCalls) { + expect(callCharge(call)).toBeLessThanOrEqual(WORKERS_AI_BATCH_CONTEXT_LIMIT); + } + + expect(result.embedded).toBe(18); + expect(result.skipped).toBe(2); + expect(result.failed).toBe(0); + expect(new Set(upsertedIds.flat()).size).toBe(18); + }); + + it("clears the 44-file commit the watermark has been held on", async () => { + // `neuron-graph-rag@92eb94d`, the other commit in the stall: sent at 20 files it + // was charged 85920 and at 16 it was charged 68736 — one quotient, 4296, across + // both counts. The unbounded (webhook) path takes all 44 in one call, so the + // split has to come from the planner rather than from the poller's file budget. + const { env, aiCalls, upsertedIds } = mkEnv(); + const commit = mkCommit( + Array.from({ length: 44 }, (_, i) => mkDiffPatch(i % 4 === 0 ? 4200 : 700)), + ); + + const result = await processAndUpsertCommitDiff(env, mkStore(), REPO, commit); + + expect(aiCalls.length).toBeGreaterThan(1); + for (const call of aiCalls) { + expect(callCharge(call)).toBeLessThanOrEqual(WORKERS_AI_BATCH_CONTEXT_LIMIT); + } + + expect(result.embedded).toBe(44); + expect(result.failed).toBe(0); + expect(aiCalls.flat()).toHaveLength(44); + expect(new Set(upsertedIds.flat()).size).toBe(44); + }); + it("keeps each embed call paired with its own slice of files", async () => { // Vectors are matched to files by position, so a split has to cut the inputs // and the files on the same boundary. Reading each call's file paths back out diff --git a/src/pipeline/embed-diff.ts b/src/pipeline/embed-diff.ts index e11ae93..072963e 100644 --- a/src/pipeline/embed-diff.ts +++ b/src/pipeline/embed-diff.ts @@ -126,7 +126,7 @@ function normaliseFileStatus(status: string): DiffFileStatus { * take at most `options.maxFiles` of what remains. * 3. Build embedding inputs = commit message + file path + patch, truncated. * 4. Batch-embed inputs via Workers AI (chunked by `planEmbeddingBatches`, which - * splits on a UTF-8 byte budget rather than a file count). + * splits on input count times longest input rather than on a file count). * 5. Upsert all vectors into Vectorize in the same chunks. * 6. Record DiffRecord rows into the Durable Object store for each indexed file. * @@ -196,16 +196,15 @@ export async function processAndUpsertCommitDiff( prepareDiffEmbeddingInput(commitMessage, f.filename, f.patch), ); - // Chunk on the UTF-8 byte total of the inputs, not on how many files the commit - // touched. A file count bounds a call only if every patch is assumed to be - // small, and a commit that breaks that assumption used to fail the whole chunk — - // which the poller reads as an uningested commit and holds the diff watermark on, - // so the surface stalls there rather than skipping past it (#236). Bytes rather - // than estimated tokens, which ran about 2.1x low on diff patches (#241), and - // rather than characters, which production measured at 1.146 tokens per character - // on non-ASCII content: byte fallback splits a character the vocabulary lacks into - // its UTF-8 bytes, so only bytes are below every split the tokenizer can make - // (#244). + // Chunk on what the endpoint charges a batch — its input count times its longest + // input, measured in UTF-8 bytes — not on how many files the commit touched. A + // file count bounds a call only if every patch is assumed to be small, and a + // commit that breaks that assumption used to fail the whole chunk — which the + // poller reads as an uningested commit and holds the diff watermark on, so the + // surface stalls there rather than skipping past it (#236). The three budgets + // before this one bounded the *sum* of the inputs and kept being rejected at the + // same batches: the endpoint pads each slot to the longest input, so the sum never + // appears in the charge, whatever unit it is summed in (#246). for (const { start, end } of planEmbeddingBatches(allInputs)) { const chunk = indexable.slice(start, end); const inputs = allInputs.slice(start, end); diff --git a/src/pipeline/embedding.test.ts b/src/pipeline/embedding.test.ts index 3003db1..96d376f 100644 --- a/src/pipeline/embedding.test.ts +++ b/src/pipeline/embedding.test.ts @@ -2,23 +2,35 @@ import { describe, it, expect } from "vitest"; import { planEmbeddingBatches, utf8ByteLength, - MAX_EMBEDDING_BATCH_BYTES, MAX_EMBEDDING_INPUT_CHARS, TOKEN_OVERHEAD_PER_INPUT, WORKERS_AI_BATCH_CONTEXT_LIMIT, } from "./embedding.js"; -/** What the planner charges one batch: the inputs' UTF-8 bytes plus their per-input - * token overhead. An upper bound on the batch's true token count, not an estimate. */ +/** What the endpoint charges one batch: its input count times its longest input, + * each input measured in UTF-8 bytes plus its per-input token overhead. The + * padding model, so the charge does not fall when short inputs join a batch. */ +function batchCharge(texts: string[]): number { + if (texts.length === 0) return 0; + const longest = Math.max( + ...texts.map((text) => utf8ByteLength(text) + TOKEN_OVERHEAD_PER_INPUT), + ); + return texts.length * longest; +} + function rangeCharge(inputs: string[], range: { start: number; end: number }): number { - return inputs - .slice(range.start, range.end) - .reduce((total, text) => total + utf8ByteLength(text) + TOKEN_OVERHEAD_PER_INPUT, 0); + return batchCharge(inputs.slice(range.start, range.end)); +} + +/** What the retired byte-sum budget charged the same inputs (#244). Kept in the + * tests only, to hold the regression fixtures inside the shape that budget passed + * and the endpoint rejected. */ +function retiredSumCharge(texts: string[]): number { + return texts.reduce((total, text) => total + utf8ByteLength(text) + TOKEN_OVERHEAD_PER_INPUT, 0); } /** What the retired character budget charged the same inputs (#242): UTF-16 code - * units plus two special tokens. Kept in the tests only, to hold the regression - * fixtures inside the shape that budget passed and the endpoint rejected. */ + * units plus two special tokens. */ function retiredCharCharge(inputs: string[]): number { return inputs.reduce((total, text) => total + text.length + 2, 0); } @@ -57,13 +69,25 @@ describe("utf8ByteLength", () => { }); }); -describe("MAX_EMBEDDING_BATCH_BYTES", () => { - it("does not exceed the endpoint's aggregate ceiling", () => { - // The whole guarantee rests on this: no token spans less than one UTF-8 byte of - // the input, byte fallback being the finest split there is, so a batch charged - // under the ceiling *in bytes* is under it in tokens too. Raise this above the - // ceiling and the bound stops holding by construction. - expect(MAX_EMBEDDING_BATCH_BYTES).toBeLessThanOrEqual(WORKERS_AI_BATCH_CONTEXT_LIMIT); +describe("WORKERS_AI_BATCH_CONTEXT_LIMIT", () => { + it("divides the four production rejections into equal per-input slots", () => { + // The measurement the bound is built on. Each rejection's reported figure is its + // input count times an integer, and the last two are the same commit sent at + // different counts with the quotient unchanged — which a true sum of the inputs + // could only produce if the 4 dropped inputs had been identical in length. + const rejections = [ + { inputs: 18, reported: 60678, perInput: 3371 }, + { inputs: 17, reported: 64413, perInput: 3789 }, + { inputs: 20, reported: 85920, perInput: 4296 }, + { inputs: 16, reported: 68736, perInput: 4296 }, + ]; + + for (const { inputs, reported, perInput } of rejections) { + expect(inputs * perInput).toBe(reported); + expect(reported).toBeGreaterThan(WORKERS_AI_BATCH_CONTEXT_LIMIT); + } + // Same commit, two counts, one quotient: the padded-slot width, not a sum. + expect(rejections[2].perInput).toBe(rejections[3].perInput); }); it("holds a batch of at least 2 maximal inputs", () => { @@ -72,9 +96,11 @@ describe("MAX_EMBEDDING_BATCH_BYTES", () => { // per-file worst case is exactly the 3 the poller assumes. Truncation caps one // input at MAX_EMBEDDING_INPUT_CHARS characters, and a UTF-16 code unit is at // most 3 UTF-8 bytes, so that product is the largest input the planner can face. + // Count times longest and the retired sum agree exactly on a batch of equal + // inputs, which is why moving the bound left this floor where it was. const maxBytesPerInput = MAX_EMBEDDING_INPUT_CHARS * 3; const minInputsPerBatch = Math.floor( - MAX_EMBEDDING_BATCH_BYTES / (maxBytesPerInput + TOKEN_OVERHEAD_PER_INPUT), + WORKERS_AI_BATCH_CONTEXT_LIMIT / (maxBytesPerInput + TOKEN_OVERHEAD_PER_INPUT), ); expect(minInputsPerBatch).toBeGreaterThanOrEqual(2); @@ -83,6 +109,16 @@ describe("MAX_EMBEDDING_BATCH_BYTES", () => { const maximal = Array.from({ length: 20 }, () => "あ".repeat(MAX_EMBEDDING_INPUT_CHARS)); expect(planEmbeddingBatches(maximal)[0].end).toBeGreaterThanOrEqual(minInputsPerBatch); }); + + it("keeps the shortest input in a batch from cheapening it", () => { + // The property that separates this axis from every sum: one large input among + // small ones is charged as if all of them were large, so a batch cannot be made + // to fit by padding it out with cheap inputs. + const one = "あ".repeat(4000); + expect(batchCharge([one])).toBe(utf8ByteLength(one) + TOKEN_OVERHEAD_PER_INPUT); + expect(batchCharge([one, "x"])).toBe(2 * batchCharge([one])); + expect(batchCharge([one, "x", "y", "z"])).toBe(4 * batchCharge([one])); + }); }); describe("planEmbeddingBatches", () => { @@ -95,8 +131,8 @@ describe("planEmbeddingBatches", () => { expect(planEmbeddingBatches(inputs)).toEqual([{ start: 0, end: 50 }]); }); - it("splits when the running total would overrun the budget", () => { - // Charge 12 each (9 bytes + 3 overhead) against a budget of 24. + it("splits when count times longest would overrun the budget", () => { + // Charge 12 each (9 bytes + 3 overhead) against a budget of 24: 2 slots fit. const inputs = Array.from({ length: 5 }, () => "x".repeat(9)); expect(planEmbeddingBatches(inputs, 24)).toEqual([ { start: 0, end: 2 }, @@ -174,12 +210,57 @@ describe("planEmbeddingBatches", () => { const ranges = planEmbeddingBatches(inputs); expect(ranges.length).toBeGreaterThan(1); for (const range of ranges) { - expect(rangeCharge(inputs, range)).toBeLessThanOrEqual(MAX_EMBEDDING_BATCH_BYTES); + expect(rangeCharge(inputs, range)).toBeLessThanOrEqual(WORKERS_AI_BATCH_CONTEXT_LIMIT); } // Nothing is dropped on the way through the split. expect(ranges[ranges.length - 1].end).toBe(16); }); + it("splits the production shape a byte-sum budget passed whole", () => { + // 2026-08-15 cron, `neuron-graph-rag@1fb0f6b`: 18 files, one long patch and a + // tail of short ones. The byte-sum budget charged the batch about a fifth of the + // ceiling and sent all 18 as one call; the endpoint answered `Max context reached + // 60678 tokens but model supports only 60000` — 18 × 3371, the longest input + // charged 18 times over. No sum-axis budget can see this batch, however tightly + // it is tuned, which is what separates this fixture from the three before it. + const longest = "x".repeat(3368); + const inputs = [longest, ...Array.from({ length: 17 }, () => "x".repeat(500))]; + + // The fixture is only a regression test while it stays in the failing zone: well + // inside the retired byte-sum budget, well over the ceiling once padded. + expect(retiredSumCharge(inputs)).toBeLessThanOrEqual(WORKERS_AI_BATCH_CONTEXT_LIMIT); + expect(batchCharge(inputs)).toBeGreaterThan(WORKERS_AI_BATCH_CONTEXT_LIMIT); + expect(batchCharge(inputs)).toBe(18 * 3371); + + const ranges = planEmbeddingBatches(inputs); + expect(ranges.length).toBeGreaterThan(1); + for (const range of ranges) { + expect(rangeCharge(inputs, range)).toBeLessThanOrEqual(WORKERS_AI_BATCH_CONTEXT_LIMIT); + } + expect(ranges[ranges.length - 1].end).toBe(18); + }); + + it("reopens a batch when a later input widens the padding under it", () => { + // The failure a running-sum planner cannot express: the first 17 inputs are + // admissible together, and the 18th makes them inadmissible *retroactively* by + // raising the width every slot is charged at. The cut therefore has to be + // decided against the range's own longest member, recomputed as the range grows. + const inputs = [...Array.from({ length: 17 }, () => "x".repeat(97)), "x".repeat(997)]; + + expect(retiredSumCharge(inputs)).toBeLessThanOrEqual(2700); + const ranges = planEmbeddingBatches(inputs, 2700); + + // 17 slots of 100 is 1700 and admissible; admitting the 18th would charge all + // 18 slots at 1000, which is 18000 — the range has to close before it. + expect(ranges).toEqual([ + { start: 0, end: 17 }, + { start: 17, end: 18 }, + ]); + for (const range of ranges) { + expect(rangeCharge(inputs, range)).toBeLessThanOrEqual(2700); + } + }); + it("gives an over-budget input a call of its own instead of stalling", () => { // The guard that makes this pass is what keeps the planner from closing an // empty range and looping on the same index forever. @@ -198,7 +279,7 @@ describe("planEmbeddingBatches", () => { // widest encoding a truncated input can reach: 3 bytes per code unit. const maximal = "あ".repeat(MAX_EMBEDDING_INPUT_CHARS); expect(utf8ByteLength(maximal) + TOKEN_OVERHEAD_PER_INPUT).toBeLessThanOrEqual( - MAX_EMBEDDING_BATCH_BYTES, + WORKERS_AI_BATCH_CONTEXT_LIMIT, ); expect(planEmbeddingBatches([maximal])).toEqual([{ start: 0, end: 1 }]); }); diff --git a/src/pipeline/embedding.ts b/src/pipeline/embedding.ts index 5e4fc14..a041c9b 100644 --- a/src/pipeline/embedding.ts +++ b/src/pipeline/embedding.ts @@ -13,20 +13,34 @@ * byte fallback breaks the same way it broke the batch budget below: 8000 * characters of Japanese can decompose into far more tokens than that window holds. * Whether the per-input axis needs a byte cap of its own is a separate question - * from the batch total this file bounds — the per-input window is enforced by the - * model, while the batch total is what the endpoint rejects outright. */ + * from the batch charge this file bounds — the per-input window is enforced by the + * model, while the batch charge is what the endpoint rejects outright. The two do + * meet in one place: this cap is what puts a ceiling on the longest input a batch + * can hold, and that ceiling is what guarantees a batch of at least 2. */ export const MAX_EMBEDDING_INPUT_CHARS = 8000; /** - * Aggregate context the Workers AI endpoint accepts across all inputs of one - * batched embed call. Not the per-input maximum bge-m3 documents (8192) — the - * batch is summed, and the endpoint reports the sum it rejected: + * Context the Workers AI endpoint accepts for one batched embed call. Not the + * per-input maximum bge-m3 documents (8192) — the endpoint charges the call as a + * whole and names the figure it rejected: * * 3030: Max context reached 85920 tokens but model supports only 60000 * - * Unpublished, so this is read off the error rather than a docs page. Kept named - * because the byte budget below is derived from it, and a budget whose reference - * is inlined reads as an arbitrary number the next time someone retunes it. + * What that figure counts is `input count × the longest input`, not the sum of the + * inputs. Four rejections divide exactly by their input counts, and two of them are + * the same commit sent at different counts: + * + * 18 inputs -> 60678 = 18 × 3371 + * 17 inputs -> 64413 = 17 × 3789 + * 20 inputs -> 85920 = 20 × 4296 + * 16 inputs -> 68736 = 16 × 4296 + * + * The last pair is what settles it: dropping 4 inputs left the per-input quotient + * unchanged at 4296, which a true sum could only do if the 4 dropped inputs were + * identical in length. The endpoint pads a batch to its longest member and charges + * every slot at that width. + * + * Unpublished, so this is read off the errors rather than a docs page. */ export const WORKERS_AI_BATCH_CONTEXT_LIMIT = 60000; @@ -79,53 +93,35 @@ export function utf8ByteLength(text: string): number { } /** - * UTF-8 byte budget for the inputs of one batched Workers AI embed call. + * What one input occupies in a padded slot of a batch, in UTF-8 bytes plus its + * per-input token overhead. * - * The ceiling itself, in bytes, because bytes *dominate* tokens rather than - * approximating them: the finest split any of these tokenizers can make is one - * token per byte of the UTF-8 input — that is what byte fallback is — so for any - * input + * Bytes because they *dominate* the token count rather than approximating it: the + * finest split any of these tokenizers can make is one token per byte of the UTF-8 + * input — that is what byte fallback is — so for any input * * tokens(input) <= utf8ByteLength(input) + TOKEN_OVERHEAD_PER_INPUT * - * and a batch whose charged total stays inside this budget is inside the ceiling. - * No calibration and no per-payload measurement, which is the property the two - * budgets before this one were reaching for and missed: - * - * - an estimated token count ran about 2.1x optimistic on diff patches and handed - * the endpoint 60678 and 64413 tokens against the ceiling; - * - a character count rested on "a token spans at least one character", which byte - * fallback breaks — a character outside the vocabulary is decomposed into its - * UTF-8 bytes, so one 3-byte Japanese character can cost 3 tokens. Production - * measured 68736 tokens on a batch charged at most 60000 characters, a ratio of - * 1.146 tokens per character. - * - * Bytes are the floor under that decomposition: nothing splits finer, so no payload - * can be the one this underestimates the way the two above were. The bound assumes - * the tokenizer's NFKC normalization does not expand the input in bytes, which holds - * except for compatibility characters that decompose into several (Arabic ligatures, - * CJK square abbreviations). That is a different class of exposure from the two - * retired premises: those broke on ordinary diff text, this one needs an input made - * predominantly of one rare block. - * - * Equality with the ceiling is admissible: the rejections name counts strictly - * above it (`Max context reached 68736 tokens but model supports only 60000`), so - * 60000 is a supported count and not the first rejected one. - * - * Held at the ceiling rather than under it by a margin because a margin is not free - * — every extra batch costs two subrequests (the AI call and its - * `VECTORIZE.upsert`) against an invocation budget this worker already overruns, - * and a bound that holds by construction has nothing left for a margin to buy. - * - * `MAX_EMBEDDING_INPUT_CHARS` truncates one input to 8000 characters, which is at - * most 24000 bytes (3 per UTF-16 code unit is the widest UTF-8 gets; a surrogate - * pair is 4 bytes across 2 units), so a batch holds at least 2 inputs however large - * each patch is. That floor is what the poller's per-file subrequest estimate rests - * on, and it is a third of what the character budget gave — a batch of Japanese - * patches now splits about three ways where it used to be one call. The extra - * subrequests are the price of the bound actually holding. + * No calibration and no per-payload measurement, which is the property two earlier + * per-input measures were reaching for and missed: an estimated token count ran + * about 2.1x optimistic on diff patches (#241), and a character count rested on "a + * token spans at least one character", which byte fallback breaks — a character + * outside the vocabulary is decomposed into its UTF-8 bytes, so one 3-byte Japanese + * character can cost 3 tokens (#244). Bytes sit below that decomposition, so no + * payload can be the one this underestimates the way those two were. What is left + * assumed is that the tokenizer's NFKC normalization does not expand the input in + * bytes, which holds except for compatibility characters that decompose into several + * (Arabic ligatures, CJK square abbreviations) — an input would have to be made + * predominantly of one rare block, where the two retired measures broke on ordinary + * diff text. + * + * This is the per-input axis only. What the endpoint charges for a batch is this + * measure taken over the *longest* input and multiplied by the input count; see + * `planEmbeddingBatches`. */ -export const MAX_EMBEDDING_BATCH_BYTES = WORKERS_AI_BATCH_CONTEXT_LIMIT; +function embeddingInputCost(text: string): number { + return utf8ByteLength(text) + TOKEN_OVERHEAD_PER_INPUT; +} /** Half-open `[start, end)` index range over a caller's input array. */ export interface EmbeddingBatchRange { @@ -134,44 +130,72 @@ export interface EmbeddingBatchRange { } /** - * Split embedding inputs into batches whose UTF-8 byte totals stay within - * `budgetBytes`. - * - * Each input is charged its own byte length plus `TOKEN_OVERHEAD_PER_INPUT`, which - * makes the charged total an upper bound on the batch's true token count rather - * than an estimate of it (see `MAX_EMBEDDING_BATCH_BYTES`). - * - * Index ranges are returned rather than the strings themselves so the caller can - * slice its own parallel arrays (files, metadata) by the same boundaries — the - * position-for-position correspondence between inputs and returned vectors is - * what the upsert depends on. + * Split embedding inputs into batches the endpoint will accept: + * + * (end - start) × max(embeddingInputCost(input) over the range) <= budget + * + * The bound is on `count × longest`, not on the sum. Three budgets before this one + * bounded the sum — by estimated tokens (#236), by characters (#242), by UTF-8 bytes + * (#244) — and the endpoint kept rejecting the same batches, because the sum does + * not appear in what it charges: it pads every slot of a batch to the longest input + * and bills the padded width across the whole count (see + * `WORKERS_AI_BATCH_CONTEXT_LIMIT` for the four measurements that show it). Tighten + * a sum-axis bound however far and it still binds the wrong quantity. + * + * Two properties follow that a sum axis does not have. The largest input sets the + * unit price for the whole batch, so a single large input among small ones is + * charged as if all of them were large; and grouping inputs of similar size is + * therefore what makes a batch efficient. Inputs are nonetheless taken in the order + * given, without sorting: the caller slices its own parallel arrays (files, + * metadata) by the returned boundaries, and the position-for-position + * correspondence between inputs and returned vectors is what the upsert depends on. + * Sorting would buy fewer calls at the cost of an index-set return type and a + * permutation the caller has to carry — and it buys nothing against the worst case + * the poller's subrequest budget is sized on, which is set by `MAX_EMBEDDING_INPUT_CHARS` + * rather than by the packing. + * + * Equality with the budget is admissible: the rejections name charges strictly above + * the ceiling (`Max context reached 60678 tokens but model supports only 60000`), so + * 60000 is a supported charge and not the first rejected one. No margin is held under + * it either — every extra batch costs two subrequests (the AI call and its + * `VECTORIZE.upsert`) against an invocation budget this worker already overruns, and + * a bound that holds by construction has nothing left for a margin to buy. * * Contract: * - order is preserved, ranges are contiguous, and every input falls in exactly one * - no returned range is empty - * - an input whose own charge already exceeds the budget occupies a range of + * - an input whose own cost already exceeds the budget occupies a range of * one. Cutting it down further belongs to the truncation axis * (`MAX_EMBEDDING_INPUT_CHARS`), and dropping it would lose a file from the index. + * - `MAX_EMBEDDING_INPUT_CHARS` truncates one input to 8000 characters, which is at + * most 24000 bytes (3 per UTF-16 code unit is the widest UTF-8 gets; a surrogate + * pair is 4 bytes across 2 units), so `2 × 24003 = 48006` fits the default budget + * and a batch holds at least 2 inputs however large each patch is. That floor is + * what the poller's per-file subrequest estimate rests on, and it is unchanged + * from the byte-sum budget this replaces. */ export function planEmbeddingBatches( inputs: string[], - budgetBytes: number = MAX_EMBEDDING_BATCH_BYTES, + budget: number = WORKERS_AI_BATCH_CONTEXT_LIMIT, ): EmbeddingBatchRange[] { const ranges: EmbeddingBatchRange[] = []; let start = 0; - let total = 0; + let longest = 0; for (let i = 0; i < inputs.length; i++) { - const cost = utf8ByteLength(inputs[i]) + TOKEN_OVERHEAD_PER_INPUT; - // Close the open range before an input that would overrun the budget. The - // `i > start` guard is what keeps an oversized input in a batch of its own - // instead of closing an empty range and looping on it forever. - if (i > start && total + cost > budgetBytes) { + const cost = embeddingInputCost(inputs[i]); + const widest = Math.max(longest, cost); + // Close the open range before an input that would overrun the budget — either + // by its own width or by widening the padding under every input already in the + // range. The `i > start` guard is what keeps an oversized input in a batch of + // its own instead of closing an empty range and looping on it forever. + if (i > start && (i - start + 1) * widest > budget) { ranges.push({ start, end: i }); start = i; - total = 0; + longest = cost; + } else { + longest = widest; } - total += cost; } if (inputs.length > start) ranges.push({ start, end: inputs.length }); diff --git a/src/poller.test.ts b/src/poller.test.ts index 388ce47..036c4ff 100644 --- a/src/poller.test.ts +++ b/src/poller.test.ts @@ -1,9 +1,9 @@ import { describe, it, expect, vi, beforeEach, afterEach } from "vitest"; import type { Env } from "./types.js"; import { - MAX_EMBEDDING_BATCH_BYTES, MAX_EMBEDDING_INPUT_CHARS, TOKEN_OVERHEAD_PER_INPUT, + WORKERS_AI_BATCH_CONTEXT_LIMIT, } from "./pipeline/embedding.js"; // `pollDiffs` fans out to the commit-diff pipeline (GitHub detail fetch + Workers @@ -559,12 +559,14 @@ describe("poller: diffFileBudgetPerPhase", () => { // amortised over the files it holds. Moving the batch budget moves that floor — // it was 7 files under a character budget and is 2 under a byte budget (#244), // since truncation caps an input at MAX_EMBEDDING_INPUT_CHARS characters and a - // UTF-16 code unit is at most 3 UTF-8 bytes. At 2 the sum is exactly 3, so this - // holds without slack and a further tightening of the batch axis fails here - // rather than silently overrunning the invocation budget. + // UTF-16 code unit is at most 3 UTF-8 bytes. The move to count-times-longest + // (#246) left it at 2: on a batch of equal inputs the two formulas agree, and + // maximal inputs is the case this floor is read off. At 2 the sum is exactly 3, + // so this holds without slack and a further tightening of the batch axis fails + // here rather than silently overrunning the invocation budget. const maxBytesPerInput = MAX_EMBEDDING_INPUT_CHARS * 3; const minFilesPerBatch = Math.floor( - MAX_EMBEDDING_BATCH_BYTES / (maxBytesPerInput + TOKEN_OVERHEAD_PER_INPUT), + WORKERS_AI_BATCH_CONTEXT_LIMIT / (maxBytesPerInput + TOKEN_OVERHEAD_PER_INPUT), ); expect(minFilesPerBatch).toBeGreaterThanOrEqual(1); diff --git a/src/poller.ts b/src/poller.ts index b511448..44940b1 100644 --- a/src/poller.ts +++ b/src/poller.ts @@ -208,15 +208,20 @@ const DIFF_SUBREQUEST_BUDGET_PER_RUN = 900; * The third is the amortised batch cost — a batch spends 2 (the Workers AI call * and its `VECTORIZE.upsert`) and holds at least 2 files, because * `MAX_EMBEDDING_INPUT_CHARS` caps one input at 8000 characters, which is at most - * 24000 UTF-8 bytes, against the 60000 bytes `MAX_EMBEDDING_BATCH_BYTES` gives a - * batch. Two files per batch puts the amortised share at exactly 1, so the worst - * case is exactly 3 — no longer rounded up from 2.29 but met on the nose. + * 24000 UTF-8 bytes, and the endpoint charges a batch its input count times its + * longest input: `2 × 24003 = 48006` against the 60000 of + * `WORKERS_AI_BATCH_CONTEXT_LIMIT`. Two files per batch puts the amortised share + * at exactly 1, so the worst case is exactly 3 — no longer rounded up from 2.29 + * but met on the nose. * * That floor moved with the batch axis: 7 files while the budget was counted in - * characters (#241), 2 now that it is counted in UTF-8 bytes (#244), because a - * Japanese character is charged 3 bytes where it used to be charged 1. The - * constant is unchanged and the derivation still holds, but it holds without slack - * — a further tightening of the batch axis lands here rather than being absorbed. */ + * characters (#241), 2 once it was counted in UTF-8 bytes (#244), because a + * Japanese character is charged 3 bytes where it used to be charged 1. Moving the + * bound off the input sum and onto count-times-longest (#246) left it at 2: the + * two formulas agree exactly when every input is maximal, which is the case this + * floor is read off. The constant is unchanged and the derivation still holds, but + * it holds without slack — a further tightening of the batch axis lands here + * rather than being absorbed. */ const DIFF_SUBREQUESTS_PER_FILE = 3; /** Per-phase subrequests that are not per-file: up to 5 commit detail fetches,