From a414c7d056193345d6bb0ddf22d6d27b5125d510 Mon Sep 17 00:00:00 2001 From: Claude Lin & Lay Date: Fri, 14 Aug 2026 19:34:56 +0900 Subject: [PATCH 1/4] fix(pipeline): bound embed batches by character count, not estimated tokens [pipeline, poller, docs, tests] MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit batch の切れ目を `estimateEmbeddingTokens` による推定から文字数へ移す。 #237 の推定器は ASCII を 3 文字/token として読むが、bge-m3 が diff patch を 割る密度は約 1.4 文字/token である。`+` / `-` 接頭、インデント、記号、短い 識別子がいずれも細かく分割されるため、推定は約 2.1x 楽観に振れていた。 #240 merge 後の 16:31 JST cron でも、18 file / 17 file の 2 commit が 「30000 token 予算に収まる」と判定されて単一 batch のまま送られ、実測 60678 / 64413 token で天井 60000 を超えて chunk 全体が失敗している。 #178 の不変条件により、vector が載らなかった commit で diff watermark が 留まるので、同じ commit が毎 cron 同じ形で落ち続けていた。 置き換えの根拠は上界であって校正ではない。BPE / SentencePiece のいずれでも 1 token は入力の 1 文字以上に対応するので、各 input に「文字数 + special token 2 個」を課金すれば、batch が天井の内側に収まることが無条件に成立する。 payload の言語にも記号密度にも依存しないため、次に想定外の payload が来ても 同じ破れ方をしない。special token を batch 固定の引当てではなく input ごとに 課金するのは、batch の input 件数に上限が無く、極小 input が多数並ぶ場合に 固定引当てでは足りなくなるためである。 - `MAX_EMBEDDING_BATCH_TOKENS` / `estimateEmbeddingTokens` / `ASCII_CHARS_PER_TOKEN` を退役。`MAX_EMBEDDING_BATCH_CHARS` （= `WORKERS_AI_BATCH_CONTEXT_LIMIT`）と `SPECIAL_TOKENS_PER_INPUT` を導入 - `WORKERS_AI_BATCH_CONTEXT_LIMIT` は天井の記録として維持し、文字数予算の 根拠として参照する。天井の下に margin は取らない — 構成上成立する上界に margin が買えるものは残っていない - truncate が 1 input を 8000 文字に抑えるので 1 batch は最低 7 file を保持 する。#240 の `DIFF_SUBREQUESTS_PER_FILE = 3` が立っていた床（3 file）より 良い側なので、file 予算は変更していない。derivation の comment のみ更新 - 実測 payload 相当（18 file、単一 call では天井超え、かつ退役した推定では 30000 token 予算に収まる）の回帰 test を追加。fixture が再現領域から 外れないよう両側を assert している - `MAX_EMBEDDING_BATCH_CHARS` / `MAX_EMBEDDING_INPUT_CHARS` の比が最小 file 数であることを test で押さえた - 入力の順序保持、input 配列と file 配列を同じ境界で切る契約、単独で予算を 超える input を単独 batch として送る扱いは変更していない #241 --- docs/0-requirements.ja.md | 2 +- docs/0-requirements.md | 2 +- src/pipeline/embed-diff.test.ts | 56 +++++++++++++-- src/pipeline/embed-diff.ts | 14 ++-- src/pipeline/embedding.test.ts | 116 +++++++++++++++++--------------- src/pipeline/embedding.ts | 100 +++++++++++++-------------- src/poller.ts | 10 +-- 7 files changed, 179 insertions(+), 121 deletions(-) diff --git a/docs/0-requirements.ja.md b/docs/0-requirements.ja.md index efc0176..b8a224b 100644 --- a/docs/0-requirements.ja.md +++ b/docs/0-requirements.ja.md @@ -255,7 +255,7 @@ Responsibilities: - metadata のみの経路（body は変わらず state / labels / milestone / assignees が変わった場合）では、mirror 書き込みの失敗を best-effort 扱いに**しない**。差分検出の基準を進めずに保持し、次の poll / webhook 配信で再試行させる。基準は IssueStore の record そのものなので、失敗した mirror を追い越して基準を進めると取り残しが恒久化する — state だけの変更は、embed 経路が待っている body 変更を二度と連れてこない（issue #209） - この経路の dense / sparse mirror は互いに独立して書く。vector が欠けている行（issue #210）でも sparse 側の state は更新される - commit diff は 1 commit 分の file リストを batch embed（Workers AI の `text: string[]` 対応を利用）し、batch ごとに 1 回の Vectorize upsert で N vector を書き込む -- batch の切れ目は file 件数ではなく推定 token 予算（`MAX_EMBEDDING_BATCH_TOKENS`）で決める。この pipeline が embed する内容の「1 token あたりの文字数」は桁で変わる — ASCII のソースで約 3、CJK の散文で約 1 — ため、件数が call を縛れるのは「どの patch も安い側である」と仮定した場合だけである。その仮定を破る commit は chunk 全体を失敗させ、vector が載らなかった commit は diff watermark が留まる対象（issue #178）なので、その commit を飛ばすのではなく surface がそこで恒久的に停止していた。しかも毎 cron 同じ commit で同じ結果になる決定論的な失敗で、同じログに同居する一過性の subrequest 超過とはそこで性質が分かれる（issue #236）。天井は bge-m3 の documented な per-input 上限 8192 token ではなく、1 call の input 全体を合算した endpoint 側の上限である。batch は合算され、拒否応答はその合計値を名指しする（`3030: Max context reached 85920 tokens but model supports only 60000`）。この値は非公開なので `WORKERS_AI_BATCH_CONTEXT_LIMIT` としてそのエラーから記録し、予算はその半分に置く。半分にしているのは効く方向の誤差が 1 つだけだからで、`estimateEmbeddingTokens` は近似であり、真値より小さく出た推定こそが call を天井の向こうへ送って stall を再現する。同時に天井から離しすぎない — batch が 1 つ増えるごとに subrequest を 2 つ消費し、この worker は既に invocation 予算を超過しているため。単独で予算を超える input はそのまま単独で送る — さらに削るのは truncate 軸（`MAX_EMBEDDING_INPUT_CHARS`）の仕事であり、落とせばその file が索引から消えるため +- batch の切れ目は file 件数でも token 数の推定でもなく、文字数予算（`MAX_EMBEDDING_BATCH_CHARS`）で決める。天井は bge-m3 の documented な per-input 上限 8192 token ではなく、1 call の input 全体を合算した endpoint 側の上限である。batch は合算され、拒否応答はその合計値を名指しする（`3030: Max context reached 85920 tokens but model supports only 60000`）。この値は非公開なので `WORKERS_AI_BATCH_CONTEXT_LIMIT` としてそのエラーから記録し、文字数予算はその天井そのものに置く。文字数が正しい単位である理由は、それが token 数の近似ではなく**上界**だからである。BPE / SentencePiece のいずれでも 1 token は入力の 1 文字以上に対応するので、各 input にその文字数と special token 2 個（`` … ``）を課金すれば、batch が天井の内側に収まることが無条件に成立する — 校正が要らず、payload の言語や記号密度にも依存しない。special token を 1 batch 固定の引当てではなく input ごとに課金するのは、batch の input 件数に上限が無く、極小 input が多数並ぶ場合に固定引当てでは足りなくなるためである。file 件数ではこれを表現できず、直前に置いていた推定でも表現できていなかった。推定は ASCII を 3 文字/token として読むが、bge-m3 が diff patch を割る密度は約 1.4 文字/token である — `+` / `-` 接頭、インデント、記号、短い識別子がいずれも細かく分割される — ため、肝心の surface で約 2.1x 楽観に振れ、実測 60678 / 64413 token の batch を「30000 token 予算に収まる」と判定して通していた（issue #241）。天井を超えた commit は chunk 全体を失敗させ、vector が載らなかった commit は diff watermark が留まる対象（issue #178）なので、その commit を飛ばすのではなく surface がそこで恒久的に停止する。しかも毎 cron 同じ commit で同じ結果になる決定論的な失敗で、同じログに同居する一過性の subrequest 超過とはそこで性質が分かれる（issue #236）。天井の下に margin は取らない — 構成上成立する上界に margin が買えるものは残っておらず、batch が 1 つ増えるごとに subrequest を 2 つ消費し、この worker は既に invocation 予算を超過しているためである。単独で予算を超える input はそのまま単独で送る — さらに削るのは truncate 軸（`MAX_EMBEDDING_INPUT_CHARS`）の仕事であり、落とせばその file が索引から消えるため。その truncate が 1 input を 8000 文字に抑えるので、payload の中身によらず 1 batch は最低 7 file を保持する。これが poller の per-file subrequest 見積りが立っている床である **索引欠落の修復.** watermark の修正は漏れを止めるだけで、既に空いた穴は埋まらない — 取り残された項目が再 fetch されるのは `updated_at` が動いたときだけで、閉じた履歴はもう動かない。`POST /admin/backfill-issue-index?repo=owner/repo`（installation guide 参照）が欠落そのものを走査する。repository の issue 番号空間は密かつ有界なので、`search_docs` に issue / PR 行が無い番号がそのまま欠落集合であり、数値 cursor が「どこまで走査したか」を厳密に表せる。同じ集合を時刻 cursor で辿ると、欠陥が突いた順序をそのまま持ち込むことになる。GitHub 側に既に無い番号（削除・transfer 済み）は 404 を返すので、retry せず計上のみ。取り込みは body-hash 判定を強制的に飛ばす: 候補はいずれも retrieval surface が欠けていると分かっている項目であり、hash が一致していると（embed 成功後に FTS5 mirror が失敗した行がこの状態になる）そのまま恒久的に skip されてしまうため。state 修復と違いこちらは embed を伴い、1 候補ごとに取り込み fan-out の全額がかかるので、呼び出し側が batch を跨いで sweep を進める。`dry_run=true` は予算を使わずに欠落量だけを測る。sweep は poller の watermark と同じ不変条件に従う: **取り込みそこねた最初の候補を cursor が追い越さない。** これにより 1 call の上限を大きく取りすぎたときの代償は 1 回分の無駄な呼び出しであって取りこぼしではなく、結果が「上限値が subrequest 予算をどれだけ正確に写しているか」に依存しなくなる（issue #216）。GitHub 側に既に無い番号は例外 — 誰が何回試しても取り込めないので、そこで止めると retry の境界にならず sweep が停止する。トレードオフは poller と同じで、恒久的に失敗する候補があると sweep は止まる。ただしそれは応答に現れる（`nextCursor` が渡した `cursor` と同じ値で返る）し、この endpoint は cron ではなく人間 / AI が駆動するので、詰まった番号を手動の `cursor` 指定で越えられる。 diff --git a/docs/0-requirements.md b/docs/0-requirements.md index fa994bc..e76d0ec 100644 --- a/docs/0-requirements.md +++ b/docs/0-requirements.md @@ -260,7 +260,7 @@ Responsibilities: - on the metadata-only path (state / labels / milestone / assignees changed, body did not) a failed mirror write is **not** best-effort: the diff baseline is held so the next poll or webhook delivery retries. The baseline is the IssueStore record itself, so advancing it past a failed mirror makes the miss permanent — a state-only change never brings the body change the embed path waits for (issue #209) - the dense and sparse mirrors on that path are written independently: a row with no vector (issue #210) still gets its sparse state updated - for commit diffs: batch-embed a commit's file list through Workers AI (`text: string[]`) and upsert each batch's N vectors in one `VECTORIZE.upsert` call -- batches are cut on an estimated token budget (`MAX_EMBEDDING_BATCH_TOKENS`), not on a file count. Characters per token vary by an order of magnitude across what this pipeline embeds — roughly 3 for ASCII source, roughly 1 for CJK prose — so N inputs bound a call only where every patch is assumed to be the cheap kind. A commit that broke the assumption failed its whole chunk, and a commit whose vectors never landed is one the diff watermark holds on (issue #178), so the surface stalled there permanently instead of passing it by — deterministically, on the same commit every cron tick, which is what separated it from the transient subrequest overruns sharing the log (issue #236). The ceiling is the endpoint's aggregate context across a call's inputs, not bge-m3's documented 8192-token per-input maximum: the batch is summed, and the rejection names the sum (`3030: Max context reached 85920 tokens but model supports only 60000`). It is unpublished, so `WORKERS_AI_BATCH_CONTEXT_LIMIT` records it from that error, and the budget is half of it. The halving covers the one error direction that matters — `estimateEmbeddingTokens` approximates, and an estimate landing under the true count is what puts a call over the ceiling and reproduces the stall — while staying near enough the ceiling to keep the batch count down, since every extra batch spends two subrequests on an invocation budget this worker already overruns. An input whose own estimate exceeds the budget is sent alone — cutting it down further is the truncation axis (`MAX_EMBEDDING_INPUT_CHARS`), and dropping it would lose the file from the index +- batches are cut on a character budget (`MAX_EMBEDDING_BATCH_CHARS`), not on a file count and not on an estimate of the token count. The ceiling is the endpoint's aggregate context across a call's inputs, not bge-m3's documented 8192-token per-input maximum: the batch is summed, and the rejection names the sum (`3030: Max context reached 85920 tokens but model supports only 60000`). It is unpublished, so `WORKERS_AI_BATCH_CONTEXT_LIMIT` records it from that error, and the character budget is that ceiling. What makes characters the right unit is that they *dominate* the token count rather than approximating it: every token of a BPE or SentencePiece vocabulary spans at least one character of the input, so charging each input its length plus its two special tokens (`` … ``) puts the batch inside the ceiling unconditionally — no calibration, and no dependence on the payload's language or punctuation density. The special tokens are charged per input rather than as one flat reserve because nothing bounds a batch's input count, and many tiny inputs is where a flat reserve comes up short. A file count cannot express any of this, and neither could the estimate that preceded it: it read ASCII at 3 characters per token, while bge-m3 splits a diff patch nearer 1.4 — `+`/`-` prefixes, indentation, punctuation and short identifiers all tokenize small — so it ran about 2.1x optimistic on the one surface that mattered and passed 60678- and 64413-token batches as fitting a 30000-token budget (issue #241). A commit that goes over fails its whole chunk, and a commit whose vectors never landed is one the diff watermark holds on (issue #178), so the surface stalls there permanently instead of passing it by — deterministically, on the same commit every cron tick, which is what separates it from the transient subrequest overruns sharing the log (issue #236). No margin is held under the ceiling: a bound that holds by construction has nothing left for a margin to buy, and every extra batch spends two subrequests on an invocation budget this worker already overruns. An input whose own charge exceeds the budget is sent alone — cutting it down further is the truncation axis (`MAX_EMBEDDING_INPUT_CHARS`), and dropping it would lose the file from the index. That same truncation caps one input at 8000 characters, so a batch holds at least 7 files whatever the payload is made of, which is the floor the poller's per-file subrequest estimate rests on **Missing-entry repair.** The watermark fix stops the leak but does not fill the hole: a stranded item is only re-fetched when its `updated_at` moves, and closed history never moves again. `POST /admin/backfill-issue-index?repo=owner/repo` (see the installation guide) walks the gap directly — the repository's issue-number space is dense and bounded, so the numbers with no `search_docs` issue / PR row are exactly the missing set, and a numeric cursor states how far the sweep has reached. A timestamp cursor over the same set would reintroduce the ordering the defect exploited. Numbers GitHub no longer has (deleted or transferred) answer 404 and are counted rather than retried. The ingest is forced past the body-hash check: every candidate is known to be missing a retrieval surface, and a matching hash — which an embed whose FTS5 mirror failed leaves behind — would otherwise skip it permanently. Unlike the state repair this one embeds, so every candidate carries the full ingest fan-out and the caller drives the sweep one batch at a time; `dry_run=true` measures the gap without spending it. The sweep obeys the same invariant as the poller's watermark: **the cursor never advances past the first candidate a call failed to ingest**, so a per-call limit set too high costs a wasted call rather than a missed item, and the result does not depend on how accurately that limit models the subrequest budget (issue #216). A number GitHub no longer has is exempt — nothing will ever ingest it, so holding there would stall the sweep instead of bounding a retry. The tradeoff is the poller's: a candidate that fails on every attempt stops the sweep. Here that is visible rather than silent (`nextCursor` comes back equal to the `cursor` passed in), and because a human or an AI drives this endpoint rather than cron, stepping over the blocking number is a matter of passing the next `cursor` by hand. diff --git a/src/pipeline/embed-diff.test.ts b/src/pipeline/embed-diff.test.ts index ba92b8f..59c6316 100644 --- a/src/pipeline/embed-diff.test.ts +++ b/src/pipeline/embed-diff.test.ts @@ -1,7 +1,11 @@ import { describe, it, expect, vi } from "vitest"; import type { Env } from "../types.js"; import { processAndUpsertCommitDiff, type GitHubCommitDetail } from "./embed-diff.js"; -import { estimateEmbeddingTokens, MAX_EMBEDDING_BATCH_TOKENS } from "./embedding.js"; +import { + MAX_EMBEDDING_BATCH_CHARS, + SPECIAL_TOKENS_PER_INPUT, + WORKERS_AI_BATCH_CONTEXT_LIMIT, +} from "./embedding.js"; import { diffVectorId } from "./vector-id.js"; const REPO = "acme/widgets"; @@ -66,7 +70,19 @@ function mkStore() { } as unknown as DurableObjectStub; } -describe("embed-diff: the batch axis is the token budget, not the file count", () => { +/** A patch of the shape the ceiling rejected: dense punctuation, short identifiers, + * a `+` on every line — where bge-m3 splits far finer than an ASCII prose ratio. */ +function mkDiffPatch(chars: number): string { + const line = "+ const value = obj?.[key] ?? { a: 1, b: [2, 3] };\n"; + return line.repeat(Math.ceil(chars / line.length)).slice(0, chars); +} + +/** What the planner charges one Workers AI call. Upper-bounds its token count. */ +function callCharge(texts: string[]): number { + return texts.reduce((sum, text) => sum + text.length + SPECIAL_TOKENS_PER_INPUT, 0); +} + +describe("embed-diff: the batch axis is the character budget, not the file count", () => { it("still sends an ordinary commit as one call", async () => { // 30 files well past the retired count cap of 20, each a small patch. const { env, aiCalls } = mkEnv(); @@ -104,8 +120,7 @@ describe("embed-diff: the batch axis is the token budget, not the file count", ( // Every call carrying more than one input stays inside the budget. for (const call of aiCalls) { if (call.length > 1) { - const total = call.reduce((sum, text) => sum + estimateEmbeddingTokens(text), 0); - expect(total).toBeLessThanOrEqual(MAX_EMBEDDING_BATCH_TOKENS); + expect(callCharge(call)).toBeLessThanOrEqual(MAX_EMBEDDING_BATCH_CHARS); } } @@ -114,6 +129,39 @@ describe("embed-diff: the batch axis is the token budget, not the file count", ( expect(new Set(upsertedIds.flat()).size).toBe(20); }); + it("splits the 18-file commit shape that an estimated budget let through", async () => { + // The payload that kept failing after #237 shipped: 18 files of ordinary diff + // patch, no single one of them oversized. bge-m3 tokenizes a patch at roughly + // 1.4 characters per token — `+`/`-` prefixes, indentation, punctuation and + // short identifiers all split small — so the retired estimator's ASCII ratio of + // 3 came in about 2.1x under the truth, judged the whole commit to fit one + // 30000-token batch, and handed the endpoint 60678 tokens against a ceiling of + // 60000. The chunk failed whole, and the diff watermark held on the commit. + const { env, aiCalls, upsertedIds } = mkEnv(); + const commit = mkCommit(Array.from({ length: 18 }, () => mkDiffPatch(4700))); + + const result = await processAndUpsertCommitDiff(env, mkStore(), REPO, commit); + + const allInputs = aiCalls.flat(); + // The fixture is only a regression test while it stays in the failing zone: over + // the ceiling as one call, yet inside the budget the estimator would have + // computed for it (85202 characters of ASCII read as ~28400 tokens at 3 each, + // under the 30000 that #237 set). Both halves are asserted so a later edit to + // the patch size cannot quietly move the fixture out of the shape it reproduces. + expect(callCharge(allInputs)).toBeGreaterThan(WORKERS_AI_BATCH_CONTEXT_LIMIT); + expect(Math.ceil(callCharge(allInputs) / 3)).toBeLessThanOrEqual(30000); + + expect(aiCalls.length).toBeGreaterThan(1); + for (const call of aiCalls) { + expect(callCharge(call)).toBeLessThanOrEqual(MAX_EMBEDDING_BATCH_CHARS); + } + + // Splitting is only the fix if every file still lands. + expect(result.embedded).toBe(18); + expect(result.failed).toBe(0); + expect(new Set(upsertedIds.flat()).size).toBe(18); + }); + it("keeps each embed call paired with its own slice of files", async () => { // Vectors are matched to files by position, so a split has to cut the inputs // and the files on the same boundary. Reading each call's file paths back out diff --git a/src/pipeline/embed-diff.ts b/src/pipeline/embed-diff.ts index 1651b49..8baf032 100644 --- a/src/pipeline/embed-diff.ts +++ b/src/pipeline/embed-diff.ts @@ -126,7 +126,7 @@ function normaliseFileStatus(status: string): DiffFileStatus { * take at most `options.maxFiles` of what remains. * 3. Build embedding inputs = commit message + file path + patch, truncated. * 4. Batch-embed inputs via Workers AI (chunked by `planEmbeddingBatches`, which - * splits on an estimated token budget rather than a file count). + * splits on a character budget rather than a file count). * 5. Upsert all vectors into Vectorize in the same chunks. * 6. Record DiffRecord rows into the Durable Object store for each indexed file. * @@ -196,11 +196,13 @@ export async function processAndUpsertCommitDiff( prepareDiffEmbeddingInput(commitMessage, f.filename, f.patch), ); - // Chunk on the estimated token total of the inputs, not on how many files the - // commit touched. A file count bounds a call only if every patch is assumed to - // be small, and a commit that breaks that assumption used to fail the whole - // chunk — which the poller reads as an uningested commit and holds the diff - // watermark on, so the surface stalls there rather than skipping past it (#236). + // Chunk on the character total of the inputs, not on how many files the commit + // touched. A file count bounds a call only if every patch is assumed to be + // small, and a commit that breaks that assumption used to fail the whole chunk — + // which the poller reads as an uningested commit and holds the diff watermark on, + // so the surface stalls there rather than skipping past it (#236). Characters + // rather than estimated tokens because the estimate ran 2.1x low on diff patches + // and let the same two commits over the ceiling three cron cycles running (#241). for (const { start, end } of planEmbeddingBatches(allInputs)) { const chunk = indexable.slice(start, end); const inputs = allInputs.slice(start, end); diff --git a/src/pipeline/embedding.test.ts b/src/pipeline/embedding.test.ts index afce34d..6888f7f 100644 --- a/src/pipeline/embedding.test.ts +++ b/src/pipeline/embedding.test.ts @@ -1,68 +1,43 @@ import { describe, it, expect } from "vitest"; import { - estimateEmbeddingTokens, planEmbeddingBatches, - MAX_EMBEDDING_BATCH_TOKENS, + MAX_EMBEDDING_BATCH_CHARS, MAX_EMBEDDING_INPUT_CHARS, + SPECIAL_TOKENS_PER_INPUT, WORKERS_AI_BATCH_CONTEXT_LIMIT, } from "./embedding.js"; -/** Sum of the per-input estimates over one planned range. */ -function rangeTokens(inputs: string[], range: { start: number; end: number }): number { +/** What the planner charges one batch: the inputs' characters plus their special + * tokens. An upper bound on the batch's true token count, not an estimate of it. */ +function rangeCharge(inputs: string[], range: { start: number; end: number }): number { return inputs .slice(range.start, range.end) - .reduce((total, text) => total + estimateEmbeddingTokens(text), 0); + .reduce((total, text) => total + text.length + SPECIAL_TOKENS_PER_INPUT, 0); } -describe("estimateEmbeddingTokens", () => { - it("counts ASCII at the assumed characters-per-token ratio", () => { - expect(estimateEmbeddingTokens("x".repeat(300))).toBe(100); +describe("MAX_EMBEDDING_BATCH_CHARS", () => { + it("does not exceed the endpoint's aggregate ceiling", () => { + // The whole guarantee rests on this: a token spans at least one character, so a + // batch charged under the ceiling *in characters* is under it in tokens too. + // Raise this above the ceiling and the bound stops holding by construction and + // goes back to being a calibration — the state that failed twice (#237, #240). + expect(MAX_EMBEDDING_BATCH_CHARS).toBeLessThanOrEqual(WORKERS_AI_BATCH_CONTEXT_LIMIT); }); - it("rounds a partial token up rather than down", () => { - expect(estimateEmbeddingTokens("x")).toBe(1); - expect(estimateEmbeddingTokens("xxxx")).toBe(2); - }); - - it("counts non-ASCII one token per code unit — the CJK worst case", () => { - expect(estimateEmbeddingTokens("あ".repeat(100))).toBe(100); - }); - - it("separates the two ranges inside one mixed input", () => { - // 300 ASCII (100 tokens) + 50 CJK (50 tokens). - expect(estimateEmbeddingTokens("x".repeat(300) + "あ".repeat(50))).toBe(150); - }); - - it("is empty for empty input", () => { - expect(estimateEmbeddingTokens("")).toBe(0); - }); - - it("separates content of equal character length by token cost", () => { - // The whole point of the axis: same chars, different budget consumption. - const ascii = "x".repeat(MAX_EMBEDDING_INPUT_CHARS); - const cjk = "あ".repeat(MAX_EMBEDDING_INPUT_CHARS); - expect(estimateEmbeddingTokens(cjk)).toBeGreaterThan(estimateEmbeddingTokens(ascii)); - }); -}); - -describe("MAX_EMBEDDING_BATCH_TOKENS", () => { - it("keeps a margin against the endpoint's aggregate ceiling", () => { - // Asserted rather than left to the comment at the constant, because the - // pressure on this number runs one way: every batch costs two subrequests on - // an axis this worker already overruns, so the temptation is to walk the - // budget up toward the ceiling. The estimator approximates, and an estimate - // that lands under the true count is what reproduces the stall this batching - // exists to prevent. A failing assertion here is not a verdict that the new - // value is wrong — it says the estimator now has to earn the thinner margin. - expect(MAX_EMBEDDING_BATCH_TOKENS).toBeLessThanOrEqual( - WORKERS_AI_BATCH_CONTEXT_LIMIT / 2, + it("holds a batch of at least 7 maximal inputs", () => { + // The floor the poller's DIFF_SUBREQUESTS_PER_FILE derivation reads off this + // constant: a batch's 2 subrequests amortise over this many files. Truncation + // caps one input at MAX_EMBEDDING_INPUT_CHARS, so the ratio of the two limits + // is the minimum file count, and it must not drop under what the poller assumes. + const minInputsPerBatch = Math.floor( + MAX_EMBEDDING_BATCH_CHARS / (MAX_EMBEDDING_INPUT_CHARS + SPECIAL_TOKENS_PER_INPUT), ); - }); + expect(minInputsPerBatch).toBeGreaterThanOrEqual(7); - it("stays above the per-input maximum a truncated input can reach", () => { - // The floor on the same number: below this, a maximal input could not be sent - // even alone, and the planner would be handing the endpoint a call it rejects. - expect(MAX_EMBEDDING_BATCH_TOKENS).toBeGreaterThanOrEqual(MAX_EMBEDDING_INPUT_CHARS); + // Asserted through the planner as well, so the arithmetic above cannot drift + // away from the behaviour it describes. + const maximal = Array.from({ length: 20 }, () => "x".repeat(MAX_EMBEDDING_INPUT_CHARS)); + expect(planEmbeddingBatches(maximal)[0].end).toBeGreaterThanOrEqual(minInputsPerBatch); }); }); @@ -77,9 +52,9 @@ describe("planEmbeddingBatches", () => { }); it("splits when the running total would overrun the budget", () => { - // 3 tokens each against a budget of 6: two per call, then the remainder. + // Charge 11 each (9 characters + 2 special tokens) against a budget of 22. const inputs = Array.from({ length: 5 }, () => "x".repeat(9)); - expect(planEmbeddingBatches(inputs, 6)).toEqual([ + expect(planEmbeddingBatches(inputs, 22)).toEqual([ { start: 0, end: 2 }, { start: 2, end: 4 }, { start: 4, end: 5 }, @@ -106,11 +81,37 @@ describe("planEmbeddingBatches", () => { for (const range of ranges) { if (range.end - range.start > 1) { - expect(rangeTokens(inputs, range)).toBeLessThanOrEqual(1000); + expect(rangeCharge(inputs, range)).toBeLessThanOrEqual(1000); } } }); + it("charges the special tokens each input carries, not one flat reserve", () => { + // Many tiny inputs is the case a per-batch reserve gets wrong: 60 characters of + // text, but 120 special tokens on top. A planner charging text alone would fit + // all 60 in one call of 60 and hand the endpoint 180. + const inputs = Array.from({ length: 60 }, () => "x"); + const ranges = planEmbeddingBatches(inputs, 60); + + expect(ranges.length).toBeGreaterThan(1); + for (const range of ranges) { + expect(rangeCharge(inputs, range)).toBeLessThanOrEqual(60); + } + }); + + it("plans the same way for scripts of the same length", () => { + // The property the retired estimator did not have. It read ASCII at 3 characters + // per token and non-ASCII at 1, so the same character count planned differently + // by script — and the ASCII figure was the one that ran low on diff patches. A + // character bound is blind to what the characters are, which is why no payload + // can be the one it underestimates. + const chars = 12000; + const ascii = Array.from({ length: 12 }, () => "x".repeat(chars)); + const cjk = Array.from({ length: 12 }, () => "あ".repeat(chars)); + + expect(planEmbeddingBatches(cjk)).toEqual(planEmbeddingBatches(ascii)); + }); + it("gives an over-budget input a call of its own instead of stalling", () => { // The guard that makes this pass is what keeps the planner from closing an // empty range and looping on the same index forever. @@ -123,10 +124,13 @@ describe("planEmbeddingBatches", () => { }); it("admits any single truncated input under the default budget", () => { - // Truncation caps an input at MAX_EMBEDDING_INPUT_CHARS; at the CJK worst case - // that is one token per character, and the default budget must still take it. + // Truncation caps an input at MAX_EMBEDDING_INPUT_CHARS, and the default budget + // must take one of those alone — otherwise the planner hands the endpoint a + // single-input call it rejects, with nowhere further to split. const maximal = "あ".repeat(MAX_EMBEDDING_INPUT_CHARS); - expect(estimateEmbeddingTokens(maximal)).toBeLessThanOrEqual(MAX_EMBEDDING_BATCH_TOKENS); + expect(maximal.length + SPECIAL_TOKENS_PER_INPUT).toBeLessThanOrEqual( + MAX_EMBEDDING_BATCH_CHARS, + ); expect(planEmbeddingBatches([maximal])).toEqual([{ start: 0, end: 1 }]); }); }); diff --git a/src/pipeline/embedding.ts b/src/pipeline/embedding.ts index 0092c88..b7036aa 100644 --- a/src/pipeline/embedding.ts +++ b/src/pipeline/embedding.ts @@ -17,59 +17,55 @@ export const MAX_EMBEDDING_INPUT_CHARS = 8000; * 3030: Max context reached 85920 tokens but model supports only 60000 * * Unpublished, so this is read off the error rather than a docs page. Kept named - * because the budget below is a margin against it, and a margin whose reference - * is inlined reads as an arbitrary number the next time someone retunes it. + * because the character budget below is derived from it, and a budget whose + * reference is inlined reads as an arbitrary number the next time someone + * retunes it. */ export const WORKERS_AI_BATCH_CONTEXT_LIMIT = 60000; /** - * Token budget for the inputs of one batched Workers AI embed call. + * Special tokens the model wraps around each input of a batch (`` … ``). * - * Half the ceiling above. The halving is sized to the one error direction that - * matters: `estimateEmbeddingTokens` approximates, and an estimate that comes in - * *under* the true count is what puts a call over the ceiling — which fails the - * whole chunk, and a commit whose vectors never landed is one the diff watermark - * holds on, so the surface stalls there rather than passing it by. Punctuation- - * dense payloads (lockfile hashes, minified sources) are where the ASCII ratio - * below runs optimistic, and 2x covers that class with room left. - * - * The margin is not free and is not larger than it needs to be. Every extra batch - * costs two subrequests (the AI call and its `VECTORIZE.upsert`) against an - * invocation budget this worker already overruns, so a budget far below the - * ceiling buys no safety and spends a neighbouring axis that is genuinely tight. - * - * A count cap cannot express any of this. Characters per token vary by an order of - * magnitude across the content this pipeline embeds — roughly 3 for ASCII source, - * roughly 1 for CJK prose — so N inputs bound the request only when every input - * is assumed to be the cheap kind. + * The only part of a batch's token count that does not come from the input text, + * so it is the only part a character count cannot cover. It is charged per input + * rather than once per batch because the batch's input count is not bounded + * anywhere: a batch of many tiny inputs is where a single flat reserve would come + * up short, and that is exactly the case a fixed constant hides. */ -export const MAX_EMBEDDING_BATCH_TOKENS = WORKERS_AI_BATCH_CONTEXT_LIMIT / 2; +export const SPECIAL_TOKENS_PER_INPUT = 2; /** - * Characters per token assumed for the ASCII range. bge-m3 tokenizes with an - * XLM-RoBERTa SentencePiece vocabulary, where English prose runs near 4 and - * punctuation-dense source code runs nearer 3. The lower figure is used because - * the diff surface is source code and an underestimate is what overruns a call. - */ -const ASCII_CHARS_PER_TOKEN = 3; - -/** - * Estimate the token cost of one embedding input. + * Character budget for the inputs of one batched Workers AI embed call. + * + * The ceiling itself, in characters, because characters *dominate* tokens rather + * than approximating them: every token of a BPE or SentencePiece vocabulary spans + * at least one character of the input, so for any input + * + * tokens(input) <= input.length + SPECIAL_TOKENS_PER_INPUT * - * Deliberately an estimate: the tokenizer is not available inside the Worker, and - * the value is only ever compared against a budget that is itself conservative. - * Non-ASCII code units are counted one token each (the CJK worst case), ASCII at - * `ASCII_CHARS_PER_TOKEN`. Counting UTF-16 code units rather than code points - * makes a surrogate pair cost two, which errs toward the safe side. + * and a batch whose charged total stays inside this budget is inside the ceiling + * unconditionally — no calibration, and no dependence on the payload's language or + * punctuation density. Which matters because the estimate this replaced was wrong + * by about 2.1x on the surface that actually failed: diff patches run near 1.4 + * characters per token, not the 3 an ASCII-prose ratio assumed, so batches judged + * to fit under a 30000-token budget reached 60678 and 64413 against the ceiling and + * failed their whole chunk. A commit whose vectors never landed is one the diff + * watermark holds on, so the surface stalled on the same commit every cron tick. + * + * Equality with the ceiling is admissible: the rejections name counts strictly + * above it (`Max context reached 60678 tokens but model supports only 60000`), so + * 60000 is a supported count and not the first rejected one. + * + * Sized in characters rather than under the ceiling by a margin because a margin + * is not free — every extra batch costs two subrequests (the AI call and its + * `VECTORIZE.upsert`) against an invocation budget this worker already overruns, + * and a bound that holds by construction has nothing left for a margin to buy. + * + * `MAX_EMBEDDING_INPUT_CHARS` truncates one input to 8000 characters, so a batch + * holds at least 7 inputs however large each patch is. That floor is what the + * poller's per-file subrequest estimate rests on. */ -export function estimateEmbeddingTokens(text: string): number { - let wide = 0; - for (let i = 0; i < text.length; i++) { - if (text.charCodeAt(i) > 127) wide++; - } - const ascii = text.length - wide; - return wide + Math.ceil(ascii / ASCII_CHARS_PER_TOKEN); -} +export const MAX_EMBEDDING_BATCH_CHARS = WORKERS_AI_BATCH_CONTEXT_LIMIT; /** Half-open `[start, end)` index range over a caller's input array. */ export interface EmbeddingBatchRange { @@ -78,8 +74,14 @@ export interface EmbeddingBatchRange { } /** - * Split embedding inputs into batches whose estimated token totals stay within - * `budgetTokens`. + * Split embedding inputs into batches whose character totals stay within + * `budgetChars`. + * + * Each input is charged its own length plus `SPECIAL_TOKENS_PER_INPUT`, which makes + * the charged total an upper bound on the batch's true token count rather than an + * estimate of it (see `MAX_EMBEDDING_BATCH_CHARS`). Length is counted in UTF-16 + * code units, so a surrogate pair costs two — one more than the code point it + * encodes, which errs on the side that keeps the bound. * * Index ranges are returned rather than the strings themselves so the caller can * slice its own parallel arrays (files, metadata) by the same boundaries — the @@ -89,24 +91,24 @@ export interface EmbeddingBatchRange { * Contract: * - order is preserved, ranges are contiguous, and every input falls in exactly one * - no returned range is empty - * - an input whose own estimate already exceeds the budget occupies a range of + * - an input whose own charge already exceeds the budget occupies a range of * one. Cutting it down further belongs to the truncation axis * (`MAX_EMBEDDING_INPUT_CHARS`), and dropping it would lose a file from the index. */ export function planEmbeddingBatches( inputs: string[], - budgetTokens: number = MAX_EMBEDDING_BATCH_TOKENS, + budgetChars: number = MAX_EMBEDDING_BATCH_CHARS, ): EmbeddingBatchRange[] { const ranges: EmbeddingBatchRange[] = []; let start = 0; let total = 0; for (let i = 0; i < inputs.length; i++) { - const cost = estimateEmbeddingTokens(inputs[i]); + const cost = inputs[i].length + SPECIAL_TOKENS_PER_INPUT; // Close the open range before an input that would overrun the budget. The // `i > start` guard is what keeps an oversized input in a batch of its own // instead of closing an empty range and looping on it forever. - if (i > start && total + cost > budgetTokens) { + if (i > start && total + cost > budgetChars) { ranges.push({ start, end: i }); start = i; total = 0; diff --git a/src/poller.ts b/src/poller.ts index f803e38..293aa04 100644 --- a/src/poller.ts +++ b/src/poller.ts @@ -206,11 +206,13 @@ const DIFF_SUBREQUEST_BUDGET_PER_RUN = 900; * * Two are fixed and per-file: the D1 FTS mirror write and the Store DO row. * The third is the amortised batch cost — a batch spends 2 (the Workers AI call - * and its `VECTORIZE.upsert`) and holds at least 3 files, because + * and its `VECTORIZE.upsert`) and holds at least 7 files, because * `MAX_EMBEDDING_INPUT_CHARS` caps one input at 8000 characters and - * `MAX_EMBEDDING_BATCH_TOKENS` gives a batch 30000 tokens, so even CJK prose at - * ~1 token per character fits 3. That puts the true figure at 2.67 and under; 3 - * is it rounded to the safe side. */ + * `MAX_EMBEDDING_BATCH_CHARS` gives a batch 60000, whatever the payload is made + * of. That puts the true figure at 2.29 and under; 3 is it rounded to the safe + * side. The floor was 3 while the batch axis was an estimated token budget + * (#237); moving that axis to characters (#241) raised the floor and left this + * constant on the safe side of its own derivation, so it is unchanged. */ const DIFF_SUBREQUESTS_PER_FILE = 3; /** Per-phase subrequests that are not per-file: up to 5 commit detail fetches, From 31a80e788a8433e07d9eb5ee2474c62406d8c1c8 Mon Sep 17 00:00:00 2001 From: Claude Lin & Lay Date: Fri, 14 Aug 2026 19:37:05 +0900 Subject: [PATCH 2/4] test(pipeline): state what the regression fixture's estimator assertion divides [tests] MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 回帰 test の第 2 assertion は、退役した推定器の値そのものではなく `callCharge`（文字数 + special token）を 3 で割っている。両者は special token の分だけずれ、割られる側が大きい = 失敗しやすい側に寄るため安全側だが、 comment がその差を書いていなかった。fixture が再現領域に留まっていることを 将来 assert し直す人が、どちらの数値を見ているのか判断できるようにする。 comment のみ。挙動の変更なし。 #241 --- src/pipeline/embed-diff.test.ts | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/src/pipeline/embed-diff.test.ts b/src/pipeline/embed-diff.test.ts index 59c6316..411be42 100644 --- a/src/pipeline/embed-diff.test.ts +++ b/src/pipeline/embed-diff.test.ts @@ -145,9 +145,12 @@ describe("embed-diff: the batch axis is the character budget, not the file count const allInputs = aiCalls.flat(); // The fixture is only a regression test while it stays in the failing zone: over // the ceiling as one call, yet inside the budget the estimator would have - // computed for it (85202 characters of ASCII read as ~28400 tokens at 3 each, - // under the 30000 that #237 set). Both halves are asserted so a later edit to - // the patch size cannot quietly move the fixture out of the shape it reproduces. + // computed for it — roughly 85200 ASCII characters, which its ratio of 3 read as + // about 28400 tokens against the 30000 that #237 set. The second assertion + // divides the charge rather than the bare character sum, which overstates the + // estimator's figure by the special tokens and so errs toward failing. Both + // halves are asserted, so a later edit to the patch size cannot quietly move the + // fixture out of the shape it reproduces. expect(callCharge(allInputs)).toBeGreaterThan(WORKERS_AI_BATCH_CONTEXT_LIMIT); expect(Math.ceil(callCharge(allInputs) / 3)).toBeLessThanOrEqual(30000); From 1996d337ad399311a847ba9b17823d333acae859 Mon Sep 17 00:00:00 2001 From: Claude Lin & Lay Date: Fri, 14 Aug 2026 19:38:36 +0900 Subject: [PATCH 3/4] fix(pipeline): correct an unverified cron-cycle count in the batching comment [pipeline] MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `embed-diff.ts` の comment が「同じ 2 commit が 3 cron cycle 続けて天井を 超えた」と書いていたが、#241 が観測として挙げているのは 14:31（#237 merge 前）と 16:31（#240 merge 後）の 2 点であり、3 回という回数は裏づけが無い。 回数は本来の論点でもない。効いているのは「#237 の推定 token 予算が入る前と 後で実測値が同一だった = その予算はこの 2 commit を一度も縛っていなかった」 という点なので、そちらを書く形に直す。 comment のみ。挙動の変更なし。 #241 --- src/pipeline/embed-diff.ts | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/pipeline/embed-diff.ts b/src/pipeline/embed-diff.ts index 8baf032..c342bfb 100644 --- a/src/pipeline/embed-diff.ts +++ b/src/pipeline/embed-diff.ts @@ -201,8 +201,9 @@ export async function processAndUpsertCommitDiff( // small, and a commit that breaks that assumption used to fail the whole chunk — // which the poller reads as an uningested commit and holds the diff watermark on, // so the surface stalls there rather than skipping past it (#236). Characters - // rather than estimated tokens because the estimate ran 2.1x low on diff patches - // and let the same two commits over the ceiling three cron cycles running (#241). + // rather than estimated tokens because the estimate ran about 2.1x low on diff + // patches, so the same two commits reported the same over-ceiling token counts + // before and after the estimated budget shipped — it never bound them (#241). for (const { start, end } of planEmbeddingBatches(allInputs)) { const chunk = indexable.slice(start, end); const inputs = allInputs.slice(start, end); From c34887a301d4794d47183e08c52f67fe0a68d87f Mon Sep 17 00:00:00 2001 From: Claude Lin & Lay Date: Fri, 14 Aug 2026 19:40:09 +0900 Subject: [PATCH 4/4] test(pipeline): name only the calibration that failed, not the file-budget PR [tests] MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `embedding.test.ts` の comment が「校正で失敗した状態 (#237, #240)」と 2 件を 並べていたが、#240 は diff 経路の file 予算であって校正ではない。校正だったのは #237 だけで、その失敗が 2 度観測された（#240 merge の前と後）という関係である。 2 件を並べると、#240 も同じ軸の失敗だったと読める。軸が違うものを同列に置くと、 次にこの assert を触る人が「file 予算も校正だった」という誤った前提を引き継ぐ。 comment のみ。挙動の変更なし。 #241 --- src/pipeline/embedding.test.ts | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/pipeline/embedding.test.ts b/src/pipeline/embedding.test.ts index 6888f7f..b56eb28 100644 --- a/src/pipeline/embedding.test.ts +++ b/src/pipeline/embedding.test.ts @@ -20,7 +20,8 @@ describe("MAX_EMBEDDING_BATCH_CHARS", () => { // The whole guarantee rests on this: a token spans at least one character, so a // batch charged under the ceiling *in characters* is under it in tokens too. // Raise this above the ceiling and the bound stops holding by construction and - // goes back to being a calibration — the state that failed twice (#237, #240). + // goes back to being a calibration — the state #237 was in when the same two + // commits kept reporting the same over-ceiling token counts (#241). expect(MAX_EMBEDDING_BATCH_CHARS).toBeLessThanOrEqual(WORKERS_AI_BATCH_CONTEXT_LIMIT); });