diff --git a/docs/0-requirements.ja.md b/docs/0-requirements.ja.md index 3cafcba..877d174 100644 --- a/docs/0-requirements.ja.md +++ b/docs/0-requirements.ja.md @@ -220,7 +220,17 @@ wiki poller は `:45` cron 専属で、GitHub Wiki content の唯一の取り込 **カバレッジ.** page 走査の上限は `MAX_WIKI_FETCHES_PER_REPO_PER_RUN`(既定 20)。page 単位ではなく **HTTP 試行単位**で数えるので、候補を複数持つ page が fan-out で上限を超えることはない。この予算は深い wiki より小さいため、走査は**循環し、保存された cursor から再開する** — 最後に probe した slug を `wiki:{repo}` watermark 行の `etag` 列に置くので、schema 変更は不要。結果として全 page が ceil(pages / 予算) run 以内に到達する。毎 run 列挙の先頭から舐め直す実装が、位置 20 以降を構造的に到達不能にし、77 page の wiki の大半を未索引のまま放置していた(issue #184)。`MAX_WIKI_EMBEDDINGS_PER_RUN`(既定 30)は別軸のまま — Workers AI の embed 予算の上限であって、走査の上限ではない。 -**先頭 page では予算より候補リストを優先する.** 予算切れで打ち切った probe は「404 を観測した」ことにはならないので、walk は cursor を進める**手前**で break し、その page を次の pass に持ち越す(issue #185)。これが自己修復するのは、次の pass がその page の候補リストを最後まで払える場合だけで、予算が 1 page の候補数を下回ると同じ page を永久に probe し直し、walk は前進しない(issue #192)。そこで、その pass でまだ 1 page も visit していない間は候補リストを予算より優先する — 先頭 page は必ず候補を試し切り、そこでの miss は候補を全て観測した上での 404 なので failure として計上し cursor も前進させる。2 page 目以降は従来どおり厳密に予算で打ち切る。超過は 1 page の候補数 - 1 が上限(最大 3 subrequest)で、pass ごとに 1 回だけ。invocation あたり 1000 の予算に対して無視できる範囲であり、pass の `fetches` が予算を超えうる唯一のケースでもある。 +**run 全体の予算.** 上に挙げた予算はすべて *repo 単位* である。wiki cron は 1 invocation の中で `POLL_REPOS` を全て歩くので、各 repo は個別には規約を守る一方、その総和を見る主体がいなかった — そして Cloudflare が課金するのは総和のほうである。2026-08-15T03:45Z の tick で観測: 5 つの repo は正常に索引され、6 つ目は raw content probe が全て `Too many subrequests by single Worker invocation` で弾かれ、誰も見ていない page について 8 件の failure を記録した。枯渇は loop が最後に到達した repo が一身に受ける。これは loop の性質であって、その repo の性質ではない。そこで `WIKI_SUBREQUEST_BUDGET_PER_RUN` が invocation に対するこの surface の取り分を宣言し、`wikiFetchBudgetForPass` がその残りを pass の fetch 上限へ変換し、`runWikiSurfaces` が各 pass の *実測* 消費を差し引いてから次の repo を通す。除算ではなく減算である — diff surface の `diffFileBudgetPerPhase` が除算なのは phase の費用が repo ごとに事前に確定するからで、wiki の走査費用は走らせてみるまで分からない。おかげで変更が無かった repo は取り分をほぼ丸ごと後続へ渡す。残予算が pass を賄えなくなった時点で、以降の repo は**呼び出し自体を行わない** — cursor は前 run が残した位置のまま据え置かれ、次の invocation がそこから再開する。深い wiki 1 つが repo 単位の fetch 予算に対して既に持っていた「上限に達したら cursor を残して正常終了」という挙動を、run 全体の軸へ持ち上げた形である。reap の予算は消費した分を全額計上する一方、admission では意図的に確保 *しない* — 取り分 250 に対して repo ごとの最悪値 50 を先取りすると、reap がほとんど何もしない通常時にも全 pass へ課税することになり、何も起きていない run で repo を先送りしてしまう。 + +**この会計単位は Cloudflare の課金単位ではなく、取り分だけでは観測された事象を直せない.** 定数は call site から読み取ったもので、`fetch` 1 回・Durable Object 往復 1 回・D1 statement 1 本・Vectorize 呼び出し 1 回・Workers AI 呼び出し 1 回をそれぞれ 1 として数えている。ここから 2 つのことが従う。重要なのは 2 つ目である。1 つ目は差の存在: この単位では、6 つ目の repo の最初の probe が例外を投げた時点の消費は約 70(全 page 未変更の場合)から約 292(全 page 再 embed の場合)の間であり、文書化された上限 1000 には遠い。poller 側から見えるものでこの差は説明できず、少なくとも 1 つの binding がこの会計の 1 より高く課金されている。Workers は実行時に消費量を数える手段を提供していないので、**このファイルのどの定数も実際の上限に対して較正できない**。2 つ目は、上の幅が計測誤差ではなく費用プロファイルそのものだという点: 未変更の page は embed fan-out が hash 比較の後で加算されるため fetch 試行分しか課金されない。つまり索引済みの wiki に対する定常運転の run は、同じ page 数の bulk import に比べてごく僅かしか消費しない。**定常状態のプロファイル —— 毎時 cron の通常の姿であり、実際に例外を投げたときの姿 —— では、取り分 250 は何も先送りせず、run は失敗したときと全く同じ順序で進む。** したがって取り分が縛るのは bulk import の形(fetch した page ごとに embed fan-out が乗るため、数 repo で 250 に達する)であって、報告された failure に対処する機構ではない。`DIFF_SUBREQUEST_BUDGET_PER_RUN` が 900 なのは diff surface の per-file 見積もりが実測された reject に対して導出されているからで、この surface にはその導出が無い以上、値は観測された幅の中での判断であって較正ではない。この否定的な結果は回帰テストで固定してあり、黙って成り立たなくなることはない。 + +**では何が対処しているか: 壁は観測できる.** 上に挙げた予算はすべて見積もりだが、上限に到達したこと自体は見積もりではない — binding が例外を投げ、しかも自分の名前を名乗る。これは推測ではなく計測である。枯渇は invocation スコープかつ終端的で、1 つの subrequest が拒否された後は同じ invocation の後続も全て拒否される。観測された run が probe ごとに例外を投げ続け、walk がそれを page の failure として記録し続けたのはそのためである。よって最初に検知した時点で run を終了する。probe 中だった page は未観測のまま cursor を動かさず、reap は丸ごと省略し(その probe も同じ壁に当たるうえ、候補集合は縮む一方なので見送りに costs は無い)、残りの repo は全て未接触で先送りする。これで**できない**のは *最初の* 拒否を防ぐことで、それが検知イベントそのものだからである。したがって「ログに `Too many subrequests` が出ない」という字義どおりの達成は、binding ごとの実費用を知らない限り不可能である。達成されるのは、1 回の拒否が「loop が最後に到達した repo における偽の page failure の連鎖」に変わらなくなることである。 + +**処理順は run ごとに回転する.** run 全体の予算を list 順に消費すると、痩せるのは毎 cron 同じ最後尾である — 溢れだけを止めた元の欠陥そのもので、結果を決めるのは依然として loop 上の位置になる。`rotateReposForRun` が tick によって走査順をずらすので、毎時 cron と N repo なら各 repo は N 時間に 1 度先頭に立ち、予算枯渇で先送りされた repo は後続の tick で早い位置に回る。offset は保存せず cron の `scheduledTime` から導出するため watermark を必要とせず、同じ tick に対しては決定的である。 + +**観測した miss と観測していない miss は別のカウンタ.** 候補が全て 404 を返した page は観測であり `failed` に計上する。probe が *例外を投げた* page は観測ではなく — subrequest 枯渇は例外を投げる — `inconclusive` に計上する。両者は "all candidates 404" という同一の文言で報告されており、実在する 8 つの page が「無い」と読めたのはそのためである。`inconclusive` でも cursor は前進させる。その page は次の周回で再試行される一方、cursor を留めると恒久的に到達できない page 1 つが walk 全体を停止させうるからである。 + +**先頭 page では予算より候補リストを優先する.** 予算切れで打ち切った probe は「404 を観測した」ことにはならないので、walk は cursor を進める**手前**で break し、その page を次の pass に持ち越す(issue #185)。これが自己修復するのは、次の pass がその page の候補リストを最後まで払える場合だけで、予算が 1 page の候補数を下回ると同じ page を永久に probe し直し、walk は前進しない(issue #192)。そこで、その pass でまだ 1 page も visit していない間は候補リストを予算より優先する — 先頭 page は必ず候補を試し切り、そこでの miss は候補を全て観測した上での 404 なので failure として計上し cursor も前進させる。2 page 目以降は従来どおり厳密に予算で打ち切る。超過は 1 page の候補数 - 1 が上限(最大 3 subrequest)で、pass ごとに 1 回だけ。pass の `fetches` が予算を超えうる唯一のケースでもある。これを吸収するのは、この文がかつて引き合いに出していた文書上の invocation 上限ではなく、後述の run 全体の取り分が持つ余裕である。そもそも取り分が 1 page 分を賄えない pass は admission されない。 **周回の完了.** 2 本目の watermark 行 `wiki-lap:{repo}` が *lap anchor* — 現在の周回がどの slug の次から始まったか — を保持する。pass は anchor の直前の page に到達した時点、つまり cursor が一周して戻ってきた時点で `wrapped: true` を返し、anchor はその page へ移動して次の周回がその次から始まる。これが admin endpoint の `done` を到達可能にしている。「この 1 回の pass で全 page を踏破した」という意味では、1 pass の fetch 予算より page 数が多い wiki で真になりようがなく、「`done` まで呼び続けろ」という手順に停止条件が無かった(issue #188)。anchor は保存した index ではなく slug 順で解決するので、周回の途中で page が増減しても desync しない。`cursor=` を明示指定した場合はその地点から新しい周回を開始する。 diff --git a/docs/0-requirements.md b/docs/0-requirements.md index 831f5b7..68d1c24 100644 --- a/docs/0-requirements.md +++ b/docs/0-requirements.md @@ -225,7 +225,17 @@ Change detection uses a content SHA-256 hash (no git blob SHA is available witho **Coverage.** The page walk is bounded by `MAX_WIKI_FETCHES_PER_REPO_PER_RUN` (default 20), counted per *HTTP attempt* rather than per page so a page needing several candidates cannot exceed the ceiling by fanning out. Because the budget is smaller than a deep wiki, the walk is **circular and resumes from a stored cursor** — the last slug probed, held in the `wiki:{repo}` watermark row's `etag` column, so no schema change was needed. Every page is therefore reached within ceil(pages / budget) runs. Restarting at the head of the enumeration each run is what made pages past position ~20 structurally unreachable and left most of a 77-page wiki unindexed (issue #184). `MAX_WIKI_EMBEDDINGS_PER_RUN` (default 30) remains a separate axis: it caps the Workers AI embed budget, not the walk. -**The budget yields to the first page.** A probe the budget cuts short is not an observed 404, so the walk breaks *before* the cursor moves and retries the page next pass (issue #185). That self-heals only when the next pass can afford the page's whole candidate list; a budget below it re-probes the same page forever and the walk never advances (issue #192). So while the pass has visited nothing, the candidate list outranks the budget: the first page always runs its probes to the end, and a miss there is a real all-candidate 404 that counts as a failure and moves the cursor. Every page after the first keeps the strict check. The overrun is bounded by one page's candidate count minus one — at most 3 subrequests, once per pass, against an invocation budget of 1000 — and it is the only case where a pass's `fetches` may exceed its budget. +**Run-wide budget.** Every budget named above is *per repo*, and the wiki cron walks all of `POLL_REPOS` inside one invocation, so each repo kept its own contract while nothing watched the sum — and the sum is what Cloudflare charges. Measured on the 2026-08-15T03:45Z tick: five repos indexed normally and the sixth had every raw-content probe rejected with `Too many subrequests by single Worker invocation`, reporting 8 failures for pages nobody had looked at. The starvation lands wholly on whichever repo the loop reaches last, which is a property of the loop and not of the repo. `WIKI_SUBREQUEST_BUDGET_PER_RUN` therefore declares the surface's share of the invocation, `wikiFetchBudgetForPass` converts what is left of it into the pass's fetch cap, and `runWikiSurfaces` subtracts each pass's *measured* spend before admitting the next repo. Subtraction rather than division — the diff surface's `diffFileBudgetPerPhase` divides, because its phases are bounded per repo up front, whereas a wiki walk's cost is only known once it has run, so a repo that finds nothing changed hands its share on almost whole. When the remainder can no longer fund a pass, the remaining repos are **not called at all**: their cursors stay where the previous run left them and the next invocation resumes there. That is the same "hit the ceiling, keep the cursor, finish clean" behaviour a single deep wiki already had on the per-repo fetch budget, lifted to the run axis. The reap's own budgets are charged in full when spent but deliberately *not* reserved at admission — reserving each repo's worst case (50) against a share of 250 would tax every pass for work the reap almost never does, and would defer repos on a run where nothing happened at all. + +**The accounting unit is not the one Cloudflare charges, and the share alone does not fix the observed case.** The constants are read off call sites — one per `fetch`, per Durable Object round trip, per D1 statement, per Vectorize call, per Workers AI call. Two things follow, and the second is the one that matters. First, the gap: in these units the failing run had spent between ~70 and ~292 when the sixth repo's first probe threw, against a documented ceiling of 1000, and nothing visible in the poller explains the difference — at least one binding costs more than the 1 this accounting gives it, and Workers expose no runtime counter to measure it with, so no constant here can be calibrated against the real limit. Second, the range above is not measurement error but the cost profile itself: an unchanged page is charged only its fetch attempts, because the embed fan-out is added past the hash-comparison, so a steady-state run over already-indexed wikis costs a fraction of a bulk import over the same page counts. **On the steady-state profile — the normal shape of an hourly cron, and the shape that actually threw — a share of 250 defers nothing and the run proceeds in exactly the order that failed.** The share is therefore a bound on the bulk-import shape, where each fetched page carries the embed fan-out and a handful of repos reaches 250 quickly; it is not the mechanism that addresses the reported failure. `DIFF_SUBREQUEST_BUDGET_PER_RUN` is 900 because the diff surface's per-file estimate was derived against measured rejections; this surface has no such derivation, so its value is a judgement inside the observed range rather than a calibration. A regression test pins the negative result so it cannot quietly stop being true. + +**What does cover it: the wall is observable.** Every budget above is an estimate, but the limit being reached is not — the binding throws and names itself, which is a measurement rather than a guess. Exhaustion is invocation-scoped and terminal: once one subrequest is rejected, every later one in the same invocation is too, which is why the observed run kept throwing probe after probe while the walk kept converting those throws into recorded page failures. So the first sighting ends the run. The page being probed is left unobserved with the cursor unmoved, the reap is skipped wholesale (its probes would hit the same wall, and a withheld reap costs nothing because the candidate set only shrinks), and every remaining repo is deferred untouched. What this cannot do is prevent the *first* rejection — that is the detection event — so the literal "no `Too many subrequests` in the log" is not achievable without knowing the real per-binding cost. What is achieved is that one rejection no longer becomes a cascade of false page failures on whichever repo the loop reached last. + +**Run order rotates.** A run-wide budget spent in list order thins the same tail every cron — the original defect with its overrun bounded, since loop position would still decide the outcome. `rotateReposForRun` offsets the walk order by the tick, so with an hourly cron and N repos each repo leads once every N hours and a repo deferred by an exhausted share is served early on a later tick. The offset is derived from the cron's `scheduledTime` rather than stored, so it needs no watermark and stays deterministic for a given tick. + +**Observed and unobserved misses are separate counters.** A page whose every candidate answered 404 is an observation and counts as `failed`. A page whose probes *threw* is not — subrequest exhaustion throws — and counts as `inconclusive` instead. Both were reported identically under the literal "all candidates 404", which is how eight pages that were present read as missing. `inconclusive` still advances the cursor: the page is retried on the next lap, whereas holding it would let one permanently unreachable page stall the walk. + +**The budget yields to the first page.** A probe the budget cuts short is not an observed 404, so the walk breaks *before* the cursor moves and retries the page next pass (issue #185). That self-heals only when the next pass can afford the page's whole candidate list; a budget below it re-probes the same page forever and the walk never advances (issue #192). So while the pass has visited nothing, the candidate list outranks the budget: the first page always runs its probes to the end, and a miss there is a real all-candidate 404 that counts as a failure and moves the cursor. Every page after the first keeps the strict check. The overrun is bounded by one page's candidate count minus one — at most 3 subrequests, once per pass — and it is the only case where a pass's `fetches` may exceed its budget. What absorbs it is the margin under the run-wide share below, not the documented invocation ceiling this sentence used to invoke; a pass is never admitted at all unless the share can fund a page for it. **Lap completion.** A second watermark row, `wiki-lap:{repo}`, holds the *lap anchor* — the slug the current sweep started after. The pass reports `wrapped: true` when it reaches the page immediately before the anchor, i.e. when the cursor has come all the way back around, and the anchor then moves to that page so the next lap starts after it. This is what makes the admin endpoint's `done` reachable: reporting "this one pass saw every page" can never be true for a wiki with more pages than one pass may fetch, so the documented "call until `done`" loop had no terminating condition (issue #188). The anchor is resolved by slug order rather than by a stored index, so a page added or deleted mid-lap cannot desync it. An explicit `cursor=` override opens a fresh lap at that point. diff --git a/docs/installation.ja.md b/docs/installation.ja.md index c6335cc..7ddb27f 100644 --- a/docs/installation.ja.md +++ b/docs/installation.ja.md @@ -278,10 +278,12 @@ POST /admin/backfill-wiki?repo=owner/repo レスポンス: ```json -{ "repo": "owner/repo", "pages": 77, "fetches": 20, "visited": 20, "embedded": 18, - "skipped": 2, "failed": 0, "removed": 3, "orphansDeferred": 5, "orphansWithheld": 0, +{ "repo": "owner/repo", "pages": 77, "fetches": 20, "subrequests": 137, "visited": 20, + "embedded": 18, "skipped": 2, "failed": 0, "inconclusive": 0, "removed": 3, + "orphansDeferred": 5, "orphansWithheld": 0, "startCursor": "", "nextCursor": "current-architecture-as-concession", - "lapAnchor": "", "wrapped": false, "enumerated": true, "done": false } + "lapAnchor": "", "wrapped": false, "enumerated": true, "exhausted": false, + "done": false } ``` 運用上の注意: @@ -292,6 +294,9 @@ POST /admin/backfill-wiki?repo=owner/repo - `enumerated: false` は `/wiki/_pages` の scrape が失敗したという意味。何も索引せず、意図的に何も削除していない。空の wiki と解釈せず再試行すること - `orphansDeferred` は、その run で**到達しなかった**削除候補の数。削除枠と probe 枠のどちらかが尽きて打ち切った分にあたる。0 になるまで呼び続ける。到達した上で見送った候補は `orphansWithheld` の側に数えられる — 枠が分かれているので、見送りが削除枠を消費して後ろに並ぶ実削除を止めることはない(issue #197) - `fetches` は、その call の**先頭** page が予算より多くの候補を必要とした場合に限り `limit` を最大 3 超える。1 page の probe は最大 4 回(ファイル名候補 2 × `md` / `markdown`)で、途中で打ち切った probe は結果を観測したことにならないため、そのままでは cursor を進めないまま毎回同じ page を probe し直すことになる。そこで各 call の先頭 page だけ候補リストを試し切らせている。2 page 目以降は予算どおりに打ち切る(issue #192) +- `failed` は、ファイル名 / 拡張子の候補が全て 404 を返したことを**観測できた** page の数。`inconclusive` は、probe が応答ではなく例外を投げたために不在を観測できなかった page の数(ネットワークエラー、または subrequest 枯渇)。両者は "all candidates 404" という同一の文言で報告されており、実在する page が「無い」と読めたのはそのためである(issue #248)。`inconclusive` が 0 でない場合、それは wiki についてではなく run についての信号であり、当該 page は次の周回で再試行される +- `subrequests` はその call が消費した量で、subrequest を発行する各 call site で数えている。cron 経路ではこれを run 全体の取り分から差し引くので、深い wiki を持つ 1 つの repo が後続の repo を枯渇させることはない。この endpoint は独立した Worker invocation なので、値は観測用に返すだけで取り分の適用は受けない。単位は call site であって Cloudflare の課金単位ではない点に注意 — 余裕の指標として読む前に要件記述を参照すること +- `exhausted: true` は、その call の途中で invocation の subrequest が尽きたことを示す。walk はそこで cursor を動かさずに停止し、reap も省略されるので、確認できていない事実に基づいて failed や削除が記録されることはない。続きは再度呼べば進む。cron 経路ではこれが run 全体を終了させ、残りの repo は未接触で先送りされる(issue #248) - `orphansWithheld` は、削除候補に挙がったが content がまだ配信されていた(あるいは実在確認が結論を出せなかった)ため削除を見送った page 数。0 でない場合、`_pages` の scrape が**実際の wiki より少なく返っている**という意味。page 自体は無傷で守られており、調べるべきは列挙のほう。見送った page 名は worker のログに出る(issue #187) - カバレッジの確認は `search_docs` の `type = 'wiki_doc'` 行と `https://github.com/{repo}/wiki/_pages` の page 一覧を突き合わせる diff --git a/docs/installation.md b/docs/installation.md index 5c01aa8..a7c90a0 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -278,10 +278,12 @@ Authentication: Response: ```json -{ "repo": "owner/repo", "pages": 77, "fetches": 20, "visited": 20, "embedded": 18, - "skipped": 2, "failed": 0, "removed": 3, "orphansDeferred": 5, "orphansWithheld": 0, +{ "repo": "owner/repo", "pages": 77, "fetches": 20, "subrequests": 137, "visited": 20, + "embedded": 18, "skipped": 2, "failed": 0, "inconclusive": 0, "removed": 3, + "orphansDeferred": 5, "orphansWithheld": 0, "startCursor": "", "nextCursor": "current-architecture-as-concession", - "lapAnchor": "", "wrapped": false, "enumerated": true, "done": false } + "lapAnchor": "", "wrapped": false, "enumerated": true, "exhausted": false, + "done": false } ``` Operational notes: @@ -292,6 +294,9 @@ Operational notes: - `enumerated: false` means the `/wiki/_pages` scrape failed — nothing was indexed and, deliberately, nothing was reaped; retry rather than treating it as an empty wiki - `orphansDeferred` counts reap candidates the call never reached, because it hit either the per-run delete cap or the per-run probe cap; keep calling until it reaches 0. A candidate that was reached and withheld is reported by `orphansWithheld`, not here — the two budgets are separate so a withheld candidate cannot spend a delete slot and stall the real deletions ordering behind it (issue #197) - `fetches` may exceed `limit` by up to 3 on a call whose *first* page needs more candidates than the budget allows. Probing a page is up to 4 attempts (two filename candidates x `md` / `markdown`), and a probe cut short mid-list is not an observed result, so the walk would otherwise re-probe that page on every call without ever moving the cursor. The first page of each call is therefore allowed to finish its candidate list; every page after it stops at the budget (issue #192) +- `failed` counts pages whose every filename / extension candidate answered an **observed** 404. `inconclusive` counts pages whose probes *threw* instead of answering, so absence was never observed — a network error, or subrequest exhaustion. The two were reported as one under the literal "all candidates 404", which is how pages that were present read as missing (issue #248). A non-zero `inconclusive` is a signal about the run, not about the wiki; the page is retried on the next lap +- `subrequests` is what the call spent, counted at each call site that issues one. The cron path subtracts it from a run-wide share so one repo's deep wiki cannot starve the repos behind it; this endpoint is its own Worker invocation, so the figure is reported for observability only and no share is applied to it. Note the unit is call sites, not what Cloudflare bills — see the requirements doc before reading it as headroom +- `exhausted: true` means the invocation ran out of subrequests during the call. The walk stopped there with the cursor unmoved and the reap was skipped, so nothing was recorded as failed or deleted on unverified evidence; call again to continue. On the cron path this ends the whole run and defers the remaining repos untouched (issue #248) - `orphansWithheld` counts reap candidates whose content still served (or whose existence probe could not conclude), so the delete was withheld. Non-zero means the `_pages` scrape came back **short of the live wiki** — the pages themselves are intact and were protected, but the enumeration is what to investigate; the worker log names each withheld page (issue #187) - verify coverage by comparing `search_docs` rows (`type = 'wiki_doc'`) against the page list at `https://github.com/{repo}/wiki/_pages` diff --git a/src/poller-wiki.test.ts b/src/poller-wiki.test.ts index 62fc5cf..e0e2674 100644 --- a/src/poller-wiki.test.ts +++ b/src/poller-wiki.test.ts @@ -820,3 +820,697 @@ describe("poller: pollWiki orphan reap", () => { expect(store.deletes).toEqual([]); }); }); + +// ── Run-wide subrequest budget (issue #248) ────────────────── + +const { + runWikiSurfaces, + wikiFetchBudgetForPass, + rotateReposForRun, + isSubrequestExhaustion, +} = await import("./poller.js"); + +/** `MAX_WIKI_FETCHES_PER_REPO_PER_RUN`, mirrored: the constant is module-private + * and what these assertions are about is the run budget outranking it. */ +const PER_REPO_FETCH_CAP = 20; + +/** + * Stub the global fetch with several fake wikis at once. + * + * The single-repo `stubWiki` above keys everything off the `REPO` constant, and + * the defect this suite covers is only visible across repos: every per-repo + * budget holds while their sum overruns the invocation ceiling. + * + * `throwAfter` makes `fetch()` raise rather than answer once that many raw + * requests have been issued, so the walk's classification of an *unobserved* + * miss can be asserted. `throwMessage` picks which kind: the default is an + * ordinary network blip, and passing the subrequest-exhaustion text reproduces + * the production failure shape, which the poller must treat differently — one + * is this page's problem, the other is the whole run's. + */ +function stubMultiWiki( + wikis: Record, + opts: { throwAfter?: number; throwMessage?: string } = {}, +) { + const rawByRepo = new Map(); + let rawCount = 0; + + const pagesHtmlFor = (repo: string, wiki: FakeWiki): string => + wiki.listed + .map( + (p) => + ``, + ) + .join("\n"); + + const fetchMock = vi.fn(async (input: string | URL) => { + const url = String(input); + + if (url.startsWith("https://github.com/") && url.includes(".wiki.git/info/refs")) { + const repo = url.slice("https://github.com/".length, url.indexOf(".wiki.git")); + return new Response("001e# service=git-upload-pack\n", { + status: wikis[repo] ? 200 : 404, + }); + } + + if (url.endsWith("/wiki/_pages")) { + const repo = url.slice("https://github.com/".length, url.indexOf("/wiki/_pages")); + const wiki = wikis[repo]; + if (!wiki || wiki.indexFails) return new Response("boom", { status: 503 }); + return new Response(pagesHtmlFor(repo, wiki), { + status: 200, + headers: { "Content-Type": "text/html" }, + }); + } + + const rawRoot = "https://raw.githubusercontent.com/wiki/"; + if (url.startsWith(rawRoot)) { + const tail = url.slice(rawRoot.length); + // `owner/repo/Page.ext` — the repo is the first two path segments. + const slash = tail.indexOf("/", tail.indexOf("/") + 1); + const repo = tail.slice(0, slash); + const file = tail.slice(slash + 1); + + rawCount++; + const seen = rawByRepo.get(repo) ?? []; + seen.push(file); + rawByRepo.set(repo, seen); + + if (opts.throwAfter !== undefined && rawCount > opts.throwAfter) { + throw new Error(opts.throwMessage ?? "Network connection lost."); + } + + const dot = file.lastIndexOf("."); + const name = decodeURIComponent(file.slice(0, dot)); + const ext = file.slice(dot + 1); + const body = wikis[repo]?.files[name]; + if (ext !== "md" || body === undefined) { + return new Response("Not Found", { status: 404 }); + } + return new Response(body, { status: 200 }); + } + + throw new Error(`unexpected fetch in multi-wiki stub: ${url}`); + }); + + vi.stubGlobal("fetch", fetchMock); + + return { + /** Raw-content requests issued for one repo. Empty = never walked. */ + rawFor: (repo: string): string[] => rawByRepo.get(repo) ?? [], + }; +} + +/** IssueStore stand-in spanning several repos, so cursors can be compared. */ +function makeMultiStore() { + const watermarks = new Map(); + + const stub = { + async fetch(request: Request): Promise { + const url = new URL(request.url); + const path = url.pathname; + + if (request.method === "GET" && path === "/wiki-docs") return Response.json([]); + if (request.method === "GET" && path === "/watermark") { + const wm = watermarks.get(url.searchParams.get("repo") ?? ""); + if (!wm) return new Response("not found", { status: 404 }); + return Response.json(wm); + } + if (request.method === "POST" && path === "/watermark") { + const body = (await request.json()) as { + repo: string; + lastPolledAt: string; + etag?: string; + }; + watermarks.set(body.repo, { + lastPolledAt: body.lastPolledAt, + etag: body.etag ?? "", + }); + return new Response("ok"); + } + return new Response("ok"); + }, + }; + + return { + stub: stub as unknown as DurableObjectStub, + cursor: (repo: string) => watermarks.get(`wiki:${repo}`)?.etag, + }; +} + +/** A wiki whose enumeration totals `pages`. Nothing is in the store, so every + * fetched page embeds and the pass costs its worst case — which is what brings + * the run budget into play within a realistic repo count. + * + * One of the `pages` is `Home`: `listWikiPages` unions it into every + * enumeration whether or not `_pages` lists it, so a fixture without a `Home` + * file hands the walk a genuine all-candidate 404 on every run. */ +function deepWiki(pages: number): FakeWiki { + const slugs = Array.from( + { length: pages - 1 }, + (_, i) => `p${String(i).padStart(2, "0")}`, + ); + return { + listed: slugs.map((slug) => ({ slug, title: slug })), + files: Object.fromEntries([...slugs, "Home"].map((s) => [s, `body of ${s}`])), + }; +} + +const repoList = (n: number): string[] => + Array.from({ length: n }, (_, i) => `acme/r${String(i).padStart(2, "0")}`); + +const wikiSet = (repos: string[], pages: number): Record => + Object.fromEntries(repos.map((r) => [r, deepWiki(pages)])); + +/** Capture the poller's console output for one run. The per-repo summary line + * is the only place a pass's `failed` / `inconclusive` counts surface once + * `runWikiSurfaces` has swallowed the summaries. */ +function captureConsole() { + const lines: string[] = []; + const record = (...args: unknown[]) => { + lines.push(args.map((a) => String(a)).join(" ")); + }; + const log = vi.spyOn(console, "log").mockImplementation(record); + const warn = vi.spyOn(console, "warn").mockImplementation(record); + const error = vi.spyOn(console, "error").mockImplementation(record); + return { + lines, + restore: () => { + log.mockRestore(); + warn.mockRestore(); + error.mockRestore(); + }, + /** The end-of-pass summary line for one repo, if the pass ran. Matched on + * the `N pages,` field as well as the prefix: warn lines share the + * `{repo} wiki: ` prefix and would otherwise be picked up instead. */ + summaryFor: (repo: string) => + lines.find((l) => l.startsWith(`${repo} wiki: `) && / \d+ pages, /.test(l)), + }; +} + +const TICK = Date.parse("2026-08-15T03:45:00Z"); + +describe("poller: wiki run-wide subrequest budget", () => { + it("hands a repo the full per-repo cap while the run budget is untouched", () => { + expect(wikiFetchBudgetForPass(900)).toBe(PER_REPO_FETCH_CAP); + }); + + it("lets the run budget outrank the per-repo cap once the remainder is thin", () => { + // The remainder must cover the pass's fixed cost — overhead plus a reap + // that fills both of its budgets — before it funds a single page, so the + // cap falls below the per-repo constant well before the budget is spent. + const thin = wikiFetchBudgetForPass(120); + expect(thin).toBeGreaterThan(0); + expect(thin).toBeLessThan(PER_REPO_FETCH_CAP); + // Monotonic in the remainder: a repo later in the run never gets more. + expect(wikiFetchBudgetForPass(200)).toBeGreaterThanOrEqual(thin); + }); + + it("returns zero rather than a token budget when a pass cannot be funded", () => { + // Zero is the caller's signal to skip the repo outright. A pass always runs + // its first page's whole candidate list (issue #192), so a budget of 1 would + // be overrun rather than respected, and the pass would spend its fixed + // overhead to make one page of progress. + expect(wikiFetchBudgetForPass(14)).toBe(0); + expect(wikiFetchBudgetForPass(0)).toBe(0); + expect(wikiFetchBudgetForPass(-100)).toBe(0); + // One unit above the floor funds exactly one page, not zero. + expect(wikiFetchBudgetForPass(15)).toBe(1); + }); + + it("leaves no repo of the production six-repo run recording a failure", async () => { + // The observed shape (2026-08-15T03:45Z tick), page counts included: five + // repos indexed normally, one wiki not enabled, and the sixth reported 8 + // failures for pages nobody had looked at — every per-repo budget held while + // their sum blew the invocation ceiling, and the tail wore all of it. + // + // Both outcomes for a trailing repo are correct now and neither records a + // failure: walked within the remaining share, or deferred untouched. What + // must not happen is the third one, a failure for a page never fetched. + const repos = [ + "acme/webhook-mcp", // 5 pages + "acme/rag-mcp", // 7 + "acme/desktop", // wiki not enabled + "acme/language", // 87 — the one that eats the run + "acme/dipper", // 5 + "acme/neuron-graph", // 12 — the tail that reported 8 failures + ]; + const wikis: Record = { + "acme/webhook-mcp": deepWiki(5), + "acme/rag-mcp": deepWiki(7), + "acme/language": deepWiki(87), + "acme/dipper": deepWiki(5), + "acme/neuron-graph": deepWiki(12), + }; + const multi = stubMultiWiki(wikis); + const store = makeMultiStore(); + const { env } = makeWikiEnv(); + const con = captureConsole(); + + try { + await runWikiSurfaces(repos, env, store.stub, TICK); + } finally { + con.restore(); + } + + let walked = 0; + for (const repo of repos) { + const summary = con.summaryFor(repo); + if (summary === undefined) { + // Deferred: never called, so its cursor cannot have moved. + expect(multi.rawFor(repo)).toEqual([]); + expect(store.cursor(repo)).toBeUndefined(); + continue; + } + walked++; + expect(summary).toContain("0 failed"); + expect(summary).toContain("0 inconclusive"); + } + // Guard against the assertions above passing vacuously on a run that + // deferred everything: the share must fund real work, not just refuse it. + expect(walked).toBeGreaterThanOrEqual(3); + // The exception the fix exists to prevent must not appear at all. + expect(con.lines.some((l) => l.includes("Too many subrequests"))).toBe(false); + }); + + it("defers repos it cannot fund instead of walking them into exhaustion", async () => { + // Enough repos that the declared share cannot cover them all in one run. + const repos = repoList(12); + const multi = stubMultiWiki(wikiSet(repos, 30)); + const store = makeMultiStore(); + const { env } = makeWikiEnv(); + const con = captureConsole(); + + try { + await runWikiSurfaces(repos, env, store.stub, TICK); + } finally { + con.restore(); + } + + const ordered = rotateReposForRun(repos, TICK); + const walked = ordered.filter((r) => multi.rawFor(r).length > 0); + const deferred = ordered.filter((r) => multi.rawFor(r).length === 0); + + expect(walked.length).toBeGreaterThan(0); + expect(deferred.length).toBeGreaterThan(0); + // Deferral is a suffix of the run order, never a hole in the middle. + expect(ordered.slice(0, walked.length)).toEqual(walked); + + for (const repo of deferred) { + // Untouched cursor is the whole point: the next run resumes these repos + // exactly where the previous one left them, rather than recording a + // failure for a page that was never fetched. + expect(store.cursor(repo)).toBeUndefined(); + expect(con.summaryFor(repo)).toBeUndefined(); + } + expect(con.lines.some((l) => l.includes("Wiki run share exhausted"))).toBe(true); + }); + + it("brings a deferred repo to the head of the order on a later tick", async () => { + // Without rotation the tail of `POLL_REPOS` is starved every run, which is + // the same "loop position decides the outcome" defect the budget removes. + const repos = repoList(12); + const store = makeMultiStore(); + const { env } = makeWikiEnv(); + + const runAt = async (tick: number) => { + const multi = stubMultiWiki(wikiSet(repos, 30)); + const con = captureConsole(); + try { + await runWikiSurfaces(repos, env, store.stub, tick); + } finally { + con.restore(); + vi.unstubAllGlobals(); + } + return new Set(repos.filter((r) => multi.rawFor(r).length > 0)); + }; + + const first = await runAt(TICK); + const deferred = repos.filter((r) => !first.has(r)); + expect(deferred.length).toBeGreaterThan(0); + + // Walk forward one tick at a time; every deferred repo must get walked + // within one full rotation of the list. + const covered = new Set(first); + for (let hour = 1; hour < repos.length; hour++) { + for (const r of await runAt(TICK + hour * 3_600_000)) covered.add(r); + } + for (const repo of deferred) expect(covered.has(repo)).toBe(true); + }); + + it("records an unobserved probe miss as inconclusive, not as a failure", async () => { + // A `fetch()` that throws says nothing about whether the page exists. The + // old message asserted "all candidates 404" for it, which is how eight + // present pages were reported as missing. + const repos = repoList(2); + stubMultiWiki(wikiSet(repos, 5), { throwAfter: 0 }); + const store = makeMultiStore(); + const { env } = makeWikiEnv(); + const con = captureConsole(); + + try { + await runWikiSurfaces(repos, env, store.stub, TICK); + } finally { + con.restore(); + } + + for (const repo of repos) { + const summary = con.summaryFor(repo); + expect(summary).toBeDefined(); + expect(summary).toContain("0 failed"); + expect(summary).not.toContain("0 inconclusive"); + } + // The misleading literal is gone; the replacement names the cause. + expect(con.lines.some((l) => l.includes("all candidates 404"))).toBe(false); + expect(con.lines.some((l) => l.includes("probe threw; absence not observed"))).toBe( + true, + ); + }); + + it("still reports an observed all-404 miss as a failure", async () => { + // The other side of the split: a page listed in `_pages` whose file is + // genuinely absent answers 404 on every candidate, and that *is* an + // observation. It must keep counting as a failure. + const wiki = deepWiki(3); + delete wiki.files.p01; + stubMultiWiki({ "acme/r00": wiki }); + const store = makeMultiStore(); + const { env } = makeWikiEnv(); + const con = captureConsole(); + + try { + await runWikiSurfaces(["acme/r00"], env, store.stub, TICK); + } finally { + con.restore(); + } + + const summary = con.summaryFor("acme/r00"); + expect(summary).toContain("1 failed"); + expect(summary).toContain("0 inconclusive"); + expect(con.lines.some((l) => l.includes("every candidate answered 404"))).toBe(true); + }); + + it("does not break a deep wiki's multi-run drain", async () => { + // `liplus-language` shape: 87 pages against a 20-page pass budget, so the + // lap takes several runs. The run-wide budget sits above the per-repo one + // and must not disturb that walk when it is the only repo in the list. + const wikis = { "acme/deep": deepWiki(87) }; + const store = makeMultiStore(); + const { env } = makeWikiEnv(); + + const seen = new Set(); + for (let run = 0; run < 5; run++) { + const multi = stubMultiWiki(wikis); + const con = captureConsole(); + try { + await runWikiSurfaces(["acme/deep"], env, store.stub, TICK + run * 3_600_000); + } finally { + con.restore(); + vi.unstubAllGlobals(); + } + for (const file of multi.rawFor("acme/deep")) { + seen.add(decodeURIComponent(file.slice(0, file.lastIndexOf(".")))); + } + // A single-repo run always affords the full per-repo cap. + expect(con.summaryFor("acme/deep")).toContain(`/${PER_REPO_FETCH_CAP} fetches`); + } + + // 87 pages at 20 per run: five runs cover every one of them. + expect(seen.size).toBe(87); + }); + + it("rotates the run order by tick without changing the list", () => { + const repos = repoList(6); + const leaders = new Set(); + for (let hour = 0; hour < repos.length; hour++) { + const ordered = rotateReposForRun(repos, TICK + hour * 3_600_000); + expect([...ordered].sort()).toEqual([...repos].sort()); + leaders.add(ordered[0]); + } + // Every repo leads exactly once across one full rotation. + expect(leaders.size).toBe(repos.length); + // Same tick, same order — the offset is derived, not random. + expect(rotateReposForRun(repos, TICK)).toEqual(rotateReposForRun(repos, TICK)); + expect(rotateReposForRun([], TICK)).toEqual([]); + }); +}); + +// ── Steady-state cost profile and the observed wall (issue #248) ───── + +/** + * IssueStore stand-in that already holds every page of every wiki, hashed to + * match, so the walk's hash comparison skips each one. + * + * This is the profile production actually runs in and the one `deepWiki` gets + * backwards. An unchanged page never reaches `processAndUpsertWikiDoc`, so it + * is charged its fetch attempts and nothing else — a sixth of what a changed + * page costs. A suite that only ever presents changed pages measures the + * expensive shape and leaves the common one untested, which is how a run-wide + * share that cannot engage on the observed tick passed review. + */ +async function makeIndexedStore(wikis: Record) { + const watermarks = new Map(); + const byRepo = new Map(); + + for (const [repo, wiki] of Object.entries(wikis)) { + const records: WikiDocRecord[] = []; + for (const [name, body] of Object.entries(wiki.files)) { + records.push({ + repo, + pageName: name, + extension: "md", + contentHash: await sha256Hex(body), + updatedAt: "2026-08-01T00:00:00Z", + }); + } + byRepo.set(repo, records); + } + + const stub = { + async fetch(request: Request): Promise { + const url = new URL(request.url); + const path = url.pathname; + + if (request.method === "GET" && path === "/wiki-docs") { + return Response.json(byRepo.get(url.searchParams.get("repo") ?? "") ?? []); + } + if (request.method === "GET" && path === "/watermark") { + const wm = watermarks.get(url.searchParams.get("repo") ?? ""); + if (!wm) return new Response("not found", { status: 404 }); + return Response.json(wm); + } + if (request.method === "POST" && path === "/watermark") { + const body = (await request.json()) as { + repo: string; + lastPolledAt: string; + etag?: string; + }; + watermarks.set(body.repo, { + lastPolledAt: body.lastPolledAt, + etag: body.etag ?? "", + }); + return new Response("ok"); + } + return new Response("ok"); + }, + }; + + return { + stub: stub as unknown as DurableObjectStub, + cursor: (repo: string) => watermarks.get(`wiki:${repo}`)?.etag, + }; +} + +/** The repo list and page counts of the 2026-08-15T03:45Z tick, in order. + * `liplus-desktop` has no wiki, so it is absent from the wiki map. */ +const OBSERVED_REPOS = [ + "acme/webhook-mcp", + "acme/rag-mcp", + "acme/desktop", + "acme/language", + "acme/dipper", + "acme/neuron-graph", +]; + +const observedWikis = (): Record => ({ + "acme/webhook-mcp": deepWiki(5), + "acme/rag-mcp": deepWiki(7), + "acme/language": deepWiki(87), + "acme/dipper": deepWiki(5), + "acme/neuron-graph": deepWiki(12), +}); + +const SUBREQUEST_EXHAUSTION = "Too many subrequests by single Worker invocation."; + +describe("poller: wiki steady-state profile and the observed wall", () => { + it("charges an unchanged page only its fetch, not the embed fan-out", async () => { + // The load-bearing asymmetry. Everything below follows from it: the + // accounted cost of a run depends on how much changed, not on page count. + const wikis = { "acme/r00": deepWiki(6) }; + const fresh = makeMultiStore(); + const indexed = await makeIndexedStore(wikis); + const { env } = makeWikiEnv(); + + stubMultiWiki(wikis); + const changed = await pollWiki("acme/r00", env, fresh.stub, { fetchBudget: 20 }); + vi.unstubAllGlobals(); + + stubMultiWiki(wikis); + const unchanged = await pollWiki("acme/r00", env, indexed.stub, { fetchBudget: 20 }); + vi.unstubAllGlobals(); + + expect(changed.embedded).toBe(6); + expect(unchanged.embedded).toBe(0); + expect(unchanged.skipped).toBe(6); + // Same pages, same fetches, and the accounted spend differs by the whole + // embed fan-out. + expect(unchanged.fetches).toBe(changed.fetches); + expect(unchanged.subrequests).toBeLessThan(changed.subrequests / 2); + }); + + it("does not engage the declared share on the observed steady-state run", async () => { + // Honest negative test. On the profile that actually threw in production, + // the accounted spend of the whole six-repo run is far under the declared + // share, so nothing defers and the run proceeds in the order that threw. + // + // This is the limitation stated in `WIKI_SUBREQUEST_BUDGET_PER_RUN`, pinned + // so it cannot quietly stop being true: the share is not what protects this + // shape, and a future change that makes this test fail has either fixed the + // unit gap or broken the cost model — both worth stopping for. + const wikis = observedWikis(); + const store = await makeIndexedStore(wikis); + const multi = stubMultiWiki(wikis); + const { env } = makeWikiEnv(); + const con = captureConsole(); + + try { + await runWikiSurfaces(OBSERVED_REPOS, env, store.stub, TICK); + } finally { + con.restore(); + } + + // Every repo walked; nothing deferred. + for (const repo of OBSERVED_REPOS) { + if (repo === "acme/desktop") continue; // no wiki — skipped, not deferred + expect(multi.rawFor(repo).length).toBeGreaterThan(0); + } + expect(con.lines.some((l) => l.includes("deferred to the next run"))).toBe(false); + + // And the accounted total sits well under the share, which is why. + const total = con.lines + .map((l) => /, (\d+) subrequests,/.exec(l)?.[1]) + .filter((n): n is string => n !== undefined) + .reduce((a, n) => a + Number(n), 0); + expect(total).toBeGreaterThan(0); + expect(total).toBeLessThan(150); + }); + + it("stops the whole run when the invocation reports subrequest exhaustion", async () => { + // What actually covers the observed failure. The share is an estimate that + // may never engage; the wall is observed, and the run ends on it. + const wikis = observedWikis(); + const store = await makeIndexedStore(wikis); + // 14 raw requests in: past the first two repos, inside a later one. + const multi = stubMultiWiki(wikis, { + throwAfter: 14, + throwMessage: SUBREQUEST_EXHAUSTION, + }); + const { env } = makeWikiEnv(); + const con = captureConsole(); + + try { + await runWikiSurfaces(OBSERVED_REPOS, env, store.stub, TICK); + } finally { + con.restore(); + } + + // No repo records a failure for a page it never observed. This is the + // literal acceptance criterion: 8 false failures became 0. + for (const repo of OBSERVED_REPOS) { + const summary = con.summaryFor(repo); + if (summary === undefined) continue; + expect(summary).toContain("0 failed"); + } + // The run stopped rather than grinding every remaining repo into the wall. + expect(con.lines.some((l) => l.includes("out of subrequests (observed)"))).toBe(true); + + const ordered = rotateReposForRun(OBSERVED_REPOS, TICK); + const walked = ordered.filter((r) => multi.rawFor(r).length > 0); + const unreached = ordered.filter((r) => multi.rawFor(r).length === 0); + expect(walked.length).toBeLessThan(ordered.length); + expect(unreached.length).toBeGreaterThan(0); + for (const repo of unreached) { + // Repos the run never reached kept their cursors; the next tick resumes + // them where the previous run left off. + expect(store.cursor(repo)).toBeUndefined(); + } + }); + + it("holds the cursor on the page the wall interrupted", async () => { + // A page whose probes threw was never observed, so the cursor must not + // advance past it — the same treatment the fetch-budget guard gives an + // inconclusive probe, on the measured axis instead of the estimated one. + const wikis = { "acme/r00": deepWiki(9) }; + const store = await makeIndexedStore(wikis); + const { env } = makeWikiEnv(); + + // Let three pages resolve, then hit the wall on the fourth. + stubMultiWiki(wikis, { throwAfter: 3, throwMessage: SUBREQUEST_EXHAUSTION }); + const con = captureConsole(); + let summary; + try { + summary = await pollWiki("acme/r00", env, store.stub, { fetchBudget: 20 }); + } finally { + con.restore(); + } + + expect(summary.exhausted).toBe(true); + expect(summary.visited).toBe(3); + expect(summary.failed).toBe(0); + expect(summary.inconclusive).toBe(0); + // Slug order puts `Home` first, so the three observed pages are Home, p00, + // p01 and the wall lands on p02. The cursor sits on the last page actually + // observed, not on the one that threw. + expect(summary.nextCursor).toBe("p01"); + expect(store.cursor("acme/r00")).toBe("p01"); + // The reap is skipped wholesale — its probes would hit the same wall. + expect(summary.removed).toBe(0); + expect(con.lines.some((l) => l.includes("reap skipped this run"))).toBe(true); + }); + + it("keeps an ordinary network error on the per-page axis", async () => { + // The contrast that makes the exhaustion branch meaningful. A blip is this + // page's problem: it counts as inconclusive, the cursor advances past it, + // and the run carries on to the remaining repos. + const wikis = { "acme/r00": deepWiki(4), "acme/r01": deepWiki(4) }; + const store = await makeIndexedStore(wikis); + const multi = stubMultiWiki(wikis, { throwAfter: 0 }); + const { env } = makeWikiEnv(); + const con = captureConsole(); + + try { + await runWikiSurfaces(["acme/r00", "acme/r01"], env, store.stub, TICK); + } finally { + con.restore(); + } + + // Both repos were walked — a blip does not end the run. + expect(multi.rawFor("acme/r00").length).toBeGreaterThan(0); + expect(multi.rawFor("acme/r01").length).toBeGreaterThan(0); + expect(con.lines.some((l) => l.includes("out of subrequests (observed)"))).toBe(false); + for (const repo of ["acme/r00", "acme/r01"]) { + expect(con.summaryFor(repo)).toContain("0 failed"); + expect(con.summaryFor(repo)).not.toContain("0 inconclusive"); + } + }); + + it("classifies the exhaustion message and nothing else", () => { + expect(isSubrequestExhaustion(new Error(SUBREQUEST_EXHAUSTION))).toBe(true); + expect(isSubrequestExhaustion(new Error("Too many subrequests"))).toBe(true); + expect(isSubrequestExhaustion(new Error("Network connection lost."))).toBe(false); + expect(isSubrequestExhaustion(new Error("429 Too Many Requests"))).toBe(false); + expect(isSubrequestExhaustion("Too many subrequests")).toBe(true); + expect(isSubrequestExhaustion(undefined)).toBe(false); + }); +}); diff --git a/src/poller.ts b/src/poller.ts index 44940b1..f574468 100644 --- a/src/poller.ts +++ b/src/poller.ts @@ -1302,10 +1302,180 @@ const MAX_WIKI_DELETIONS_PER_REPO_PER_RUN = 5; * * Sized at 3x the delete budget. A probe costs at most `WIKI_EXTENSIONS.length` * = 2 subrequests, so this ceiling is 30, alongside the ~4-per-delete fan-out - * of at most 5 deletes. Both are spent outside the walk's fetch budget and stay - * well inside the Worker's 1000-subrequest invocation ceiling (issue #130). */ + * of at most 5 deletes. Both are spent outside the walk's fetch budget — but + * not outside the run's: they are charged against + * `WIKI_SUBREQUEST_BUDGET_PER_RUN` when spent, which is what keeps a reap-heavy + * repo from starving the ones behind it (issue #248). They are not *reserved* + * at admission, so this pair is the overrun an admitted pass may add. */ const MAX_WIKI_REAP_PROBES_PER_REPO_PER_RUN = 15; +/** Subrequests the wiki surface may spend in one `WIKI_CRON` invocation. + * + * Every budget above this one is **per repo**. Each repo therefore keeps its + * own contract while nothing watches the sum, and the sum is what Cloudflare + * charges: the wiki cron walks `POLL_REPOS` inside a single invocation, so the + * ceiling is shared across every repo in the list. On the 2026-08-15T03:45Z + * tick five repos indexed normally and the sixth had every raw-content probe + * rejected with `Too many subrequests by single Worker invocation`, reporting + * 8 failures for pages nobody had looked at. The starvation lands wholly on + * whichever repo the loop reaches last, and that is a property of the loop, + * not of the repo. + * + * **Read this before trusting the number.** The unit here is call sites this + * poller can see — one per `fetch`, per Durable Object round trip, per D1 + * statement, per Vectorize call, per Workers AI call — and that is not the + * unit Cloudflare charges. Two facts fix how far this share can be trusted: + * + * 1. The gap is large and unexplained. In these units the observed run had + * spent somewhere between ~70 (every page unchanged) and ~292 (every page + * re-embedded) when the sixth repo's first probe threw, against a ceiling + * documented as 1000. Nothing visible here accounts for the difference, so + * at least one binding costs more than the 1 this file gives it. Which one, + * and by how much, is unmeasured — and Workers expose no runtime counter to + * measure it with, so no constant in this file can be calibrated against it. + * 2. The cost profile spans that whole range on its own. An unchanged page is + * charged only its fetch attempts, because `WIKI_SUBREQUESTS_PER_EMBED` is + * added past the hash-comparison `continue`. A steady-state run over + * already-indexed wikis therefore costs a fraction of a bulk-import run + * over the same page counts. + * + * Together those mean **this share does not address the failure that opened + * the issue.** At 250 the observed steady-state shape defers nothing and the + * run proceeds in exactly the order that threw. What covers that case is + * `isSubrequestExhaustion` — the exhaustion is *observable* when it happens, + * which is a measurement rather than an estimate, and the walk stops the run + * on it. Do not read this constant as the protection; read it as a bound on + * the shape it can actually bound. + * + * What it does bound: bulk import, where each fetched page carries the embed + * fan-out at `WIKI_SUBREQUESTS_PER_FETCHED_PAGE`, so a handful of repos with + * changed wikis reaches 250 well before a steady-state run would. That is the + * runaway shape — a repo appended to `POLL_REPOS`, or a wiki newly enabled — + * and bounding it costs only latency, since bulk imports already span runs. + * `DIFF_SUBREQUEST_BUDGET_PER_RUN` is 900 because the diff surface's per-file + * estimate was derived against measured rejections; this one has no such + * derivation and must not borrow that figure's confidence. Its absolute value + * is a judgement inside the observed range, not a calibration. Moving it + * requires a measurement, and the only one that would settle it is the real + * per-binding cost. + * + * This is a ceiling over the per-repo budgets, not a replacement for them. + * `wikiFetchBudgetForPass` converts what is left of it into the pass's fetch + * cap, and `runWikiSurfaces` subtracts each pass's measured spend. */ +const WIKI_SUBREQUEST_BUDGET_PER_RUN = 250; + +/** + * Recognise Cloudflare's subrequest-exhaustion error. + * + * This is the one part of the run axis that is *measured* rather than + * estimated. Every constant in this file counts call sites, which is not the + * unit the ceiling is charged in (see `WIKI_SUBREQUEST_BUDGET_PER_RUN`), so no + * declared share can be relied on to engage before the real limit. The limit + * being hit, however, is directly observable: the binding throws, and it names + * itself. + * + * The exhaustion is invocation-scoped and terminal — nothing is returned to the + * budget mid-invocation, so once one subrequest is rejected every later one in + * the same invocation is rejected too. The observed run demonstrates exactly + * that: the Worker kept running and kept throwing, probe after probe, and the + * walk kept converting those throws into recorded page failures. So the correct + * response to the first sighting is to stop the run, not to continue and retry. + * + * Matched on the message rather than an error type because Workers surfaces it + * as a plain `Error`. Substring rather than equality because the text carries a + * trailing clause ("...by single Worker invocation.") that is not contractual. + */ +export function isSubrequestExhaustion(err: unknown): boolean { + const message = err instanceof Error ? err.message : String(err); + return message.includes("Too many subrequests"); +} + +/** Subrequests one changed wiki page costs downstream of its raw fetch, read + * off `processAndUpsertWikiDoc`: the Workers AI embed, the Vectorize upsert, + * the D1 FTS mirror write, the Store DO row, and *two* for the graph-edge + * index — `indexWikiEdges` queries the repo's known slugs before it batches + * the edge writes, so it is two round trips rather than the one its single + * call site suggests. Charged per *embedded* page; an unchanged page stops at + * its hash comparison and costs only its fetch. */ +const WIKI_SUBREQUESTS_PER_EMBED = 6; + +/** Subrequests one reaped wiki page costs: Vectorize delete, FTS5 row delete, + * graph-edge delete, Store DO row delete. */ +const WIKI_SUBREQUESTS_PER_DELETE = 4; + +/** Subrequests one repo's pass spends outside the page walk and the reap: + * the `.wiki.git` existence probe, the `_pages` scrape, the resume-cursor + * read, the lap-anchor read, the store's wiki-doc snapshot, the D1 query for + * indexed pages, and the two watermark writes at the end. */ +const WIKI_REPO_OVERHEAD_SUBREQUESTS = 8; + +/** Worst-case subrequests one fetched page costs: the fetch attempt itself + * plus the embed fan-out it may authorize. Used to convert a remaining + * subrequest budget into a fetch-attempt cap. */ +const WIKI_SUBREQUESTS_PER_FETCHED_PAGE = 1 + WIKI_SUBREQUESTS_PER_EMBED; + +/** + * Fetch attempts one repo's wiki pass may issue, given what is left of the + * invocation's declared share. + * + * The run-wide budget is spent by subtraction rather than by division: a repo + * with a five-page wiki and nothing changed returns almost all of its share, so + * the repos behind it keep the full per-repo cap instead of a sixth of it. The + * diff surface divides instead (`diffFileBudgetPerPhase`) because its phases + * are bounded per repo up front; the wiki walk's cost is only known once it has + * run. + * + * Returns 0 when the remainder cannot cover the pass's overhead plus one page. + * The caller must then skip the repo entirely rather than call `pollWiki` with + * a budget of zero: a pass always lets its *first* page run its whole candidate + * list (issue #192), so a zero budget would be exceeded rather than respected, + * and the pass would spend its overhead for one page's worth of progress. + * + * The reap's own budgets are deliberately **not** reserved here, though they + * are charged in full once spent. Reserving each repo's worst-case reap (50) + * against a share of 250 would tax every pass for work the reap almost never + * does — the candidate set is empty on a healthy wiki — and would defer repos + * on a run where nothing happened at all. What absorbs an admitted pass's reap + * instead is the margin between this declared share and the real ceiling, which + * is the same margin the unit mismatch above already demands. + * + * @param remaining subrequests still unspent in this invocation's share + */ +export function wikiFetchBudgetForPass(remaining: number): number { + const spendable = remaining - WIKI_REPO_OVERHEAD_SUBREQUESTS; + if (spendable < WIKI_SUBREQUESTS_PER_FETCHED_PAGE) return 0; + return Math.min( + MAX_WIKI_FETCHES_PER_REPO_PER_RUN, + Math.floor(spendable / WIKI_SUBREQUESTS_PER_FETCHED_PAGE), + ); +} + +/** + * Rotate the repo list so a different repo leads each run. + * + * A run-wide budget spent in list order always thins the same tail: the repos + * ahead take what they need and whoever is last takes the remainder, cron after + * cron. That is the same shape as the defect this budget removes — the loop + * position decides the outcome, not the repo — so bounding the overrun without + * moving the position would leave the unfairness intact and merely quiet. + * + * The offset is derived from the tick rather than stored, so it needs no + * watermark and no schema, and it is deterministic for a given scheduled time, + * which is what makes it testable. With an hourly cron and N repos every repo + * leads once every N hours. + * + * @param repos the configured list, unmodified + * @param tick the cron's scheduled time in epoch milliseconds + */ +export function rotateReposForRun(repos: readonly T[], tick: number): T[] { + if (repos.length === 0) return []; + const hours = Math.floor(tick / 3_600_000); + // `%` on a negative tick yields a negative index; epoch times before 1970 are + // not reachable here, but the guard costs nothing and removes the branch. + const offset = ((hours % repos.length) + repos.length) % repos.length; + return repos.map((_, i) => repos[(i + offset) % repos.length]); +} + /** Fraction of the indexed page set which, once the reap candidate set reaches * it, is logged as an anomaly. Warn-only on purpose: a ratio cannot separate a * legitimate bulk cleanup (wiki tidy-up, mass rename) from an enumeration that @@ -1629,6 +1799,14 @@ async function writeWikiLapAnchor( * * Returns null when no extension matches (page may have been deleted, renamed, * or moved to an unsupported format). + * + * `errored` separates an *observed* miss from an unobserved one. A candidate + * that answered 404 says the file is not there; a candidate whose `fetch()` + * threw says nothing at all — and subrequest exhaustion throws, so under a + * blown invocation budget every candidate errors and the caller used to log + * "all candidates 404" for pages it had never reached (issue #248). The two + * outcomes are indistinguishable in the return value alone, so the flag + * carries the difference out. */ async function fetchWikiContent( repo: string, @@ -1638,6 +1816,14 @@ async function fetchWikiContent( ): Promise<{ result: { content: string; extension: string } | null; attempts: number; + /** True when at least one candidate threw instead of answering. */ + errored: boolean; + /** True when a candidate threw *because the invocation is out of + * subrequests*. Distinct from `errored`: a network blip is this page's + * problem, whereas exhaustion is the whole run's and nothing after it can + * succeed. The caller stops the run on it rather than moving to the next + * page (issue #248). */ + exhausted: boolean; }> { const exts = preferredExtension ? [preferredExtension, ...WIKI_EXTENSIONS.filter((e) => e !== preferredExtension)] @@ -1654,9 +1840,11 @@ async function fetchWikiContent( // whose title-derived name misses is resolved by the slug on the *second* // attempt rather than after the whole extension set has been walked. let attempts = 0; + let errored = false; + let exhausted = false; for (const ext of exts) { for (const name of names) { - if (attempts >= maxAttempts) return { result: null, attempts }; + if (attempts >= maxAttempts) return { result: null, attempts, errored, exhausted }; const url = `https://raw.githubusercontent.com/wiki/${repo}/${encodeURIComponent(name)}.${ext}`; attempts++; try { @@ -1667,9 +1855,23 @@ async function fetchWikiContent( return { result: { content: await resp.text(), extension: ext }, attempts, + errored, + exhausted, }; } } catch (err) { + errored = true; + if (isSubrequestExhaustion(err)) { + // Every remaining candidate would throw the same way, so stop + // probing this page rather than spending the rest of the list + // proving it. + exhausted = true; + console.error( + `fetchWikiContent probe ${ext} failed for ${repo}/${name}: ` + + `invocation is out of subrequests — abandoning this page's probes.`, + ); + return { result: null, attempts, errored, exhausted }; + } console.error( `fetchWikiContent probe ${ext} failed for ${repo}/${name}:`, err instanceof Error ? err.message : String(err), @@ -1677,7 +1879,7 @@ async function fetchWikiContent( } } } - return { result: null, attempts }; + return { result: null, attempts, errored, exhausted }; } /** Verdict of the pre-reap existence probe. `gone` is the only one that @@ -1723,29 +1925,41 @@ async function probeWikiPageAlive( repo: string, pageName: string, preferredExtension?: string, -): Promise { +): Promise<{ verdict: WikiReapProbe; attempts: number; exhausted: boolean }> { const exts = preferredExtension ? [preferredExtension, ...WIKI_EXTENSIONS.filter((e) => e !== preferredExtension)] : Array.from(WIKI_EXTENSIONS); let inconclusive = false; + let attempts = 0; for (const ext of exts) { const url = `https://raw.githubusercontent.com/wiki/${repo}/${encodeURIComponent(pageName)}.${ext}`; + attempts++; try { const resp = await fetch(url, { headers: { "User-Agent": "github-rag-mcp/0.1.0" }, }); - if (resp.ok) return "alive"; + if (resp.ok) return { verdict: "alive", attempts, exhausted: false }; if (resp.status !== 404) inconclusive = true; } catch (err) { + inconclusive = true; + if (isSubrequestExhaustion(err)) { + // `inconclusive` already withholds the delete, which is the safe + // verdict; the flag is what stops the reap loop from probing the rest + // of the candidate list into the same wall (issue #248). + console.error( + `Reap probe ${ext} failed for wiki ${repo}/${pageName}: ` + + `invocation is out of subrequests — reap abandoned for this run.`, + ); + return { verdict: "inconclusive", attempts, exhausted: true }; + } console.error( `Reap probe ${ext} failed for wiki ${repo}/${pageName}:`, err instanceof Error ? err.message : String(err), ); - inconclusive = true; } } - return inconclusive ? "inconclusive" : "gone"; + return { verdict: inconclusive ? "inconclusive" : "gone", attempts, exhausted: false }; } /** Budget / cursor overrides for one wiki poll pass. Defaults are the cron @@ -1777,9 +1991,23 @@ export interface WikiPollSummary { fetches: number; /** Pages whose content was examined this pass. */ visited: number; + /** Subrequests this pass spent, counted at every call site that issues one. + * The run-wide walker subtracts this from the invocation's declared share, + * so the figure is an accounting input rather than a diagnostic: the fan-out + * of an embed and of a reap is charged by the constants above, and every + * HTTP / DO / D1 round-trip the pass makes itself is charged as one. */ + subrequests: number; embedded: number; skipped: number; + /** Pages whose every candidate answered an observed 404. A real miss. */ failed: number; + /** Pages whose candidates all *threw* rather than answering, so absence was + * never observed. Kept off `failed` because recording a failure for a page + * nobody reached is what made the wiki surface's own exhaustion read as a + * property of the last repo's wiki (issue #248). The cursor still advances: + * the page is retried on the next lap, whereas holding it would let one + * permanently unreachable page stall the walk. */ + inconclusive: number; removed: number; /** Orphan candidates the reap loop never reached, because it stopped on the * delete or the probe budget. A withheld candidate *was* reached, so it is @@ -1803,6 +2031,12 @@ export interface WikiPollSummary { wrapped: boolean; /** False when the `_pages` index could not be read; no reaping happened. */ enumerated: boolean; + /** True when the invocation ran out of subrequests during this pass. The walk + * stopped at that page with the cursor unmoved and the reap was skipped, and + * the caller must not start another repo — the budget is invocation-scoped, + * so nothing after it can succeed either (issue #248). Unlike every other + * budget signal on this surface, this one is observed rather than estimated. */ + exhausted: boolean; } /** @@ -1836,6 +2070,11 @@ export interface WikiPollSummary { * cap is two budgets, not one: a withheld candidate spends a probe but no * delete slot, so it cannot hold the head of the sorted candidate list and * starve the real deletions behind it (issue #197). + * + * Every budget this function holds is scoped to one repo. The invocation + * ceiling they are really spending against is shared with every other repo in + * the run, and nothing inside one pass can see that; the pass therefore reports + * its measured `subrequests` and `runWikiSurfaces` owns the run axis (#248). */ export async function pollWiki( repo: string, @@ -1849,14 +2088,20 @@ export async function pollWiki( const probeBudget = MAX_WIKI_REAP_PROBES_PER_REPO_PER_RUN; const persistCursor = opts.persistCursor ?? true; + // Charged at every site that issues a subrequest, so an early return carries + // out what it actually spent rather than a nominal per-repo figure. + let subrequests = 0; + const empty = (startCursor: string): WikiPollSummary => ({ repo, pages: 0, fetches: 0, + subrequests, visited: 0, embedded: 0, skipped: 0, failed: 0, + inconclusive: 0, removed: 0, orphansDeferred: 0, orphansWithheld: 0, @@ -1865,16 +2110,19 @@ export async function pollWiki( lapAnchor: startCursor, wrapped: false, enumerated: false, + exhausted: false, }); // Cheap existence probe so repos without wiki incur a single HEAD-equivalent // round-trip per cron run instead of three (probe + index + content). + subrequests++; const hasWiki = await wikiExists(repo); if (!hasWiki) { console.log(`${repo} wiki: not enabled or not accessible — skip`); return empty(""); } + subrequests++; const index = await listWikiPages(repo); const pages = index.pages; if (!index.ok) { @@ -1887,18 +2135,21 @@ export async function pollWiki( // enumeration, so a readable index always yields at least one page. An empty // set only means the scrape failed, which the guard above already returned on. + if (opts.cursor === undefined) subrequests++; const startCursor = opts.cursor ?? (await readWikiCursor(storeStub, repo)); // The lap anchor marks where the current sweep began. An explicit cursor // override is an operator saying "start the walk here", so it opens a fresh // lap at that point; otherwise the stored anchor carries across passes and // falls back to the current cursor the first time a repo is walked. + if (opts.cursor === undefined) subrequests++; const storedAnchor = opts.cursor !== undefined ? null : await readWikiLapAnchor(storeStub, repo); const lapAnchor = storedAnchor ?? startCursor; // Snapshot the existing wiki doc records so we can detect deletes and pick // a per-page preferred extension on subsequent polls. + subrequests++; const existingResp = await storeStub.fetch( new Request(`http://store/wiki-docs?repo=${encodeURIComponent(repo)}`), ); @@ -1911,12 +2162,14 @@ export async function pollWiki( let embedded = 0; let skipped = 0; let failed = 0; + let inconclusive = 0; let removed = 0; let orphansWithheld = 0; let fetches = 0; let visited = 0; let nextCursor = startCursor; let wrapped = false; + let exhausted = false; // Resume at the first page ordering strictly after the cursor; wrap to the // head when the cursor sits at (or past) the end, or when it names a page @@ -1957,14 +2210,40 @@ export async function pollWiki( // it found it, so a budget below one page's candidate count stalls the walk // forever instead of self-healing on the next pass (issue #192). Letting the // first page finish its probes costs at most `candidates - 1` extra - // subrequests, once per pass, against an invocation budget of 1000. - const { result: fetched, attempts } = await fetchWikiContent( + // subrequests, once per pass. What absorbs that overrun is the margin under + // `WIKI_SUBREQUEST_BUDGET_PER_RUN`, not the documented 1000 this comment + // used to invoke: the run's real headroom is the declared share, and in the + // accounting that share is calibrated in, 1000 is not a reachable figure + // (issue #248). The exemption is admitted on the same terms either way — + // `wikiFetchBudgetForPass` never admits a pass it cannot fund a page for. + const { + result: fetched, + attempts, + errored, + exhausted: pageExhausted, + } = await fetchWikiContent( repo, page, prior?.extension, visited === 0 ? Number.POSITIVE_INFINITY : fetchBudget - fetches, ); fetches += attempts; + subrequests += attempts; + + if (pageExhausted) { + // The invocation is out of subrequests. Nothing about this page was + // observed, so break *before* the cursor moves and leave it for the next + // run — the same treatment the fetch-budget guard below gives an + // inconclusive probe, on the axis that is measured rather than estimated. + // Recording a failure here instead is the whole defect: 8 present pages + // reported missing because the walk kept going after the wall (#248). + exhausted = true; + console.warn( + `${repo} wiki: invocation out of subrequests while probing ${page.slug}. ` + + `Cursor held at ${nextCursor || ""}; the run stops here and resumes next tick.`, + ); + break; + } if (!fetched && visited > 0 && fetches >= fetchBudget) { // The budget ran out inside this page's candidate list, so "no content" @@ -1988,8 +2267,24 @@ export async function pollWiki( // The slug was discovered in `_pages` but no extension served. Treat as // a transient miss and skip — the next poll will retry without spending // an embedding budget here. - console.warn(`No content fetched for ${repo}/wiki/${page.slug} (all candidates 404)`); - failed++; + // + // Which counter it lands in turns on whether the miss was *observed*. + // Every candidate answering 404 is an observation; a candidate whose + // `fetch()` threw is not, and the old message asserted the first shape + // for both. Under subrequest exhaustion — which throws — that read as + // eight 404s on a wiki whose pages were all present (issue #248). + if (errored) { + inconclusive++; + console.warn( + `No content fetched for ${repo}/wiki/${page.slug} (probe threw; absence not ` + + `observed — see the fetchWikiContent errors above for the cause). Retried next lap.`, + ); + } else { + failed++; + console.warn( + `No content fetched for ${repo}/wiki/${page.slug} (every candidate answered 404)`, + ); + } continue; } @@ -1999,6 +2294,7 @@ export async function pollWiki( continue; } + subrequests += WIKI_SUBREQUESTS_PER_EMBED; const result = await processAndUpsertWikiDoc( env, storeStub, @@ -2019,7 +2315,17 @@ export async function pollWiki( // the live FTS index: a page missing from the store but still in search_docs // is exactly the case a store-only diff cannot see, and it is the one that // actually happened in production (issue #184, cause E). - const indexed = await listIndexedWikiPages(env, repo); + // Skipped wholesale once the walk hit the wall: the reap's first act is a D1 + // query and its per-candidate probes are subrequests too, so every one of + // them would throw. Withholding a reap costs nothing — the candidate set only + // shrinks, so the drain resumes next run (issue #248). + if (exhausted) { + console.warn(`${repo} wiki: reap skipped this run — invocation out of subrequests.`); + } + if (!exhausted) subrequests++; + const indexed = exhausted + ? { pages: [] as string[], ok: false } + : await listIndexedWikiPages(env, repo); const orphanSet = new Set(); for (const w of existing) { if (!currentSlugs.has(w.pageName)) orphanSet.add(w.pageName); @@ -2037,7 +2343,7 @@ export async function pollWiki( ...existing.map((w) => w.pageName), ...indexed.pages, ]).size; - if (indexedTotal > 0 && orphans.length / indexedTotal >= WIKI_ORPHAN_RATIO_WARN) { + if (!exhausted && indexedTotal > 0 && orphans.length / indexedTotal >= WIKI_ORPHAN_RATIO_WARN) { console.warn( `${repo} wiki: reap set is ${orphans.length}/${indexedTotal} of the indexed pages ` + `(>= ${WIKI_ORPHAN_RATIO_WARN}). Legitimate bulk deletion and a short ` + @@ -2052,17 +2358,28 @@ export async function pollWiki( // for the enumeration to recover (issue #197). let probes = 0; for (const pageName of orphans) { - if (removed >= deleteBudget || probes >= probeBudget) break; + // `exhausted` joins the two budgets as a third stop condition. Every + // candidate left over is reported as deferred, which is exactly what it is. + if (exhausted || removed >= deleteBudget || probes >= probeBudget) break; // Existence check before the delete. The candidate is only "orphaned" as // far as the enumeration knows, and the enumeration is exactly what may // have come back short (issue #187). probes++; - const probe = await probeWikiPageAlive( - repo, - pageName, - existingMap.get(pageName)?.extension, - ); + const { + verdict: probe, + attempts: probeAttempts, + exhausted: probeExhausted, + } = await probeWikiPageAlive(repo, pageName, existingMap.get(pageName)?.extension); + subrequests += probeAttempts; + if (probeExhausted) { + // First sighting of the wall can land here rather than in the walk, when + // the walk finished inside its fetch budget. Same response: stop, withhold, + // let the caller end the run. + exhausted = true; + orphansWithheld++; + continue; + } if (probe !== "gone") { orphansWithheld++; console.warn( @@ -2076,6 +2393,7 @@ export async function pollWiki( } const wvid = await wikiDocVectorId(repo, pageName); + subrequests += WIKI_SUBREQUESTS_PER_DELETE; // Each surface is torn down independently: a Vectorize failure must not // strand the D1 rows, which are the ones users actually retrieve. for (const [surface, run] of [ @@ -2123,29 +2441,36 @@ export async function pollWiki( if (persistCursor) { if (nextCursor !== startCursor) { + subrequests++; await writeWikiCursor(storeStub, repo, nextCursor); } if (nextLapAnchor !== storedAnchor) { + subrequests++; await writeWikiLapAnchor(storeStub, repo, nextLapAnchor); } } console.log( `${repo} wiki: ${pages.length} pages, ${visited} visited, ${fetches}/${fetchBudget} fetches, ` + - `${embedded} embedded, ${skipped} unchanged, ${failed} failed, ${removed} deleted, ` + + `${subrequests} subrequests, ` + + `${embedded} embedded, ${skipped} unchanged, ${failed} failed, ` + + `${inconclusive} inconclusive, ${removed} deleted, ` + `${orphansWithheld} reap withheld, ` + `cursor ${startCursor || ""} -> ${nextCursor || ""}, ` + - `lap ${lapAnchor || ""}${wrapped ? " complete" : ` -> ${lapFinalSlug}`}`, + `lap ${lapAnchor || ""}${wrapped ? " complete" : ` -> ${lapFinalSlug}`}` + + `${exhausted ? ", SUBREQUESTS EXHAUSTED" : ""}`, ); return { repo, pages: pages.length, fetches, + subrequests, visited, embedded, skipped, failed, + inconclusive, removed, orphansDeferred, orphansWithheld, @@ -2154,6 +2479,7 @@ export async function pollWiki( lapAnchor, wrapped, enumerated: true, + exhausted, }; } @@ -3152,14 +3478,91 @@ async function runWikiSurface( repo: string, env: Env, storeStub: DurableObjectStub, -): Promise { + fetchBudget: number, +): Promise<{ spent: number; exhausted: boolean }> { try { - await pollWiki(repo, env, storeStub); + const summary = await pollWiki(repo, env, storeStub, { fetchBudget }); + return { spent: summary.subrequests, exhausted: summary.exhausted }; } catch (err) { console.error( `Failed to poll wiki for ${repo}:`, err instanceof Error ? err.message : String(err), ); + // A throw leaves the spend unmeasured, and the pass had at least been + // admitted, so charge it its overhead rather than nothing. Charging zero + // would let a repo that fails early hand the whole share on to the next one + // and reintroduce the overrun from the far side. + // + // An exhaustion that escapes as a throw rather than a summary flag — from a + // store or D1 call outside the paths that classify it — still ends the run. + // The budget is invocation-scoped, so the next repo would only rediscover + // the same wall at the cost of its own overhead. + return { + spent: WIKI_REPO_OVERHEAD_SUBREQUESTS, + exhausted: isSubrequestExhaustion(err), + }; + } +} + +/** + * Walk every repo's wiki within one invocation's declared subrequest share. + * + * Two things happen here that a plain `for` over `POLL_REPOS` cannot do. + * + * The share is spent by subtraction: each pass reports what it actually cost + * and the remainder carries to the next repo, so a repo that finds nothing + * changed hands its budget on almost whole. When the remainder can no longer + * fund a pass, the remaining repos are **not called at all** — their cursors + * stay where the last run left them and the next invocation resumes there. + * That is the same "hit the ceiling, keep the cursor, finish clean" behaviour a + * single deep wiki already had on the per-repo fetch budget, lifted to the run + * axis. Skipping is what keeps the alternative off the table: calling `pollWiki` + * with nothing left produced probe exceptions the walk then recorded as page + * failures, for pages it had never reached (issue #248). + * + * The order rotates per tick, because a budget spent in list order starves + * whoever is last, every run (`rotateReposForRun`). + */ +export async function runWikiSurfaces( + repos: readonly string[], + env: Env, + storeStub: DurableObjectStub, + tick: number, +): Promise { + const ordered = rotateReposForRun(repos, tick); + let remaining = WIKI_SUBREQUEST_BUDGET_PER_RUN; + const deferred: string[] = []; + let hitTheWall = false; + + for (const repo of ordered) { + if (hitTheWall) { + deferred.push(repo); + continue; + } + const fetchBudget = wikiFetchBudgetForPass(remaining); + if (fetchBudget === 0) { + deferred.push(repo); + continue; + } + const { spent, exhausted } = await runWikiSurface(repo, env, storeStub, fetchBudget); + remaining -= spent; + hitTheWall = exhausted; + } + + if (deferred.length > 0) { + // Two different reasons land here and the log must not blur them: the + // declared share is an estimate that may never engage, while the wall is an + // observation that already has (`WIKI_SUBREQUEST_BUDGET_PER_RUN`). + const cause = hitTheWall + ? `Invocation ran out of subrequests (observed). The declared share of ` + + `${WIKI_SUBREQUEST_BUDGET_PER_RUN} did not engage first — it counts call sites, ` + + `not what Cloudflare charges.` + : `Wiki run share exhausted (${WIKI_SUBREQUEST_BUDGET_PER_RUN} accounted subrequests).`; + console.warn( + `${cause} ${deferred.length} repo(s) deferred to the next run with cursors ` + + `unmoved: ${deferred.join(", ")}. Rotation advances each by one position per ` + + `tick, so every repo leads within ${ordered.length} ticks.`, + ); } } @@ -3232,9 +3635,9 @@ export async function handleScheduled( } if (controller.cron === WIKI_CRON) { - for (const repo of repos) { - await runWikiSurface(repo, env, storeStub); - } + // Not a plain loop: the wiki surface's budgets are all per repo, so the + // sum across the list is what overruns the invocation ceiling (issue #248). + await runWikiSurfaces(repos, env, storeStub, controller.scheduledTime); return; }