Paginate mylist sync to handle TorBox's 10,000-item per-response cap#2
Merged
Merged
Conversation
TorBox's /torrents/mylist and /usenet/mylist cap each response at ~10,000 items regardless of the requested `limit`. listGeneric made a single, un-paginated call, so on accounts with more than 10k torrents the OLDEST items were silently dropped from the sync — they never entered the WebDAV tree and their files became unbrowsable/unplayable, while the newest 10k masked the loss. Page through with `offset` in fixed windows (listPageSize = 1000) until a short page signals the end, accumulating every item. `params.Limit`, when > 0, now acts as a ceiling on the TOTAL number fetched rather than a per-request limit. Observed on a ~10.9k-torrent account: a single `limit=50000` call returned exactly 10,000; the remaining ~899 (oldest) torrents were missing from the mount and only reappeared after pagination. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fredddi43
force-pushed
the
paginate-mylist-sync
branch
from
a1ea1ba to
7f87b71
Compare
There was a problem hiding this comment.
Pull request overview
This PR fixes TorBox “mylist” sync truncation for accounts with more than ~10,000 items by paginating listGeneric requests using offset and accumulating results until the final (short) page is reached, ensuring older torrents/usenet entries are included in the local SQLite tree and thus exposed via WebDAV.
Changes:
- Add pagination to
Client.listGenericusing a fixedlistPageSizewindow andoffsetiteration. - Redefine
ListFilesParams.Limitsemantics to act as a total fetch ceiling (rather than a per-request limit). - Add a regression test to ensure pagination continues past the per-response cap behavior.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| internal/torbox/client.go | Implements paginated fetching for TorBox mylist endpoints and returns accumulated results. |
| internal/torbox/client_test.go | Updates param expectations and adds a pagination regression test. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+169
to
184
| for { | ||
| u := base.JoinPath(endpoint) | ||
| q := u.Query() | ||
| q.Set("bypass_cache", strconv.FormatBool(params.BypassCache)) | ||
| q.Set("offset", strconv.Itoa(offset)) | ||
| q.Set("limit", strconv.Itoa(listPageSize)) | ||
| u.RawQuery = q.Encode() | ||
|
|
||
| req, err := http.NewRequestWithContext(ctx, http.MethodGet, u.String(), http.NoBody) | ||
| if err != nil { | ||
| return nil, fmt.Errorf("torbox: creating %s request: %w", label, err) | ||
| } | ||
| req.Header.Set("Authorization", "Bearer "+c.apiKey) | ||
|
|
||
| slog.Debug("torbox "+label, "offset", params.Offset, "limit", params.Limit) | ||
| slog.Debug("torbox "+label, "offset", offset, "limit", listPageSize) | ||
|
|
Comment on lines
+416
to
+428
| server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { | ||
| calls++ | ||
| var data []Torrent | ||
| switch r.URL.Query().Get("offset") { | ||
| case "0": | ||
| data = make([]Torrent, listPageSize) // full page → more follow | ||
| case strconv.Itoa(listPageSize): | ||
| data = make([]Torrent, 7) // short page → end | ||
| default: | ||
| t.Errorf("unexpected offset %q", r.URL.Query().Get("offset")) | ||
| } | ||
| json.NewEncoder(w).Encode(apiResponse[[]Torrent]{Data: data, Success: boolPtr(true)}) | ||
| })) |
mainlink0435
added a commit
that referenced
this pull request
Jun 26, 2026
sync.limit was a holdover from before PR #2's pagination fix. It capped the number of items fetched per sync cycle, but the pagination loop already handles termination via the short-page signal. Removing the ceiling means all torrents and Usenet items are always synced — no silent data loss on larger accounts. Deleted: - Limit field from ListFilesParams, SyncConfig, SyncWorker, server Config - maxTotal ceiling check in listGeneric loop - SyncLimit display from landing page - sync.limit config key, defaults, validation, tests - All docs references to sync.limit
Fredddi43
pushed a commit
to Fredddi43/warpbox
that referenced
this pull request
Jun 26, 2026
…ize) The const listPageSize (1000) from PR mainlink0435#2 is now an adjustable config parameter with default 5000. A new sync.list_page_size config option controls the per-request window, with a safe fallback defaultListPageSize (1000) if the param is not set. Changes: - internal/config/config.go — ListPageSize field, default 5000, validate 1–10000 - internal/torbox/client.go — PageSize on ListFilesParams, renamed const - internal/metadata/sync.go — listPageSize field threaded through to API calls - cmd/warpbox/main.go — pass cfg.Sync.ListPageSize to NewSyncWorker - config.yml.example, config.yml — document the new option - docs/config-tuning.md, docs/tech-spec.md — reference and profile updates - config_test.go, sync_test.go — test callers updated
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this fix?
warpbox's metadata sync silently drops the oldest torrents on accounts with
more than 10k torrents. TorBox's
mylistendpoints cap each response at ~10,000items regardless of the
limitparam, andlistGenericmade a single,un-paginated call — so only the newest 10,000 ever synced. The missing torrents
never enter the SQLite tree, so they don't appear over WebDAV and their files
are unbrowsable/unplayable.
This paginates
listGenericwithoffsetin fixedlistPageSize(1000)windows until a short page ends the list, accumulating all items.
ListFilesParams.Limit, when > 0, becomes a ceiling on the total fetched ratherthan a per-request limit. Both torrents and usenet go through
listGeneric, soboth are fixed.
refs #1
How to test
GET /v1/api/torrents/mylist?bypass_cache=true&limit=50000&offset=0returnsexactly 10,000 items.
(
/webdav/__all__/) entry count to your real torrent count:list and play.
CGO_ENABLED=1 go test ./internal/torbox/—TestListPaginatesPastCapcovers full-page → short-page accumulation across multiple requests.
Checklist
CGO_ENABLED=1 go test ./...)go vet ./...is cleanrefs #1)🤖 Generated with Claude Code