🎯 Fix premature termination in API pagination when page size is less than requested

[EncarnacionP]

📝 Description

The CLI's API client currently handles pagination by checking if the number of items returned in a single page is less than the requested page size (e.g., per_page). If it is, the pagination loop terminates, assuming there are no more records to fetch.

However, the GitHub API (especially search endpoints or under specific load-balancing conditions) does not guarantee that a page will be completely filled, even if more items exist. It can return fewer items than requested on an intermediate page while still providing a Link header pointing to the next page. Relying on the returned item count to terminate pagination causes incomplete data retrieval.

The pagination logic must be updated to strictly rely on the presence of the next relation in the HTTP Link header (or cursor-based pagination metadata for GraphQL) rather than the size of the returned item array.

🎯 Acceptance Criteria

• Pagination must not terminate early if a page returns fewer items than the requested limit, provided a next page link is present in the response headers.
• Pagination must terminate under the following conditions:
  • The total number of retrieved items reaches the user-specified limit (if any).
  • The response headers do not contain a Link header with rel="next".
  • The API returns an empty list of items (safety fallback to prevent infinite loops).
• The fix must apply to both REST API pagination helpers and any GraphQL pagination wrappers if applicable.
• Existing commands that rely on pagination (e.g., listing PRs, issues, or repositories) must continue to function correctly.

🛠️ Technical Specifications & Context

In the EncarnacionP/cli repository (typically built on Go and utilizing GitHub API clients):

• The pagination logic is likely located in the API client wrapper, such as api/client.go, api/queries_repo.go, or a dedicated pagination utility package (e.g., api/paginator.go).
• Look for loops that iterate over pages, specifically checking for conditions resembling:

  if len(results) < limit {
      break
  }

• This logic should be refactored to parse the Link header. For example, using a helper to extract the next page URL:

  // Pseudocode for correct pagination check
  nextURL := findNextPageURL(resp.Header.Get("Link"))
  if nextURL == "" {
      break
  }

• Ensure that if a limit is explicitly requested by the user (e.g., --limit 100), the accumulator stops once that limit is reached, even if more pages are available.

🧪 Verification & Testing

Automated Tests

1. Add a unit test in the API client test suite (e.g., api/client_test.go).
2. Mock the GitHub API responses:
   • Request 1: Page 1 (requested per_page=30).
     • Response 1: Returns 15 items. Header: Link: <https://api.github.com/...page=2>; rel="next".
   • Request 2: Page 2 (requested per_page=30).
     • Response 2: Returns 10 items. Header: No Link header (or no next relation).
3. Assert that the client successfully performs both requests and returns a combined total of 25 items.

Manual Verification

1. Run a command that paginates over a large resource (e.g., gh issue list --limit 50).
2. Simulate or target an endpoint known to return sparse pages, and verify that the CLI retrieves the full requested limit.

---

---

This repo is using Opire - what does it mean? 👇

💵 Everyone can add rewards for this issue commenting /reward 100 (replace 100 with the amount).
🕵️‍♂️ If someone starts working on this issue to earn the rewards, they can comment /try to let everyone know!
🙌 And when they open the PR, they can comment /claim #1 either in the PR description or in a PR's comment.

🪙 Also, everyone can tip any user commenting /tip 20 @EncarnacionP (replace 20 with the amount, and @EncarnacionP with the user to tip).

📖 If you want to learn more, check out our documentation.

[EncarnacionP]

/reward 10

[opirebot]

😅 Ups... you cannot create a reward of $10.00. It needs to be at least $20.00

[testdasislyt]

Hi, this looks like an interesting issue, and I'd like to take a crack at it.

My initial thought is to modify the pagination loop to explicitly look for an X-Next-Page or Link header (or similar mechanism provided by the GitHub API for subsequent pages) to determine if more data exists. If such a mechanism isn't available or consistently reliable across all endpoints, the fallback would be to continue fetching until an empty page is returned, rather than relying solely on items_returned < per_page as a termination condition. This would ensure we fetch all available items even if pages are partially filled.

Is this still available to work on? If so, I'm happy to dive deeper into the current implementation and propose a more detailed plan.

[testdasislyt]

Hi there,

I've read through this issue and understand the problem: the current pagination logic terminates prematurely because it relies solely on the items_returned < requested_page_size heuristic, which is unreliable for APIs like GitHub's that can return partial pages even if more data exists.

My initial thought would be to leverage the Link header for pagination, specifically looking for a rel="next" link. This is the standard and most robust way to determine if more pages are available from many REST APIs, including GitHub. If a rel="next" link is present, we continue fetching; otherwise, we terminate.

If Link headers aren't consistently available or practical to parse within the current client architecture, a simpler fallback could be to continue fetching until an empty page is returned, rather than just a partially filled one. While potentially incurring one extra API call at the very end, it would reliably fetch all data.

Is this issue still available to be picked up? If so, I'd be happy to contribute.

Thanks!

[Saadat555]

/try

I can implement the correct pagination check by parsing the Link header and extracting the next page URL, replacing the unreliable len(results) < limit check. I will make sure the fallback loop condition and limit accumulators are properly adjusted, and I will include unit tests simulating sparse intermediate API responses.

Check if you're the only one trying 👇

There are already people trying to solve this issue. You're welcome to tackle it independently, but we encourage you to reach out to the contributors that are trying to solve the issue if you'd like to collaborate with them.

The other contributor is: @Autonomy-Labs-Tech.