Skip to content

feat(gh push): fixed-batch processing and --id single-item push#547

Merged
SorraTheOrc merged 2 commits intomainfrom
copilot/enable-batching-in-wl-gh-push
Mar 10, 2026
Merged

feat(gh push): fixed-batch processing and --id single-item push#547
SorraTheOrc merged 2 commits intomainfrom
copilot/enable-batching-in-wl-gh-push

Conversation

Copy link
Contributor

Copilot AI commented Mar 10, 2026

wl gh push pushed all candidates in one pass — unreliable on large worklogs with no partial-progress persistence on failure.

Changes

Batching (src/commands/github.ts)

  • Splits itemsToProcess into fixed windows of 10; calls upsertIssuesFromWorkItems per batch
  • Persists db.upsertItems(updatedItems) after each successful batch — partial progress is durable
  • Stops immediately on first batch error; wraps error with Batch N/M failed: … for diagnostics
  • Accumulates GithubSyncResult and timing across batches; all existing output/log paths unchanged
  • Pre-builds commentsByItemId map once so each batch receives only its relevant comments

--id flag

  • New --id <work-item-id> option filters candidates to a single item (batch of 1) after pre-filtering
  • Throws immediately if the specified ID is not a push candidate
  • Respects --no-update-timestamp and all other existing flags
# Push a single item by ID
wl gh push --repo owner/repo --id WL-ABC123

# Full push still works as before, now in batches of 10
wl gh push --repo owner/repo --all

Tests (tests/cli/github-push-batching.test.ts)

  • --id happy path, not-found error, timestamp write/skip
  • Multi-batch path (15 items), exact-batch-size boundary (10 items), zero-item edge case

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • https://api.github.com/graphql
    • Triggering command: /usr/bin/gh gh issue create --repo owner/name --title Item 1 --body-file - (http block)
    • Triggering command: /usr/bin/gh gh issue create --repo owner/name --title Item 2 --body-file - (http block)
    • Triggering command: /usr/bin/gh gh issue create --repo owner/name --title Item without GitHub mapping --body-file - (http block)

If you need me to access, download, or install something from one of these locations, you can either:

Original prompt

This section details on the original issue you should resolve

<issue_title>Enable batching in wl gh push</issue_title>
<issue_description>

Headline

Enable batching in wl gh push to process work items in fixed batches of 10, persist progress after each batch, and support pushing a single item by id.

Problem statement

wl gh push currently attempts to push all candidate work items in one pass. On large worklogs this is unreliable and slow; failures mid-run can leave the process partially complete with no persisted progress. We need a deterministic, conservative batching approach that reduces per-run load and ensures partial progress is persisted safely.

Users

  • Primary: developers and CLI users who run wl gh push against repositories with many work items.
  • Secondary: automated agents/workflows that call wl gh push as part of larger automation.

Example user stories

  • As a developer with many work items, I want wl gh push to complete in predictable batches so a single failure doesn't require reprocessing everything.
  • As an agent, I want partial progress persisted so retries resume from the last successful batch.
  • As an operator, I want a simple CLI form to push a single work item when required: wl gh push --id <WL-...>.

Success criteria

  • The push operation processes candidate items in fixed batches of 10 and attempts to upsert each batch to GitHub.
  • After each successfully completed batch the updated item mappings are written back to the DB (via db.upsertItems) so progress is persisted.
  • On any API/network error the command stops immediately, returns a non-zero exit code, and reports the failing batch and error; no further batches are attempted (stop-on-first-error behaviour).
  • Support wl gh push --id <work-item-id> to push a single specified item (behaves like a batch of 1) and persist its mapping on success.
  • Existing behaviors remain unchanged unless explicitly noted: --all, --no-update-timestamp, pre-filtering, and comment upserts must continue to work as before.

Constraints

  • Batch size is fixed at 10 (non-configurable for this change per user decision).
  • Command must stop on first batch error (no retries as part of this change).
  • Updated item mappings must be written after each batch (to avoid large atomic commits and to make partial progress durable).
  • Keep compatibility with existing flags (--all, --no-update-timestamp) and JSON output shape.
  • Do not change label/issue payload logic; only the iteration & persistence strategy is in scope.

Existing state

  • src/commands/github.ts contains the gh push command that currently builds the full candidate list and calls upsertIssuesFromWorkItems(itemsToProcess, ...) which performs upserts and returns updatedItems.
  • src/github-sync.ts is the heavy-lifter that upserts issues/comments and currently expects to receive the full candidate set; it already implements per-item skip logic and concurrency control.
  • Pre-filtering (last-push timestamp) exists in src/github-pre-filter.ts and is used to reduce itemsToProcess before calling the sync logic.
  • Related intake and bug items exist in the worklog: e.g. WL-0MLX34EAV1DGI7QD (sub-issue self-link), WL-0MM8Q1MQU02G8820 (delegation/new-issue duplication), and the skip-unchanged last-push draft.

Desired change

  • Implement batching around the existing upsert flow. Two viable approaches:
    1. Batch in src/commands/github.ts: split itemsToProcess and commentsToProcess into 10-item windows, call upsertIssuesFromWorkItems per batch, persist returned updatedItems after each batch, stop on error.
    2. Batch inside src/github-sync.ts: accept an additional batchSize parameter and process incoming items in internal batches, persisting mappings via a provided callback after each internal batch. (Conservative preference: implement batching in commands/github.ts to minimize changes to sync core.)
  • Add CLI --id <work-item-id> flag to gh push to allow pushing a single work item; when provided, build a single-item batch and run the same flow (persist mappings and update last-push timestamp behavior should match other runs).
  • Ensure --no-update-timestamp continues to skip writing last-push timestamp; when pushing a single id the command should still honor --no-update-timestamp.

Related work

  • Files: src/commands/github.ts (push entrypoint), src/github-sync.ts (upsert logic), src/github-pre-filter.ts (last-push pre-filter), DATA_SYNCING.md (process notes).
  • Work items (single-line summaries):
    • WL-0MLX34EAV1DGI7QD — wl gh push: sub-issue self-link error; guards and data-corruption fixes in hierarchy linking (relevant to hierarchy/link phases).
    • WL-0MLX34RED18U7KB7 — Clear corrupted githubIssueNumber data for 33 items sharing issue 675 (data hygiene related to pushes).
    • WL-0MM8Q1MQU02G8820 — Github delegation should not create duplicate issues (related to single-item pushes an...

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

- Add --id <work-item-id> flag to push a single work item by ID
- Implement fixed 10-item batching in src/commands/github.ts:
  - Splits itemsToProcess into 10-item windows
  - Calls upsertIssuesFromWorkItems per batch, persists updatedItems after each
  - Stops immediately on first batch error with descriptive message
  - Accumulates results across all batches for unified output
- Add GithubSyncResult to imports from github-sync.js
- Add tests for --id flag and batching behavior

Co-authored-by: SorraTheOrc <250240+SorraTheOrc@users.noreply.github.com>
Copilot AI changed the title [WIP] Enable batching in wl gh push for work items feat(gh push): fixed-batch processing and --id single-item push Mar 10, 2026
@SorraTheOrc SorraTheOrc marked this pull request as ready for review March 10, 2026 23:30
@SorraTheOrc SorraTheOrc merged commit 760a6b0 into main Mar 10, 2026
4 checks passed
@SorraTheOrc SorraTheOrc deleted the copilot/enable-batching-in-wl-gh-push branch March 10, 2026 23:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Enable batching in wl gh push

2 participants