Problem
Two scalability issues in pkg/datastore/github.go.
1. O(pending runs × queued jobs) with JSON parsing in the inner loop
In the pending-run detection (pkg/datastore/github.go:58-78), for every pending workflow run the code iterates all queued jobs and calls github.ParseWebHook (full JSON parse of the stored webhook payload) per pair. With N runs and M jobs that is N×M parses per tick.
2. One goroutine per repository, no concurrency limit
pkg/datastore/github.go:97-130 spawns a goroutine per recently-active repository to call the GitHub API, with no semaphore or errgroup.SetLimit. On installations with many repositories this bursts requests and eats the rate limit budget that the starter/runner paths also depend on.
Suggested fix
- Parse each queued job's payload once, collect run IDs into a
map[int64]struct{}, then check pending runs against the set (O(N+M)).
- Use
errgroup.Group with SetLimit(n) (or a semaphore.Weighted) for the per-repository fan-out.
Problem
Two scalability issues in
pkg/datastore/github.go.1. O(pending runs × queued jobs) with JSON parsing in the inner loop
In the pending-run detection (
pkg/datastore/github.go:58-78), for every pending workflow run the code iterates all queued jobs and callsgithub.ParseWebHook(full JSON parse of the stored webhook payload) per pair. With N runs and M jobs that is N×M parses per tick.2. One goroutine per repository, no concurrency limit
pkg/datastore/github.go:97-130spawns a goroutine per recently-active repository to call the GitHub API, with no semaphore orerrgroup.SetLimit. On installations with many repositories this bursts requests and eats the rate limit budget that the starter/runner paths also depend on.Suggested fix
map[int64]struct{}, then check pending runs against the set (O(N+M)).errgroup.GroupwithSetLimit(n)(or asemaphore.Weighted) for the per-repository fan-out.