Skip to content

fix: rework Auth0 API calls to allow large numbers of users#190

Merged
marius-mather merged 18 commits intomainfrom
fix/sync-rate-limits
Mar 4, 2026
Merged

fix: rework Auth0 API calls to allow large numbers of users#190
marius-mather merged 18 commits intomainfrom
fix/sync-rate-limits

Conversation

@marius-mather
Copy link
Collaborator

@marius-mather marius-mather commented Mar 2, 2026

Description

AAI-743: need to allow fetching more than 1000 users at a time to properly sync users into the backend from Auth0

Changes

  • Update get_all_role_users to use checkpoint-based iteration, page-based only allows 1000 users
  • Update sync_auth0_users to use a user export, since otherwise we can't get more than 1000 users
  • Update tests

Checklist

  • I have commented my code, particularly in hard-to-understand areas
  • I have added unit / integration tests that prove my fix is effective or that my feature works
  • I have run all tests locally and they pass
  • I have updated the documentation (if applicable)
  • For any new secrets, I have updated the shared spreadsheet and the GitHub Secrets.

How to Test Manually (if necessary)

Run uv run pytest

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates Auth0 integration to support syncing more than 1000 users by switching role-user fetching to checkpoint-based iteration and switching full user sync to Auth0’s user export job flow.

Changes:

  • Implement checkpoint-based iteration for fetching all users assigned to a role.
  • Add Auth0 user export job creation/polling + CSV download/parsing, and update sync_auth0_users to use it.
  • Update/extend tests and introduce Faker as a dev dependency for data generation.

Reviewed changes

Copilot reviewed 6 out of 7 changed files in this pull request and generated 8 comments.

Show a summary per file
File Description
auth0/client.py Adds export job APIs + download logic; updates role-user retrieval to support checkpoints.
scheduled_tasks/tasks.py Introduces ExportedUser, CSV parsing, and switches sync_auth0_users to export-based sync.
tests/test_auth0_client.py Updates role-user pagination tests and adds a checkpoint-iteration test.
tests/scheduled_tasks/test_tasks.py Updates sync test to mock exported-user flow instead of paged API calls.
tests/datagen.py Adds ExportedUserFactory and introduces Faker usage.
pyproject.toml Adds faker to dev dependencies.
uv.lock Lockfile update for new dev dependency.

@marius-mather marius-mather requested a review from uwwint March 4, 2026 03:53
@marius-mather marius-mather merged commit d3e4632 into main Mar 4, 2026
5 checks passed
@marius-mather marius-mather deleted the fix/sync-rate-limits branch March 4, 2026 04:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants