Skip to content

feat: E2E test suite for api mode#346

Merged
ArthurCRodrigues merged 3 commits into
mainfrom
feature/e2e-test-suite
May 28, 2026
Merged

feat: E2E test suite for api mode#346
ArthurCRodrigues merged 3 commits into
mainfrom
feature/e2e-test-suite

Conversation

@ArthurCRodrigues

Copy link
Copy Markdown
Member

Context

The Autograder required a comprehensive E2E test suite to ensure the stability of the entire submission lifecycle across all grading templates in a live environment. Previous tests were largely unit-based or used mocks, which didn't capture infrastructure-related issues or integration bugs between the API, Sandbox Manager, and the various templates.

Solution

This PR introduces a robust E2E test suite that validates the Autograder in a live Docker environment:

  1. E2E Test Suite: Implemented a new testing directory tests/e2e with 20+ scenarios covering:
    • Submission Lifecycle: Creation, polling, detail retrieval, and history verification.
    • Template Coverage: Exhaustive testing for input_output, web_dev, and static_analysis (5+ scenarios each).
    • Assignment Management: Full CRUD operations for grading configurations.
    • M2M Ingestion: Verification of external result ingestion (e.g., from GitHub Actions).
  2. Environment Orchestration: Added scripts/run_e2e_tests.sh to automate building and starting the Docker environment, running tests, and cleaning up.
  3. Infrastructure Fixes:
    • Lazy Docker Initialization: Refactored SandboxManager to lazy-load the Docker client, preventing API crashes in remote mode when Docker is not locally available.
    • Integration Auth: Configured default integration token in docker-compose.yml to support authorized requests during E2E runs.
  4. Diagnostics: Improved test output with detailed state printing on failures to facilitate debugging in CI environments.

Further clarifications

  • api_testing template scenarios are currently marked as xfail due to known networking constraints in the sandbox environment that require separate architectural changes.
  • AI-based tests are skipped unless OPENAI_API_KEY is provided in the environment.

Related issues

Closes #302

Checklist

  • I linked the related issue(s) and explained the motivation.
  • I kept this PR focused and scoped to a single concern.
  • I added or updated tests for changed behavior.
  • I ran the relevant tests locally.
  • I updated documentation when needed.

Copilot AI review requested due to automatic review settings May 27, 2026 22:39

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@ArthurCRodrigues ArthurCRodrigues changed the title feat: comprehensive E2E test suite for autograder feat: E2E test suite for api mode May 28, 2026

@ArthurCRodrigues ArthurCRodrigues left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

@ArthurCRodrigues ArthurCRodrigues merged commit 493107a into main May 28, 2026
2 checks passed
@ArthurCRodrigues ArthurCRodrigues deleted the feature/e2e-test-suite branch May 28, 2026 10:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Decouple SetupStep into Specific Steps

2 participants