feat: E2E test suite for api mode#346
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
The Autograder required a comprehensive E2E test suite to ensure the stability of the entire submission lifecycle across all grading templates in a live environment. Previous tests were largely unit-based or used mocks, which didn't capture infrastructure-related issues or integration bugs between the API, Sandbox Manager, and the various templates.
Solution
This PR introduces a robust E2E test suite that validates the Autograder in a live Docker environment:
tests/e2ewith 20+ scenarios covering:input_output,web_dev, andstatic_analysis(5+ scenarios each).scripts/run_e2e_tests.shto automate building and starting the Docker environment, running tests, and cleaning up.SandboxManagerto lazy-load the Docker client, preventing API crashes in remote mode when Docker is not locally available.docker-compose.ymlto support authorized requests during E2E runs.Further clarifications
api_testingtemplate scenarios are currently marked asxfaildue to known networking constraints in the sandbox environment that require separate architectural changes.OPENAI_API_KEYis provided in the environment.Related issues
Closes #302
Checklist