diff --git a/README.md b/README.md index 648e467e..c5a13433 100644 --- a/README.md +++ b/README.md @@ -356,7 +356,7 @@ jobs: - **Module overview and internals:** [docs/github_action/README.md](docs/github_action/README.md) - **Configuration reference and troubleshooting:** [docs/github_action/configuration.md](docs/github_action/configuration.md) - **Reference implementation:** [webtech-network/demo-autograder](https://github.com/webtech-network/demo-autograder) -- **How to adapt the demo to your course:** [docs/github_action/demo-autograder.md](docs/github_action/demo-autograder.md) +- **How to adapt the demo to your course:** [docs/github_action/quick-start.md](docs/github_action/quick-start.md) --- diff --git a/docs/github_action/README.md b/docs/github_action/README.md index 3721f0ca..f1cd4676 100644 --- a/docs/github_action/README.md +++ b/docs/github_action/README.md @@ -1,52 +1,54 @@ -# GitHub Action module +# GitHub Action — Overview -This module packages the Autograder as a Docker-based GitHub Action so repositories can grade submissions automatically in CI. +The Prisma Autograder ships as a Docker-based GitHub Action that grades student submissions automatically inside CI workflows. It is designed for **GitHub Classroom** but works with any repository that follows the expected directory layout. -It is primarily designed for GitHub Classroom style workflows, but can be used by any repository that follows the expected directory contract. +--- -## Execution modes +## How it works -| Mode | How config is loaded | When to use | -|------|---------------------|-------------| -| `repo` (default) | From config files checked out inside `submission/.github/autograder/` | Repository-based setups where grading config lives in the student repo | -| `external` | Fetched from the Autograder Cloud API at runtime | Centralised setups where a single Autograder Cloud instance manages all assignments | +``` +Student pushes code → GitHub Action triggers → Autograder grades → Results exported +``` -## What it does (repo mode) +The Action: -When a workflow runs in `repo` mode: +1. Reads student files from the `submission/` directory. +2. Loads grading configuration (from repo files **or** from the Autograder Cloud). +3. Builds and executes the full grading pipeline (criteria tree → sandbox → grading → feedback). +4. Exports the result (to GitHub Classroom check run **or** to the Autograder Cloud API). -1. Reads student files from `submission/`. -2. Reads grading config from `submission/.github/autograder/`. -3. Builds and executes the Autograder pipeline. -4. Updates the GitHub check run with the final score. -5. Optionally commits `relatorio.md` with feedback. +--- -## What it does (external mode) +## Execution modes -When a workflow runs in `external` mode: +The Action supports two mutually exclusive modes: -1. Fetches the grading configuration from the Autograder Cloud (`GET /api/v1/configs/id/{id}`). -2. Reads student files from `submission/`. -3. Builds and executes the Autograder pipeline. -4. Posts the grading result back to the Autograder Cloud (`POST /api/v1/submissions/external-results`). -5. If grading fails for any reason, posts a `failed` status payload before exiting non-zero. +| Mode | Config source | Result destination | Use case | +|------|--------------|-------------------|----------| +| **`repo`** (default) | JSON files in `submission/.github/autograder/` | GitHub check run + `relatorio.md` | Config lives in the student repo | +| **`external`** | Autograder Cloud API | Autograder Cloud API | Centralised config managed by the instructor | -## Internal architecture +### Repo mode flow -| Component | Responsibility | -|---|---| -| [`action.yml`](https://github.com/webtech-network/autograder/blob/main/action.yml) | Declares Action inputs/outputs and maps inputs to container env vars. | -| [`Dockerfile.actions`](https://github.com/webtech-network/autograder/blob/main/Dockerfile.actions) | Builds runtime image used by the Action. | -| [`github_action/entrypoint.sh`](https://github.com/webtech-network/autograder/blob/main/github_action/entrypoint.sh) | Validates required env vars and executes Python entrypoint with flags. | -| [`github_action/main.py`](https://github.com/webtech-network/autograder/blob/main/github_action/main.py) | Parses arguments, validates runtime options, reads submission files, and starts grading. | -| [`github_action/github_action_service.py`](https://github.com/webtech-network/autograder/blob/main/github_action/github_action_service.py) | Builds pipeline from JSON configs and handles GitHub export operations. | -| [`github_action/github_classroom_exporter.py`](https://github.com/webtech-network/autograder/blob/main/github_action/github_classroom_exporter.py) | Export adapter that reports score and feedback through `GithubActionService`. | -| [`github_action/cloud_client.py`](https://github.com/webtech-network/autograder/blob/main/github_action/cloud_client.py) | HTTP client for the Autograder Cloud API (config fetch + result submission). Uses exponential back-off on 5xx/network errors; raises `CloudClientError` on 4xx. | -| [`github_action/cloud_exporter.py`](https://github.com/webtech-network/autograder/blob/main/github_action/cloud_exporter.py) | Export adapter for external mode. Builds the `ExternalResultCreate` payload and calls `CloudClient`. Provides `submit_failure()` for the failure path. | +``` +submission/.github/autograder/criteria.json ──┐ +submission/.github/autograder/feedback.json ──┤──► build_pipeline() ──► run ──► GitHub Check Run +submission/.github/autograder/setup.json ──┘ └► relatorio.md +``` + +### External mode flow + +``` +Autograder Cloud ──GET /api/v1/configs/id/{id}──► build_pipeline() ──► run ──► POST /api/v1/submissions/external-results +``` -## Repository contract required by the Action +If grading fails in external mode, a `status: "failed"` payload is submitted before the Action exits non-zero. -Your workflow should checkout repository content into a folder named `submission`: +--- + +## Repository contract + +Your workflow **must** checkout the repository into a directory named `submission`: ```yaml - uses: actions/checkout@v4 @@ -54,27 +56,111 @@ Your workflow should checkout repository content into a folder named `submission path: submission ``` -Inside that checkout, the Action expects: +### Repo mode file requirements + +``` +submission/ +├── .github/ +│ └── autograder/ +│ ├── criteria.json # Required — grading rubric +│ ├── feedback.json # Optional (can be {}) +│ └── setup.json # Optional (can be {}) +├── index.html # Student files (read recursively) +├── styles.css +└── app.js +``` + +### External mode file requirements -- `submission/.github/autograder/criteria.json` (required) -- `submission/.github/autograder/feedback.json` (optional, empty object allowed) -- `submission/.github/autograder/setup.json` (optional, empty object allowed) +Only the student source files are needed. No `.github/autograder/` directory required — config comes from the cloud. -## Important runtime notes +``` +submission/ +├── main.py # Student files (read recursively) +└── utils.py +``` + +!!! note + The Action skips `.git/` and `.github/` directories when collecting student files. -- `template-preset: custom` is currently rejected by the Action entrypoint. -- `feedback-type: ai` requires `openai-key`. -- The current notification logic looks for a check run named `grading`. -- `include-feedback: "true"` enables feedback generation and may update `relatorio.md`. -- In `external` mode, `include_feedback` is taken from the cloud config, **not** from the `include-feedback` action input. -- In `external` mode, `submission-language` is validated against the `languages` list in the cloud config. Omitting it defaults to the first language in that list. +--- -## Reference repository +## Internal architecture + +| Component | Responsibility | +|-----------|---------------| +| `action.yml` | Declares Action inputs/outputs; maps inputs to container environment variables | +| `Dockerfile.actions` | Builds the runtime Docker image (Python 3.10 + Node.js 20) | +| `github_action/entrypoint.sh` | Validates required env vars; executes Python entrypoint with CLI flags | +| `github_action/main.py` | Parses arguments, validates options, reads submission files, starts grading | +| `github_action/github_action_service.py` | Builds pipeline from config; handles GitHub/Cloud export | +| `github_action/cloud_client.py` | HTTP client for Cloud API with retry + exponential backoff | +| `github_action/cloud_exporter.py` | Builds result payload and submits to Cloud API | +| `github_action/github_classroom_exporter.py` | Reports score to GitHub Classroom check run | + +### Docker image + +The Action runs inside a Docker container built from `Dockerfile.actions`: + +- **Base:** `python:3.10-slim` +- **Includes:** Node.js 20 (for JavaScript template grading), `tree`, `curl` +- **Entry:** `/app/github_action/entrypoint.sh` + +--- + +## Supported template presets + +| Preset | Language | Description | +|--------|----------|-------------| +| `webdev` | HTML/CSS/JS | Web development assignments (DOM, CSS, structure) | +| `input_output` | Python, Java, C++ | Standard I/O programs with test cases | +| `api` | Any | REST API endpoint testing | + +!!! warning + `template-preset: custom` is currently **not supported** by the Action and will cause an immediate exit. + +--- + +## Feedback modes + +| Mode | Requires | Description | +|------|----------|-------------| +| `default` | Nothing extra | Rule-based feedback from the grading pipeline | +| `ai` | `openai-key` | AI-generated feedback using OpenAI API | + +When `include-feedback: "true"` (repo mode), the Action commits feedback as `relatorio.md` to the repository. + +--- + +## Quick reference + +Minimal repo-mode workflow: + +```yaml +- uses: webtech-network/autograder@main + with: + template-preset: "webdev" + feedback-type: "default" + include-feedback: "true" +``` + +Minimal external-mode workflow: + +```yaml +- uses: webtech-network/autograder@main + with: + execution-mode: "external" + grading-config-id: "42" + autograder-cloud-url: ${{ secrets.AUTOGRADER_CLOUD_URL }} + autograder-cloud-token: ${{ secrets.AUTOGRADER_CLOUD_TOKEN }} + feedback-type: "default" + template-preset: "input_output" +``` -Use **[`webtech-network/demo-autograder`](https://github.com/webtech-network/demo-autograder)** as the baseline implementation. It demonstrates a working workflow, config files, and expected repository layout. +--- -## Next +## Next steps -- Configuration details: [configuration.md](configuration.md) -- External mode deep-dive: [external-mode.md](external-mode.md) -- Demo walkthrough: [demo-autograder.md](demo-autograder.md) +- [Configuration Reference](configuration.md) — all inputs, outputs, secrets, and permissions +- [Quick Start Guide](quick-start.md) — step-by-step setup for both modes +- [External Mode](external-mode.md) — deep dive into cloud-based grading diff --git a/docs/github_action/configuration.md b/docs/github_action/configuration.md index 1a69a522..9c01995e 100644 --- a/docs/github_action/configuration.md +++ b/docs/github_action/configuration.md @@ -1,166 +1,226 @@ -# GitHub Action configuration reference +# Configuration Reference -This page describes how to configure `webtech-network/autograder@main` in your workflow and what each input affects at runtime. +Complete reference for all inputs, outputs, secrets, and permissions required by `webtech-network/autograder@main`. -## Recommended workflow skeleton +--- -### Repo mode (default) +## Inputs -```yaml -name: Autograder -on: - push: - branches: [main] - pull_request: - branches: [main] - workflow_dispatch: - -jobs: - grading: - permissions: write-all - runs-on: ubuntu-latest - if: github.actor != 'github-classroom[bot]' - steps: - - name: Checkout repository - uses: actions/checkout@v4 - with: - path: submission - - - name: Run Autograder - uses: webtech-network/autograder@main - with: - template-preset: "webdev" - feedback-type: "default" - include-feedback: "true" - openai-key: ${{ secrets.ENGINE }} -``` +All inputs are defined in [`action.yml`](https://github.com/webtech-network/autograder/blob/main/action.yml). -### External mode +### Core inputs -```yaml -name: Autograder (External) -on: - push: - branches: [main] - workflow_dispatch: - -jobs: - grading: - runs-on: ubuntu-latest - if: github.actor != 'github-classroom[bot]' - steps: - - name: Checkout repository - uses: actions/checkout@v4 - with: - path: submission - - - name: Run Autograder - uses: webtech-network/autograder@main - with: - execution-mode: "external" - grading-config-id: "42" # internal DB id - autograder-cloud-url: ${{ secrets.AUTOGRADER_CLOUD_URL }} - autograder-cloud-token: ${{ secrets.AUTOGRADER_CLOUD_TOKEN }} - submission-language: "python" # optional - feedback-type: "default" - template-preset: "input_output" # required by the parser; value is ignored in external mode -``` +| Input | Required | Default | Description | +|-------|:--------:|---------|-------------| +| `template-preset` | **yes** | — | Template key for the grading pipeline. Built-in values: `webdev`, `api`, `input_output`. In external mode the value is ignored (template comes from the cloud config), but the field is still **required by the parser**. | +| `feedback-type` | **yes** | — | Feedback generation strategy: `"default"` (rule-based) or `"ai"` (OpenAI-powered). | +| `github-token` | no | `${{ github.token }}` | GitHub token for API operations (check run updates, repo access). | +| `openai-key` | no | — | OpenAI API key. **Required** when `feedback-type` is `"ai"`. | +| `app-token` | no | — | Separate token for repository access. Falls back to `github-token` if omitted. | +| `include-feedback` | no | `"false"` | `"true"` or `"false"`. When true, generates feedback and commits `relatorio.md`. **Ignored in external mode** (value is taken from the cloud config). | +| `locale` | no | `"en"` | Locale for feedback messages. Supported: `"en"`, `"pt-br"`. | -## Inputs +### External mode inputs -From [`action.yml`](https://github.com/webtech-network/autograder/blob/main/action.yml): - -| Input | Required | Default | Notes | -|---|---:|---|---| -| `github-token` | no | `${{ github.token }}` | Used for GitHub API operations (repo/check run access). | -| `template-preset` | yes | - | Template key used by the grading pipeline. Built-ins include `webdev`, `api`, and `input_output`. In external mode the pipeline reads the template from the cloud config, but this field is still required by the parser. | -| `feedback-type` | yes | - | `default` or `ai`. | -| `custom-template` | no | - | Currently not usable because `template-preset: custom` is rejected by `github_action/main.py`. | -| `openai-key` | no | - | Required only when `feedback-type` is `ai`. | -| `app-token` | no | - | Optional token for repository access; if omitted, runtime falls back to `github-token`. | -| `include-feedback` | no | `"false"` | Must be `"true"` or `"false"` string. Ignored in external mode (value is taken from the cloud config). | -| `execution-mode` | no | `"repo"` | `"repo"` or `"external"`. Determines where grading configuration is loaded from. | -| `grading-config-id` | no | - | Internal DB id of the grading config on the Autograder Cloud. Required when `execution-mode` is `external`. | -| `autograder-cloud-url` | no | - | Base URL of the Autograder Cloud instance (e.g. `https://cloud.example.com`). Required when `execution-mode` is `external`. | -| `autograder-cloud-token` | no | - | Integration token for the Autograder Cloud API. Required when `execution-mode` is `external`. Should be stored as a repository secret. | -| `submission-language` | no | - | Language of the student submission (e.g. `python`, `java`). Validated against the cloud config's `languages` list. Defaults to the first language in the list when omitted. Only used in `external` mode. | +These inputs are required **only** when `execution-mode` is `"external"`: -## Outputs +| Input | Required | Default | Description | +|-------|:--------:|---------|-------------| +| `execution-mode` | no | `"repo"` | `"repo"` or `"external"`. Determines where grading config is loaded from. | +| `grading-config-id` | conditional | — | Internal database ID of the grading config on the Autograder Cloud. Must be an integer. | +| `autograder-cloud-url` | conditional | — | Base URL of the Autograder Cloud instance (e.g. `https://autograder.myschool.edu`). Trailing slashes are stripped automatically. | +| `autograder-cloud-token` | conditional | — | Bearer token for the Cloud API. Must match `AUTOGRADER_INTEGRATION_TOKEN` on the server. | +| `submission-language` | no | — | Language of the student submission (e.g. `"python"`, `"java"`, `"javascript"`). Validated against the cloud config's `languages` list. Defaults to the first language in the list when omitted. | + +### Unsupported inputs + +| Input | Status | Notes | +|-------|--------|-------| +| `custom-template` | ❌ Not usable | `template-preset: custom` is rejected by the entrypoint. Reserved for future use. | -`action.yml` declares: +--- + +## Outputs | Output | Description | -|---|---| -| `result` | Base64-encoded JSON with grading results. | +|--------|-------------| +| `result` | Base64-encoded JSON containing grading results. | + +--- -## Required repository structure +## Configuration files (repo mode) -The Action loads config from the checkout rooted at `submission/`: +When running in repo mode, the Action reads configuration from `submission/.github/autograder/`: -```text -submission/ - .github/ - autograder/ - criteria.json # required - feedback.json # optional (can be {}) - setup.json # optional (can be {}) +### `criteria.json` (required) + +Defines the grading rubric — the hierarchical criteria tree with weighted subjects and tests. + +```json +{ + "test_library": "web_dev", + "base": { + "weight": 100, + "subjects": [ + { + "subject_name": "content_and_style", + "weight": 50, + "subjects": [...] + } + ] + } +} +``` + +Key fields: + +- `test_library` — which template library to use (`web_dev`, `input_output`, `api_testing`) +- `base.subjects` — hierarchical tree of subjects, each with `weight` and either `subjects` (children) or `tests` +- Each test specifies `file`, `name` (test function), and optional `parameters` + +### `feedback.json` (optional) + +Controls feedback generation behavior. Can be an empty object `{}`. + +```json +{ + "general": { + "show_score": true, + "show_passed_tests": false, + "add_report_summary": true + }, + "default": {} +} ``` -It also reads student files recursively from `submission/`, excluding `.git` and `.github` directories. +### `setup.json` (optional) + +Provides setup configuration for the grading environment. Can be an empty object `{}`. -## Secrets and permissions +```json +{} +``` + +--- -### Repo mode secrets +## Secrets -- `ENGINE` (or another secret mapped to `openai-key`) when using AI feedback. -- Optional token secret mapped to `app-token` if you need separate credentials. +### Repo mode -### External mode secrets +| Secret | Maps to input | When needed | +|--------|:-------------:|-------------| +| `ENGINE` (or custom name) | `openai-key` | When using `feedback-type: "ai"` | +| Custom token secret | `app-token` | When separate credentials are needed for repo access | + +### External mode | Secret | Maps to input | Description | -|--------|--------------|-------------| -| `AUTOGRADER_CLOUD_URL` | `autograder-cloud-url` | Base URL of the Autograder Cloud instance | -| `AUTOGRADER_CLOUD_TOKEN` | `autograder-cloud-token` | Integration token (set `AUTOGRADER_INTEGRATION_TOKEN` on the server) | +|--------|:-------------:|-------------| +| `AUTOGRADER_CLOUD_URL` | `autograder-cloud-url` | Base URL of the Autograder Cloud | +| `AUTOGRADER_CLOUD_TOKEN` | `autograder-cloud-token` | Integration token matching server's `AUTOGRADER_INTEGRATION_TOKEN` | + +!!! tip "Generating a secure token" + ```bash + openssl rand -hex 32 + ``` + Set this as `AUTOGRADER_INTEGRATION_TOKEN` on the server and as `AUTOGRADER_CLOUD_TOKEN` in your repository/org secrets. -> Generate a strong token on the server: `openssl rand -hex 32` +--- -### Permissions +## Permissions -`permissions: write-all` is required in **repo mode** because the Action may: +### Repo mode -- update the workflow check run with score summary; -- commit `relatorio.md` when feedback is enabled. +```yaml +permissions: write-all +``` -In **external mode**, no GitHub repository write operations are performed, so elevated permissions are not required. +Required because the Action may: -## Troubleshooting +- Update the workflow check run with the score summary +- Commit `relatorio.md` when feedback is enabled +- The check run export looks for a check run named **`grading`** — keep your job name aligned -### `FileNotFoundError: criteria.json file not found` +### External mode -- Ensure your workflow checks out repository to `path: submission`. -- Ensure config is at `submission/.github/autograder/criteria.json`. -- This error only occurs in **repo mode**; in external mode, config is fetched from the cloud. +No elevated permissions required. The Action only communicates with the Autograder Cloud API and does not write to the repository. -### `grading-config-id`, `autograder-cloud-url`, or `autograder-cloud-token` missing +--- -- All three are required when `execution-mode` is `external`. -- The entrypoint validates these before starting the pipeline and exits non-zero if any are absent. +## Environment variables (internal) -### `OpenAI API key is required for AI feedback mode` +The `action.yml` maps inputs to container environment variables consumed by `entrypoint.sh`: + +| Input | Environment variable | +|-------|---------------------| +| `github-token` | `GITHUB_TOKEN` | +| `template-preset` | `TEMPLATE_PRESET` | +| `feedback-type` | `FEEDBACK_TYPE` | +| `custom-template` | `CUSTOM_TEMPLATE` | +| `openai-key` | `OPENAI_KEY` | +| `app-token` | `APP_TOKEN` | +| `include-feedback` | `INCLUDE_FEEDBACK` | +| `execution-mode` | `EXECUTION_MODE` | +| `grading-config-id` | `GRADING_CONFIG_ID` | +| `autograder-cloud-url` | `AUTOGRADER_CLOUD_URL` | +| `autograder-cloud-token` | `AUTOGRADER_CLOUD_TOKEN` | +| `submission-language` | `SUBMISSION_LANGUAGE` | +| `locale` | `LOCALE` | + +Additionally, the Action uses these GitHub-provided variables: + +| Variable | Usage | +|----------|-------| +| `GITHUB_WORKSPACE` | Root path for file operations | +| `GITHUB_ACTOR` | Student username (passed as `--student-name`) | +| `GITHUB_REPOSITORY` | Repository identifier for API calls | +| `GITHUB_RUN_ID` | Workflow run ID for check run updates | +| `GITHUB_SHA` | Commit SHA included in result metadata | +| `GITHUB_REF` | Branch ref included in result metadata | + +--- -- Provide `openai-key` when `feedback-type: ai`. +## Validation rules + +The entrypoint enforces these rules before starting the pipeline: + +| Condition | Error | +|-----------|-------| +| `feedback-type: "ai"` without `openai-key` | `ValueError: OpenAI API key is required for AI feedback mode` | +| `execution-mode: "external"` without `grading-config-id` | `ValueError: grading-config-id is required` | +| `execution-mode: "external"` without `autograder-cloud-url` | `ValueError: autograder-cloud-url is required` | +| `execution-mode: "external"` without `autograder-cloud-token` | `ValueError: autograder-cloud-token is required` | +| `grading-config-id` not parseable as integer | `ValueError: grading-config-id must be an integer` | +| `include-feedback` not `"true"` or `"false"` | `ValueError: Invalid value for --include-feedback` | +| `template-preset: "custom"` | `SystemExit: Currently, this system does not accept custom templates` | + +--- + +## Troubleshooting + +### `FileNotFoundError: criteria.json file not found` + +- Ensure your workflow checks out to `path: submission` +- Verify the config is at `submission/.github/autograder/criteria.json` +- This error only occurs in repo mode + +### Check run not updated with score + +- The export logic searches for a check run named **`grading`** +- Ensure your job name matches: `jobs: grading:` ### `Invalid value for --include-feedback` -- Use exact strings: `"true"` or `"false"`. +- Use exact strings: `"true"` or `"false"` (not booleans) + +### `OpenAI API key is required for AI feedback mode` -### Check run not updated +- Provide `openai-key` input when `feedback-type: "ai"` -- Current implementation searches for a check run named `grading`. -- Keep your job id/name aligned with that expectation. +--- ## See also -- Module internals: [README.md](README.md) -- External mode deep-dive: [external-mode.md](external-mode.md) -- Demo repository walkthrough: [demo-autograder.md](demo-autograder.md) +- [Overview](README.md) — how the Action works +- [Quick Start Guide](quick-start.md) — complete setup walkthrough +- [External Mode](external-mode.md) — cloud-based grading details diff --git a/docs/github_action/demo-autograder.md b/docs/github_action/demo-autograder.md deleted file mode 100644 index c4f90aaf..00000000 --- a/docs/github_action/demo-autograder.md +++ /dev/null @@ -1,56 +0,0 @@ -# Demo walkthrough: webtech-network/demo-autograder - -The repository **[`webtech-network/demo-autograder`](https://github.com/webtech-network/demo-autograder)** is the best reference for a working GitHub Classroom-style setup using this Action. - -## Why this repo matters - -It shows, in one place: - -- a complete `classroom.yml` workflow; -- grading criteria and feedback config files; -- a realistic `submission/` codebase to grade; -- generated feedback artifact (`relatorio.md`). - -## Key files in the demo - -| File | Purpose | -|---|---| -| `.github/workflows/classroom.yml` | Defines workflow triggers and runs `webtech-network/autograder@main`. | -| `.github/autograder/criteria.json` | Defines rubric structure and test cases. | -| `.github/autograder/feedback.json` | Controls feedback display settings. | -| `.github/autograder/setup.json` | Optional setup file (can be empty). | -| `submission/` | Student files that the rubric evaluates. | - -## How demo layout works with this Action - -In the demo workflow, repository checkout uses: - -```yaml -with: - path: submission -``` - -That makes the config files available exactly where this Action expects them: - -- `submission/.github/autograder/criteria.json` -- `submission/.github/autograder/feedback.json` -- `submission/.github/autograder/setup.json` - -## Adapting the demo for your own course - -1. Start from the demo workflow file and keep the checkout path as `submission`. -2. Replace `.github/autograder/criteria.json` with your assignment rubric. -3. Tune `.github/autograder/feedback.json` for your feedback strategy. -4. Keep the job named `grading` to match current check-run export behavior. -5. Add the `openai-key` secret only if you use `feedback-type: ai`. - -## Common adaptations - -- Change triggers (`push`, `pull_request`, `workflow_dispatch`) to match your class workflow. -- Switch `template-preset` between `webdev`, `api`, or `input_output` depending on assignment type. -- Keep feedback enabled during development (`include-feedback: "true"`) to inspect `relatorio.md`. - -## Related documentation - -- GitHub Action module overview: [README.md](README.md) -- Full input/output and runtime contract: [configuration.md](configuration.md) diff --git a/docs/github_action/external-mode.md b/docs/github_action/external-mode.md index e740e00f..43c9e8a5 100644 --- a/docs/github_action/external-mode.md +++ b/docs/github_action/external-mode.md @@ -1,71 +1,44 @@ -# External mode +# External Mode -External mode lets the GitHub Action load grading configuration from a central **Autograder Cloud** instance instead of reading JSON files from the student repository. Results are posted back to the same instance, making every submission visible through the Cloud's submission endpoints. +External mode lets the GitHub Action load grading configuration from a central **Autograder Cloud** instance instead of reading JSON files from the student repository. Results are posted back to the Cloud, making every submission visible through its API. --- -## Overview +## When to use external mode -In **repo mode** (the default), configuration travels with the student repository: - -``` -student repo → criteria.json / feedback.json / setup.json → pipeline → check run + relatorio.md -``` - -In **external mode**, configuration is owned by the instructor and hosted on the Cloud: - -``` -Autograder Cloud → grading config → pipeline → Autograder Cloud (result) -``` - -The student repository only needs to contain the source code under `submission/`. No config files are required. +- You manage a multi-repository classroom and want a **single place** to configure grading criteria. +- You want grading results stored centrally rather than scattered across individual check runs. +- You need to change grading criteria without pushing commits to every student repo. --- -## Call sequence +## How it works ``` GitHub Action │ - ├─── GET /api/v1/configs/id/{grading_config_id} ─────► Autograder Cloud - │ (auth: AUTOGRADER_CLOUD_TOKEN) │ - │◄────────────────────────────────────────────────────── config JSON + ├── GET /api/v1/configs/id/{grading_config_id} ──────► Autograder Cloud + │ (auth: Bearer AUTOGRADER_CLOUD_TOKEN) │ + │◄──────────────────────────────────────────────────── config JSON │ - ├─── build_pipeline(criteria_config, template, …) + ├── build_pipeline(criteria_config, template, …) │ - ├─── run_autograder(pipeline, student_name, files) - │ (grading runs locally inside the Action container) + ├── run_autograder(pipeline, student_name, files) + │ (grading runs locally inside the Action container) │ - └─── POST /api/v1/submissions/external-results ─────► Autograder Cloud - (auth: AUTOGRADER_CLOUD_TOKEN) + └── POST /api/v1/submissions/external-results ──────► Autograder Cloud + (auth: Bearer AUTOGRADER_CLOUD_TOKEN) ``` -### Failure path +### Failure handling -If grading raises an exception after the config is fetched, a failure payload (`status: "failed"`, `final_score: 0`) is submitted to the Cloud **before** the Action exits non-zero. This guarantees the Cloud always records a terminal state for every run. +If grading raises an exception **after** the config is fetched, a failure payload is submitted before the Action exits: ``` -run_autograder() raises - │ - ├─── POST /api/v1/submissions/external-results (status: "failed") - │ - └─── SystemExit(1) +run_autograder() raises ──► POST external-results (status: "failed", score: 0) ──► exit 1 ``` -If the config fetch itself fails (network error, 404, invalid token), no result is posted and the Action exits non-zero immediately. - ---- - -## Required secrets - -Configure these as **repository secrets** (or organization secrets shared across classroom repos): - -| Secret | Description | -|--------|-------------| -| `AUTOGRADER_CLOUD_URL` | Base URL of your Autograder Cloud instance, e.g. `https://autograder.myschool.edu` | -| `AUTOGRADER_CLOUD_TOKEN` | Integration token (`AUTOGRADER_INTEGRATION_TOKEN` value set on the server) | - -> Generate a strong token: `openssl rand -hex 32` +If the config fetch itself fails (network, 404, bad token), no result is posted and the Action exits non-zero immediately. --- @@ -97,24 +70,33 @@ jobs: autograder-cloud-token: ${{ secrets.AUTOGRADER_CLOUD_TOKEN }} submission-language: "python" feedback-type: "default" - template-preset: "input_output" # required by the parser; value unused in external mode + template-preset: "input_output" + locale: "pt-br" ``` -### `submission-language` - -- Optional. When omitted, the first language in the cloud config's `languages` list is used. -- When provided, it is validated against the `languages` list. An unsupported value causes the Action to exit non-zero **without posting a result**. +!!! note + `template-preset` is required by the argument parser but its value is **ignored** in external mode — the template comes from the cloud config. --- -## Step-by-step setup +## Required inputs + +| Input | Description | +|-------|-------------| +| `execution-mode` | Must be `"external"` | +| `grading-config-id` | Integer ID of the grading config on the Cloud | +| `autograder-cloud-url` | Base URL of the Autograder Cloud (e.g. `https://autograder.myschool.edu`) | +| `autograder-cloud-token` | Bearer token matching the server's `AUTOGRADER_INTEGRATION_TOKEN` | +| `template-preset` | Any valid value (required by parser, not used) | +| `feedback-type` | `"default"` or `"ai"` | + +### Optional inputs -1. **Deploy the Autograder Cloud** and confirm the API is reachable. -2. **Generate an integration token** and set `AUTOGRADER_INTEGRATION_TOKEN` on the server. -3. **Create a grading configuration** via `POST /api/v1/configs` and note the returned `id`. -4. **Add secrets** `AUTOGRADER_CLOUD_URL` and `AUTOGRADER_CLOUD_TOKEN` to the repository (or organisation). -5. **Add the workflow file** above to `.github/workflows/autograder.yml` in each student repository. -6. **Push a submission** and verify a result appears under `GET /api/v1/submissions/config/{id}`. +| Input | Default | Description | +|-------|---------|-------------| +| `submission-language` | First language in config | Validated against the cloud config's `languages` list | +| `locale` | `"en"` | Locale for feedback messages (`"en"`, `"pt-br"`) | +| `openai-key` | — | Required only when `feedback-type: "ai"` | --- @@ -123,35 +105,230 @@ jobs: | Behaviour | Repo mode | External mode | |-----------|-----------|---------------| | Config source | `submission/.github/autograder/*.json` | Autograder Cloud API | -| `include_feedback` | From `include-feedback` action input | From cloud config | +| `include_feedback` | From `include-feedback` action input | From cloud config field | | Result export | GitHub check run + `relatorio.md` | Autograder Cloud API | -| Repository write permissions needed | Yes (`write-all`) | No | -| Failure reporting | GitHub check run status | `POST /api/v1/submissions/external-results` with `status: "failed"` | +| Repository write permissions | Required (`write-all`) | Not required | +| Failure reporting | Check run status | `POST external-results` with `status: "failed"` | +| Student repo needs config files | Yes | No | --- -## Troubleshooting +## Cloud config structure + +When fetched from `GET /api/v1/configs/id/{id}`, the config contains: + +```json +{ + "id": 42, + "template_name": "input_output", + "languages": ["python", "java"], + "include_feedback": true, + "criteria_config": { "...": "criteria tree JSON" }, + "feedback_config": { "...": "feedback settings" }, + "setup_config": { "...": "setup options" } +} +``` + +| Field | Corresponds to (repo mode) | +|-------|---------------------------| +| `criteria_config` | `criteria.json` | +| `feedback_config` | `feedback.json` | +| `setup_config` | `setup.json` | +| `template_name` | `template-preset` action input | +| `include_feedback` | `include-feedback` action input | +| `languages` | Available submission languages | + +--- + +## Result payload + +The Action submits this payload to `POST /api/v1/submissions/external-results`: + +```json +{ + "grading_config_id": 42, + "external_user_id": "student-github-username", + "username": "student-github-username", + "language": "python", + "status": "completed", + "final_score": 85.5, + "feedback": "## Grading Report\n...", + "result_tree": { "...": "full result tree" }, + "focus": { "...": "focus analysis" }, + "execution_time_ms": 3421, + "error_message": null, + "submission_metadata": { + "repository": "org/student-repo", + "commit_sha": "abc123...", + "run_id": "12345678", + "actor": "student-username", + "ref": "refs/heads/main" + } +} +``` + +On failure, `status` is `"failed"`, `final_score` is `0.0`, and `error_message` contains the exception description. + +--- + +## Setup guide + +### Step 1: Deploy the Autograder Cloud + +Ensure the API is running and reachable from GitHub Actions runners: + +```bash +curl https://your-cloud-url/api/v1/health +``` + +### Step 2: Generate an integration token + +```bash +openssl rand -hex 32 +``` + +Set this as `AUTOGRADER_INTEGRATION_TOKEN` on the server before starting the API. + +### Step 3: Create a grading configuration + +Use `POST /api/v1/configs` to create a config. Note the returned `id`. -### `CloudClientError: 401` -The `AUTOGRADER_CLOUD_TOKEN` does not match the server's `AUTOGRADER_INTEGRATION_TOKEN`. Regenerate and update both. +```bash +curl -X POST https://your-cloud-url/api/v1/configs \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "template_name": "input_output", + "languages": ["python"], + "include_feedback": true, + "criteria_config": { "...": "your criteria" }, + "feedback_config": {}, + "setup_config": {} + }' +``` + +### Step 4: Add repository secrets + +Add to the repository (or organisation level for all classroom repos): + +| Secret | Value | +|--------|-------| +| `AUTOGRADER_CLOUD_URL` | `https://your-cloud-url` | +| `AUTOGRADER_CLOUD_TOKEN` | The token from Step 2 | + +### Step 5: Add the workflow + +Create `.github/workflows/autograder.yml` in each student repository with the workflow example above. + +### Step 6: Verify + +Push a submission and check: + +```bash +curl -H "Authorization: Bearer $TOKEN" \ + https://your-cloud-url/api/v1/submissions/config/42 +``` + +--- + +## Migrating from repo mode + +Existing repo-mode workflows **continue to work without changes**. External mode is opt-in. + +### Migration strategy + +1. Leave existing repo-mode workflows untouched. +2. For new assignments, use external mode with a Cloud config. +3. For existing assignments, migrate one at a time: + - Create the equivalent Cloud config + - Update the workflow to external mode + - Remove `.github/autograder/` from the student repo template + +### Config field mapping + +| Repo-mode source | Cloud config field | +|------------------|--------------------| +| `criteria.json` | `criteria_config` | +| `feedback.json` | `feedback_config` | +| `setup.json` | `setup_config` | +| `template-preset` action input | `template_name` | +| `include-feedback` action input | `include_feedback` | + +--- + +## Operational checklist + +### Server side + +- [ ] `AUTOGRADER_INTEGRATION_TOKEN` is set and non-empty +- [ ] API is reachable from GitHub Actions runners (`/api/v1/health` returns 200) +- [ ] Grading config created via `POST /api/v1/configs` — note the `id` +- [ ] Config includes at least one entry in `languages` + +### Workflow side + +- [ ] `AUTOGRADER_CLOUD_URL` secret set on repository/org +- [ ] `AUTOGRADER_CLOUD_TOKEN` secret set on repository/org +- [ ] `execution-mode: "external"` in the workflow +- [ ] `grading-config-id` matches the server config `id` +- [ ] `template-preset` set to any valid value (parser requirement) +- [ ] `permissions: write-all` removed (not needed) + +--- + +## Smoke test + +### 1. Verify Cloud connectivity + +```bash +# Health check +curl https://your-cloud-url/api/v1/health + +# Verify config exists and token works +curl -H "Authorization: Bearer $TOKEN" \ + https://your-cloud-url/api/v1/configs/id/42 +``` + +### 2. Trigger a test run + +Push a known-good submission. Verify: + +- Action completes successfully +- Result appears: `GET /api/v1/submissions/config/42` + +### 3. Verify failure recording + +Push a broken submission. Verify: + +- Action exits with code 1 +- A `status: "failed"` record exists on the Cloud + +--- + +## Troubleshooting -### `CloudClientError: 404` -The `grading-config-id` does not exist on the server. Verify the id with `GET /api/v1/configs`. +| Symptom | Cause | Fix | +|---------|-------|-----| +| `CloudClientError: HTTP 401` | Wrong token | Regenerate and sync `AUTOGRADER_INTEGRATION_TOKEN` ↔ `AUTOGRADER_CLOUD_TOKEN` | +| `CloudClientError: HTTP 404` | Wrong config ID | Verify with `GET /api/v1/configs` | +| `ValueError: Language 'X' not supported` | Language not in config's `languages` list | Add to cloud config or fix `submission-language` | +| `ValueError: Config has no languages defined` | Empty `languages` in cloud config | Add at least one language | +| Action exits 1 but no result on Cloud | Config fetch failed | Check logs for network/auth error | +| Server returns 503 | `AUTOGRADER_INTEGRATION_TOKEN` not set | Set env var and restart API | +| `CloudConnectionError` after retries | Server unreachable or persistent 5xx | Check server health and network | -### `ValueError: Language 'X' not supported` -The `submission-language` value is not in the config's `languages` list. Update the cloud config or correct the workflow input. +### Retry behaviour -### `ValueError: Config has no languages defined` -The cloud config's `languages` list is empty. Update the config to include at least one language. +The Cloud client uses exponential backoff for transient errors: -### Action exits non-zero but no result on Cloud -Config fetch failed before any result could be posted. Check the Action log for the specific error (network, auth, or missing arguments). +- **4xx responses** → immediate failure (no retry) +- **5xx / network errors** → up to 3 retries with delays of 1s, 2s, 4s +- Timeouts: 10s connect, 30s read --- ## See also -- [configuration.md](configuration.md) — full inputs reference -- [README.md](README.md) — module architecture -- [API documentation](../API.md) — Cloud API endpoints -- [Rollout guide](../guides/external-mode-rollout.md) — migration and operational checklist +- [Configuration Reference](configuration.md) — all inputs and validation rules +- [Overview](README.md) — general architecture +- [API Documentation](../API.md) — Cloud API endpoints diff --git a/docs/github_action/quick-start.md b/docs/github_action/quick-start.md new file mode 100644 index 00000000..478f6c49 --- /dev/null +++ b/docs/github_action/quick-start.md @@ -0,0 +1,344 @@ +# Quick Start Guide + +This guide walks you through setting up the Prisma Autograder GitHub Action from scratch in both execution modes. + +--- + +## Repo mode — step by step + +### 1. Create the grading config directory + +In your assignment repository template, create: + +``` +.github/ +└── autograder/ + ├── criteria.json + ├── feedback.json + └── setup.json +``` + +### 2. Define your criteria + +Create `.github/autograder/criteria.json` with your grading rubric: + +```json +{ + "test_library": "web_dev", + "base": { + "weight": 100, + "subjects": [ + { + "subject_name": "html_structure", + "weight": 60, + "tests": [ + { + "file": "submission/index.html", + "name": "has_tag", + "parameters": [ + { "name": "tag", "value": "header" }, + { "name": "required_count", "value": 1 } + ] + } + ] + }, + { + "subject_name": "css_styling", + "weight": 40, + "tests": [ + { + "file": "submission/styles.css", + "name": "check_flexbox_usage" + } + ] + } + ] + } +} +``` + +### 3. Configure feedback (optional) + +Create `.github/autograder/feedback.json`: + +```json +{ + "general": { + "show_score": true, + "show_passed_tests": false, + "add_report_summary": true + }, + "default": {} +} +``` + +### 4. Add the workflow + +Create `.github/workflows/classroom.yml`: + +```yaml +name: Autograder +on: + push: + branches: [main] + pull_request: + branches: [main] + workflow_dispatch: + +jobs: + grading: + permissions: write-all + runs-on: ubuntu-latest + if: github.actor != 'github-classroom[bot]' + steps: + - name: Checkout repository + uses: actions/checkout@v4 + with: + path: submission + + - name: Run Autograder + uses: webtech-network/autograder@main + with: + template-preset: "webdev" + feedback-type: "default" + include-feedback: "true" +``` + +!!! important + The `path: submission` in the checkout step is **required**. The Action expects all files under `submission/`. + +### 5. Add secrets (if using AI feedback) + +If you want AI-powered feedback: + +```yaml + - name: Run Autograder + uses: webtech-network/autograder@main + with: + template-preset: "webdev" + feedback-type: "ai" + include-feedback: "true" + openai-key: ${{ secrets.ENGINE }} +``` + +Add your OpenAI API key as the repository secret `ENGINE`. + +### 6. Verify + +Push a commit and check: + +- ✅ The workflow runs successfully +- ✅ The check run shows "Autograding Result" with the score +- ✅ `relatorio.md` appears in the repo (if feedback enabled) + +--- + +## External mode — step by step + +### 1. Set up the Autograder Cloud + +Ensure your Cloud instance is running and accessible. Verify: + +```bash +curl https://your-cloud-url/api/v1/health +``` + +### 2. Create a grading configuration + +```bash +curl -X POST https://your-cloud-url/api/v1/configs \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "template_name": "input_output", + "languages": ["python"], + "include_feedback": true, + "criteria_config": { + "test_library": "input_output", + "base": { + "weight": 100, + "subjects": [...] + } + }, + "feedback_config": { "general": { "show_score": true } }, + "setup_config": {} + }' +``` + +Note the returned `id` (e.g. `42`). + +### 3. Add repository secrets + +| Secret | Value | +|--------|-------| +| `AUTOGRADER_CLOUD_URL` | `https://your-cloud-url` | +| `AUTOGRADER_CLOUD_TOKEN` | Your integration token | + +### 4. Add the workflow + +Create `.github/workflows/autograder.yml`: + +```yaml +name: Autograder (External) +on: + push: + branches: [main] + workflow_dispatch: + +jobs: + grading: + runs-on: ubuntu-latest + if: github.actor != 'github-classroom[bot]' + steps: + - name: Checkout repository + uses: actions/checkout@v4 + with: + path: submission + + - name: Run Autograder + uses: webtech-network/autograder@main + with: + execution-mode: "external" + grading-config-id: "42" + autograder-cloud-url: ${{ secrets.AUTOGRADER_CLOUD_URL }} + autograder-cloud-token: ${{ secrets.AUTOGRADER_CLOUD_TOKEN }} + submission-language: "python" + feedback-type: "default" + template-preset: "input_output" + locale: "pt-br" +``` + +### 5. Verify + +Push a submission and check: + +- ✅ The workflow completes without errors +- ✅ A result appears on `GET /api/v1/submissions/config/42` + +--- + +## Demo repository + +The **[`webtech-network/demo-autograder`](https://github.com/webtech-network/demo-autograder)** repository is a complete working example of repo-mode grading. + +### What it demonstrates + +- A fully configured `classroom.yml` workflow +- Real `criteria.json` for a web development assignment +- Feedback configuration +- Student submission files that get graded +- Generated `relatorio.md` feedback artifact + +### Demo repository structure + +``` +demo-autograder/ +├── .github/ +│ ├── workflows/ +│ │ └── classroom.yml # Workflow definition +│ └── autograder/ +│ ├── criteria.json # Web dev grading rubric +│ ├── feedback.json # Feedback display settings +│ └── setup.json # Setup configuration +├── submission/ +│ ├── index.html # Student HTML +│ ├── styles.css # Student CSS +│ └── app.js # Student JavaScript +└── relatorio.md # Generated feedback report +``` + +### Demo workflow + +```yaml +name: Autograder +on: + push: + branches: [main] + pull_request: + branches: [main] + workflow_dispatch: + +jobs: + grading: + permissions: write-all + runs-on: ubuntu-latest + if: github.actor != 'github-classroom[bot]' + steps: + - name: Checkout repository + uses: actions/checkout@v4 + with: + path: submission + + - name: Run Autograder + uses: webtech-network/autograder@main + with: + template-preset: "webdev" + feedback-type: "default" + include-feedback: "true" + openai-key: ${{ secrets.ENGINE }} +``` + +### Adapting for your course + +1. Fork/copy the demo structure into your assignment template +2. Replace `criteria.json` with your rubric +3. Adjust `template-preset` to match your assignment type: + - `"webdev"` — HTML/CSS/JS assignments + - `"input_output"` — stdin/stdout programs (Python, Java, C++) + - `"api"` — REST API testing +4. Keep the job named `grading` (required for check run export) +5. Adjust triggers to match your workflow (push, PR, manual dispatch) + +--- + +## Common patterns + +### GitHub Classroom integration + +For GitHub Classroom, the autograder integrates with the classroom bot: + +```yaml +if: github.actor != 'github-classroom[bot]' +``` + +This prevents the Action from running on the initial bot commit when creating student repos. + +### Multiple assignment types + +You can have different workflows for different assignments in the same org by varying `template-preset` and `criteria.json`: + +=== "Web Dev assignment" + + ```yaml + template-preset: "webdev" + ``` + +=== "Python I/O assignment" + + ```yaml + template-preset: "input_output" + ``` + +=== "API assignment" + + ```yaml + template-preset: "api" + ``` + +### Localization + +Set the `locale` input to control feedback language: + +```yaml +locale: "pt-br" # Portuguese (Brazil) +locale: "en" # English (default) +``` + +--- + +## See also + +- [Configuration Reference](configuration.md) — all inputs and outputs +- [External Mode](external-mode.md) — cloud-based grading deep dive +- [Overview](README.md) — architecture and concepts +- [Criteria Configuration Examples](../guides/criteria_configuration_examples.md) — writing rubrics diff --git a/docs/guides/github_module.md b/docs/guides/github_module.md index 61f76030..4ceb4510 100644 --- a/docs/guides/github_module.md +++ b/docs/guides/github_module.md @@ -6,6 +6,6 @@ The GitHub Action documentation now lives in a dedicated section: - [GitHub Action module overview](../github_action/README.md) - [GitHub Action configuration reference](../github_action/configuration.md) -- [Demo walkthrough (`webtech-network/demo-autograder`)](../github_action/demo-autograder.md) +- [Demo walkthrough (`webtech-network/demo-autograder`)](../github_action/quick-start.md) For day-to-day setup, start with the new section above. diff --git a/docs/index.md b/docs/index.md index b5884204..872382ff 100644 --- a/docs/index.md +++ b/docs/index.md @@ -36,7 +36,7 @@ This documentation is organized to be both practical for first-time users and de - [GitHub Action module overview](github_action/README.md) - [GitHub Action configuration reference](github_action/configuration.md) -- [Demo repository walkthrough](github_action/demo-autograder.md) +- [Demo repository walkthrough](github_action/quick-start.md) ## Quick Access diff --git a/mkdocs.yml b/mkdocs.yml index 17ff94fc..f5183be5 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -52,8 +52,9 @@ nav: - REST API Reference: API.md - GitHub Action: - Overview: github_action/README.md - - Configuration: github_action/configuration.md - - Demo Repository Walkthrough: github_action/demo-autograder.md + - Quick Start Guide: github_action/quick-start.md + - Configuration Reference: github_action/configuration.md + - External Mode: github_action/external-mode.md - Pipeline Deep Dive: - Overview: pipeline/README.md - 01 Load Template: pipeline/01-load-template.md