diff --git a/.github/workflows/robot-tests.yml b/.github/workflows/robot-tests.yml index 9b38406..69abc2f 100644 --- a/.github/workflows/robot-tests.yml +++ b/.github/workflows/robot-tests.yml @@ -5,7 +5,22 @@ on: branches: [ main ] push: branches: [ main ] - workflow_dispatch: {} # allows manual trigger from the Actions tab + # Nightly full regression run — DISABLED for now. + # To turn it back on: uncomment the two lines below and merge this file + # into `main` (GitHub only reads `schedule` triggers from the workflow + # file version on the repo's default branch, not from feature branches). + # schedule: + # - cron: '0 3 * * *' # 03:00 UTC = 00:00 (meia-noite) em Brasília + workflow_dispatch: # allows manual trigger from the Actions tab + inputs: + tag: + description: 'Robot tag to run' + required: false + default: regression + type: choice + options: + - smoke + - regression concurrency: group: robot-tests-${{ github.ref }} @@ -21,6 +36,21 @@ jobs: - name: Checkout code uses: actions/checkout@v4 + - name: Determine which tag to run + id: tag + shell: bash + run: | + # PRs get the fast "smoke" subset for quick feedback; pushes to + # main and the nightly schedule run the full "regression" suite. + # workflow_dispatch lets a human pick either one on demand. + if [ "${{ github.event_name }}" = "pull_request" ]; then + echo "value=smoke" >> "$GITHUB_OUTPUT" + elif [ "${{ github.event_name }}" = "workflow_dispatch" ]; then + echo "value=${{ github.event.inputs.tag }}" >> "$GITHUB_OUTPUT" + else + echo "value=regression" >> "$GITHUB_OUTPUT" + fi + - name: Set up Python uses: actions/setup-python@v5 with: @@ -53,6 +83,7 @@ jobs: run: | robot \ --variable HEADLESS:True \ + --include ${{ steps.tag.outputs.value }} \ --outputdir results \ --xunit xunit.xml \ tests/ @@ -61,14 +92,21 @@ jobs: # global_page.resource's "New Browser" keyword (CLI --variable takes # precedence over the *** Variables *** default), so the suite runs # headless in CI while local runs keep the resource's own default. + # --include filters by [Tags]: "smoke" is the small, fast, + # critical-path subset run on every PR; "regression" covers the + # whole suite (every test carries it) and runs on merges to main + # and the nightly schedule. - name: Publish summary on GitHub Actions if: always() run: | { echo "### Robot Framework Suite Result" + echo "Tag executed: \`${{ steps.tag.outputs.value }}\`" if [ -f results/output.xml ]; then - echo "Full report available in the **robot-framework-results** artifact." + echo "Full report (including the **Statistics by Tag** breakdown" + echo "for smoke/regression/critical/high/medium and ui) is" + echo "available in the **robot-framework-results** artifact." fi } >> "$GITHUB_STEP_SUMMARY" diff --git a/README.md b/README.md index 36b91cc..f1189c5 100644 --- a/README.md +++ b/README.md @@ -264,17 +264,50 @@ button on the Cadastro de Usuário card`), which simply delegate to the `newUser_page.resource` / `newProduct_page.resource` / `home_page.resource` keywords described above. -## Continuous Integration +## Tags + +Every test case carries three kinds of `[Tags]`, so the suite can be sliced +without touching test code: + +| Dimension | Tags | Meaning | +|-------------------|------------------------------------|---------| +| **Execution set** | `smoke`, `regression` | `regression` is on every test (the full suite). `smoke` marks the small, fast subset of critical happy paths — currently 5 of the 14 tests — meant to run on every PR for quick feedback. | +| **Criticality** | `critical`, `high`, `medium` | `critical` = core journeys the app is unusable without (login, admin create user/product, add to cart). `high` = important supporting flows (listing, quantity, clearing). `medium` = negative/validation edge cases. | +| **Layer** | `ui` | All current tests drive the browser end-to-end (API is only used for setup/teardown). Kept as an explicit tag so future API-only suites can be filtered out (`--exclude ui`) or in (`--include ui`) separately. | + +Module tags (`login`, `admin`, `store`/`list`) are also kept so a single +feature area can be run in isolation, e.g. `robot --include admin tests/`. + +Tags are combinable, e.g. run every critical test regardless of area: -`.github/workflows/robot-tests.yml` runs the full suite on GitHub Actions: +```bash +robot --include critical tests/ +``` + +Robot Framework's `report.html` automatically breaks down pass/fail stats +per tag ("Statistics by Tag") with no extra flags required — once the tags +above are applied, that view already gives a success rate by criticality +(`critical`/`high`/`medium`) for free, which is what gets reported to +stakeholders. + +## Continuous Integration -- **Triggers**: pull requests and pushes to `main`, plus manual runs via - `workflow_dispatch`. Runs are cancelled/deduped per branch (`concurrency`). +`.github/workflows/robot-tests.yml` runs the suite on GitHub Actions, using +tags to control how much of it runs per trigger: + +- **Triggers**: + - `pull_request` to `main` → runs `--include smoke` only, for fast PR + feedback. + - `push` to `main` and a nightly `schedule` (03:00 UTC) → run + `--include regression`, the full suite. + - `workflow_dispatch` → manual run with a `tag` input (`smoke` or + `regression`, defaults to `regression`). + - Runs are cancelled/deduped per branch (`concurrency`). - Sets up Python and Node.js (the Browser library driver needs Node), installs `requirements.txt`, and runs `rfbrowser init` (with the downloaded Playwright browsers cached across runs). -- Executes `robot --outputdir results --xunit xunit.xml tests/`; the job - fails if any test fails. +- Executes `robot --include --outputdir results --xunit xunit.xml + tests/`; the job fails if any test fails. - Uploads `log.html`, `report.html`, and `output.xml` as the `robot-framework-results` artifact, and `xunit.xml` as the `robot-junit-report` artifact (both kept for 15 days), and writes a short diff --git a/tests/admin/admin.robot b/tests/admin/admin.robot index 2291e8e..190dbeb 100644 --- a/tests/admin/admin.robot +++ b/tests/admin/admin.robot @@ -11,7 +11,7 @@ Test Teardown Cleanup Login Test *** Test Cases *** Admin Wants To Create a New User [Documentation] This test case verifies that an admin can create a new user. - [Tags] admin smoke + [Tags] admin ui smoke regression critical ${name} ${email} ${password}= Generate Random User Given the admin wants to create a new user When the admin clicks on the "Cadastrar" button on the Cadastro de Usuário card @@ -21,14 +21,14 @@ Admin Wants To Create a New User Admin Wants To See The List Of Users [Documentation] This test case verifies that an admin can see the list of users. - [Tags] admin smoke + [Tags] admin ui regression high Given the admin wants to see the list of users When the admin clicks on the "Listar" button on the Listar Usuários card Then the list of users should be displayed Admin Wants To Create A New Product [Documentation] This test case verifies that an admin can create a new product. - [Tags] admin smoke + [Tags] admin ui smoke regression critical ${productName} ${price} ${description} ${quantity}= Generate Random Product Given the admin wants to create a new product When the admin clicks on the "Cadastrar" button on the Cadastrar Produtos card @@ -38,7 +38,7 @@ Admin Wants To Create A New Product Admin Wants To List All Products [Documentation] This test case verifies that an admin can see the list of products. - [Tags] admin smoke + [Tags] admin ui regression high Given the admin wants to see the list of products When the admin clicks on the "Listar" button on the Listar Produtos card Then the list of products should be displayed diff --git a/tests/login/login.robot b/tests/login/login.robot index 58f0932..243f3c9 100644 --- a/tests/login/login.robot +++ b/tests/login/login.robot @@ -13,7 +13,7 @@ Test Teardown Cleanup Login Test User With Valid Credentials Should Be Redirected To The Home [Documentation] A user who enters a valid email and password and clicks ... "Entrar" must be redirected to the home page. - [Tags] login smoke + [Tags] login ui smoke regression critical Given a user is trying to login When this user types the email ${EMAIL} And this user types the password ${PASSWORD} @@ -22,7 +22,7 @@ User With Valid Credentials Should Be Redirected To The Home Admin Can Access Admin Page [Documentation] This test case verifies that an admin user can access the admin page. - [Tags] admin smoke + [Tags] admin login ui smoke regression critical Given the admin user wants to access the admin page When this admin user types the email and password Then the admin page should be displayed @@ -31,7 +31,7 @@ User With Invalid Email Should See Error Message [Documentation] A user who enters an invalid email and/or password and clicks ... "Entrar" must see a message informing that the ... credentials are invalid. - [Tags] login + [Tags] login ui regression medium Given a user is trying to login When this user types the email email.invalido@qa.com.br And this user types the password ${PASSWORD} @@ -42,7 +42,7 @@ User With Invalid Password Should See Error Message [Documentation] A user who enters an invalid email and/or password and clicks ... "Entrar" must see a message informing that the ... credentials are invalid. - [Tags] login + [Tags] login ui regression medium Given a user is trying to login When this user types the email ${EMAIL} And this user types the password senhaInvalida123 @@ -53,7 +53,7 @@ User With Blank Email Should See Error Message [Documentation] A user who enters an invalid email and/or password and clicks ... "Entrar" must see a message informing that the ... credentials are invalid. - [Tags] login + [Tags] login ui regression medium Given a user is trying to login When this user types the password ${PASSWORD} And clicks on Entrar @@ -63,7 +63,7 @@ User With Blank Password Should See Error Message [Documentation] A user who enters an invalid email and/or password and clicks ... "Entrar" must see a message informing that the ... credentials are invalid. - [Tags] login + [Tags] login ui regression medium Given a user is trying to login When this user types the email ${EMAIL} And clicks on Entrar @@ -72,7 +72,7 @@ User With Blank Password Should See Error Message User With Invalid Email Format Should See Error Message [Documentation] A user who enters a valid email and password and clicks ... "Entrar" must be redirected to the home page. - [Tags] login smoke + [Tags] login ui regression medium Given a user is trying to login When this user types the email teste@teste And this user types the password ${PASSWORD} diff --git a/tests/store/store.robot b/tests/store/store.robot index 8bc92ad..bdfb8bf 100644 --- a/tests/store/store.robot +++ b/tests/store/store.robot @@ -10,7 +10,7 @@ Test Teardown Cleanup Login Test *** Test Cases *** User Can Add 2 Items to List [Documentation] This test case verifies that a user can add two items to the list. - [Tags] list smoke + [Tags] store list ui smoke regression critical Given the user wants to add 2 items to the list When the user adds the first item to the list Logitech MX Vertical And the user adds the second item to the list Samsung 60 polegadas @@ -18,7 +18,7 @@ User Can Add 2 Items to List User Can Increase Quantity of an Item in the List [Documentation] This test case verifies that a user can increase the quantity of an item in the list. - [Tags] list smoke + [Tags] store list ui regression high Given the user wants to increase item in the list When the user adds an item to the list Logitech MX Vertical And the user increases the quantity of the item in the list @@ -26,7 +26,7 @@ User Can Increase Quantity of an Item in the List User Can Clear the List [Documentation] This test case verifies that a user can clear the list. - [Tags] list smoke + [Tags] store list ui regression high Given the user wants to clear the list When the user adds an item to the list Logitech MX Vertical And the user clears the list