diff --git a/.github/workflows/robot-tests.yml b/.github/workflows/robot-tests.yml index 2b495a7..1b18739 100644 --- a/.github/workflows/robot-tests.yml +++ b/.github/workflows/robot-tests.yml @@ -5,12 +5,12 @@ on: branches: [ main ] push: branches: [ main ] - # Nightly full regression run — DISABLED for now. - # To turn it back on: uncomment the two lines below and merge this file + # Nightly full regression run — ENABLED for now. + # To turn it back off: comment the two lines below and merge this file # into `main` (GitHub only reads `schedule` triggers from the workflow # file version on the repo's default branch, not from feature branches). - # schedule: - # - cron: '0 3 * * *' # 03:00 UTC = 00:00 (midnight) in Brasília + schedule: + - cron: '0 3 * * *' # 03:00 UTC = 00:00 (midnight) in Brasília workflow_dispatch: # allows manual trigger from the Actions tab inputs: tag: @@ -86,7 +86,8 @@ jobs: - name: Run Robot Framework tests run: | - robot \ + pabot \ + --processes 3 \ --listener allure_robotframework:allure-results \ --variable HEADLESS:True \ --include ${{ steps.tag.outputs.value }} \ @@ -94,6 +95,9 @@ jobs: --xunit xunit.xml \ tests/ continue-on-error: false # PR/merge must fail if any test breaks + # pabot runs suites in parallel (--processes 4) instead of robot's + # single process; all other flags below behave the same as with + # plain `robot`, since pabot is a drop-in wrapper around it. # HEADLESS:True overrides the ${HEADLESS} variable used in # global_page.resource's "New Browser" keyword (CLI --variable takes # precedence over the *** Variables *** default), so the suite runs diff --git a/.gitignore b/.gitignore index 715a404..1f831a1 100644 --- a/.gitignore +++ b/.gitignore @@ -13,4 +13,7 @@ browser/traces/ allure-results/ allure-report/ allure-history/ -gh-pages/ \ No newline at end of file +gh-pages/ + +# Pabot (parallel runner) +.pabotsuitenames \ No newline at end of file diff --git a/README.md b/README.md index dea6b23..5b9328a 100644 --- a/README.md +++ b/README.md @@ -402,3 +402,148 @@ rfbrowser init ```bash robot -d results tests ``` + +## Running the tests in parallel (Pabot) + +[`pabot`](https://pabot.org) (`robotframework-pabot`, included in +`requirements.txt`) runs each **suite** in its own process. In this project a +"suite" is one directory under `tests/` — `tests/admin`, `tests/login`, +`tests/store` — each holding a single `.robot` file, so by default pabot +parallelizes exactly at that granularity: + +- `admin.robot`, `login.robot`, and `store.robot` run **concurrently**, in + separate processes — they don't share state (each test creates and tears + down its own user/product via the API), so running them side by side is + safe. +- The test cases **inside** each `.robot` file keep running **sequentially**, + one after another in that suite's single process. + +```bash +pabot --processes 3 --outputdir results tests +``` + +`--processes 3` matches the current number of suites (one process per +suite is enough since pabot doesn't split further); raising it doesn't buy +more parallelism today but won't break anything either — it just leaves the +extra processes idle. Any other `robot` flag (`--include smoke`, +`--variable HEADLESS:True`, `--listener allure_robotframework:allure-results`, +...) works the same way with `pabot`. + +### Test-level splitting (`--testlevelsplit`) + +`--testlevelsplit` makes pabot split *inside* each suite too, so individual +test cases (not just whole suites) get scheduled onto the process pool. Each +split test still runs in its own pabot subprocess — its own Python +interpreter and its own Browser/Playwright driver — so it's just as isolated +from its sibling tests as suites already are from each other: nothing about +this project's tests (each of which creates and tears down its own +user/product via the API) makes them unsafe to run concurrently. + +That was verified empirically, not just assumed: across 13 separate +`--testlevelsplit` runs (all 21 tests split, mixed freely across `admin`, +`login`, and `store`, at process counts from 3 to 8, several repeated to +check for flakiness) the only failure that ever showed up was `Regular User +Should Not Be Able To Access The Admin Home Page`, the pre-existing +`known-issue` test — the exact same, single failure `--processes 3` alone +also produces. No data race, no session bleed, no flaky failure caused by +concurrency ever appeared. + +**But it isn't a reliable speed win today, so it isn't the default.** With +only 3 suites and 4-9 tests each, `--processes 3` already parallelizes as +much as matters, and splitting further mostly adds per-test process +start-up overhead (a fresh interpreter + Browser driver per test instead of +per suite) plus contention between more concurrent Chromium instances than +the machine has cores for. Measured on an 8-core machine, running the full +`tests/` tree: + +| Command | Runs | Elapsed time (avg) | Spread | +| --- | --- | --- | --- | +| `pabot --processes 3 tests` (default, no split) | 4 | **54.1s** | 52.4s – 55.4s (±1.1s) | +| `pabot --testlevelsplit --processes 3 tests` | 1 | 70.0s | — | +| `pabot --testlevelsplit --processes 5 tests` | 2 | 64.8s | 56.7s – 72.9s | +| `pabot --testlevelsplit --processes 6 tests` | 2 | 52.6s | 52.3s – 52.9s | +| `pabot --testlevelsplit --processes 8 tests` | 4 | 50.8s | 44.9s – 60.9s (±6.4s) | + +No process count tested consistently beat the plain `--processes 3` default +by a reliable margin — the best individual runs did (as fast as 44.9s at +`--processes 8`), but so did the worst (60.9s, also at `--processes 8`), and +the average across every split variant (56.5s) actually landed slightly +*behind* the default's 54.1s. The variance looks driven by the shared, +third-party ServeRest demo backend's response times more than by local CPU +contention, which local process-count tuning can't fix. The default's tight +±1.1s spread makes it the more predictable choice for CI, where a +consistent run time matters as much as the average. + +**When `--testlevelsplit` would earn its keep:** if a single suite grows +large enough that it — not the number of suites — becomes the bottleneck +(e.g. `admin.robot` growing from today's 8 tests to 25+), suite-level +parallelism stays capped at 3 processes no matter how many cores are +available, while test-level splitting can spread that one suite's tests +across every core. At that point, split *only* the oversized suite instead +of the whole `tests/` tree, so the other suites keep running in their own +dedicated process and you don't reintroduce the run-to-run variance seen +above: + +```bash +pabot --testlevelsplit --processes 4 --outputdir results tests/admin +``` + +For finer control — e.g. splitting just the large suite while still running +the other suites in the same pabot invocation — pabot supports an +`--ordering` file that mixes `--suite` lines (suite stays whole) with +`--test` lines (that suite's tests get split individually), see the +[pabot docs](https://pabot.org) for the exact syntax. + +## Viewing results with Allure + +CI already publishes an [Allure](https://allurereport.org/) report for every +run (see [Continuous Integration](#continuous-integration)); the same report +can be generated locally. + +`allure-robotframework` (already in `requirements.txt`) provides the +`allure_robotframework` **listener**, which writes Allure's raw result files +as the suite runs. Turning it on just means adding `--listener` to the +`robot`/`pabot` command already in use: + +```bash +robot --listener allure_robotframework:allure-results -d results tests +``` + +Rendering those raw results into an HTML report additionally requires the +Allure **command-line tool**, a separate Java-based install (not a Python +package, so it's not in `requirements.txt`): + +```bash +npm install -g allure-commandline +``` + +Then, to build and open the report: + +```bash +# generates the report into allure-report/ and opens it in the browser +allure serve allure-results +``` + +`allure serve` is the quickest option for a one-off local look (it uses a +temp folder under the hood). To keep the generated report as a file instead +— e.g. to share it — generate and open it explicitly: + +```bash +allure generate allure-results --clean -o allure-report +allure open allure-report +``` + +Every `robot`/`pabot` run appends new results into `allure-results/` rather +than replacing it, so a report generated without `--clean` shows the +accumulated history of every local run, not just the latest one; delete +`allure-results/` first (or keep using `--clean` when generating) for a +report scoped to a single run. + +The listener works the same way together with `pabot` — each subprocess +writes its own result files into the same `allure-results/` folder (Allure +uses one file per test, so parallel writers don't conflict): + +```bash +pabot --processes 3 --listener allure_robotframework:allure-results --outputdir results tests +allure serve allure-results +``` diff --git a/requirements.txt b/requirements.txt index 7d81bdc..b712b20 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,5 +1,6 @@ robotframework robotframework-browser robotframework-requests +robotframework-pabot Faker==26.0.0 allure-robotframework \ No newline at end of file diff --git a/resources/variables/global.resource b/resources/variables/global.resource index 4a6d379..7d95a4c 100644 --- a/resources/variables/global.resource +++ b/resources/variables/global.resource @@ -5,4 +5,4 @@ Library Browser ${BASE_URL} https://front.serverest.dev ${API_BASE_URL} https://serverest.dev ${BROWSER} chromium -${HEADLESS} False +${HEADLESS} True diff --git a/serve b/serve new file mode 100644 index 0000000..38ae36d Binary files /dev/null and b/serve differ