Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .config/nextest.toml
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@ max-threads = 1
filter = """
binary(diesel_user_repository)
| binary(diesel_user_preferences_repository)
| binary(diesel_login_users_adapters)
| binary(diesel_profile_interests_adapters)
| binary(diesel_route_annotation_repository)
| binary(diesel_example_data_runs_repository)
| binary(er_snapshots_bdd)
Expand All @@ -29,9 +31,18 @@ filter = """
| binary(catalogue_descriptor_ingestion_bdd)
| binary(catalogue_descriptor_read_models_bdd)
| binary(offline_bundle_walk_session_bdd)
| binary(osm_ingestion_bdd)
| binary(overpass_enrichment_bdd)
| binary(ports_behaviour)
| binary(pg_embed_isolation)
| binary(route_queue_apalis_bdd)
| binary(schema_baseline_bdd)
| binary(startup_mode_composition_bdd)
| binary(user_interests_revision_conflicts_bdd)
| binary(user_state_profile_interests_startup_modes_bdd)
| binary(user_state_schema_audit_bdd)
| binary(user_state_startup_modes_bdd)
| binary(users_list_pagination_bdd)
"""
test-group = "pg-embed"
threads-required = 4
Expand Down
21 changes: 20 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -138,10 +138,17 @@ jobs:
path: |
~/.cargo/registry
~/.cargo/git
~/.theseus/postgresql
target
key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
restore-keys: ${{ runner.os }}-cargo-
- name: Cache PostgreSQL embedded binaries
uses: ubicloud/cache@v4
with:
Comment on lines +145 to +146
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail
# Verify unpinned workflow actions in CI workflow.
rg -nP '^\s*uses:\s*[^@\s]+@v[0-9]+(\.[0-9]+)*\s*$' .github/workflows/ci.yml

Repository: leynos/wildside

Length of output: 201


Pin ubicloud/cache to an immutable commit SHA (avoid tag-drift).

ci.yml uses ubicloud/cache@v4 unpinned (lines 42, 53, 136, 145); replace @v4 with a full commit SHA for each use.

🧰 Tools
🪛 zizmor (1.25.2)

[error] 145-145: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)

(unpinned-uses)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/ci.yml around lines 145 - 146, The workflow uses
ubicloud/cache@v4 which is an unpinned tag; update each occurrence of the string
"uses: ubicloud/cache@v4" in .github/workflows/ci.yml to reference an immutable
commit SHA instead (e.g., "uses: ubicloud/cache@<COMMIT_SHA>"); ensure you
replace all instances so every use of the ubicloud/cache action is pinned to the
same verified commit SHA and commit the change.

path: |
~/.theseus/postgresql
~/.cache/pg-embedded/binaries
key: ${{ runner.os }}-pg-embedded-${{ hashFiles('**/Cargo.lock') }}
restore-keys: ${{ runner.os }}-pg-embedded-
Comment on lines +148 to +151
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Use a stable cache key for PostgreSQL binaries.

Replace the Cargo.lock-derived key on Line 150 with a PostgreSQL-version-scoped key so unrelated Rust dependency updates do not evict warmed binary archives and reintroduce cold-download flakiness.

Proposed fix
       - name: Cache PostgreSQL embedded binaries
         uses: ubicloud/cache@v4
         with:
           path: |
             ~/.theseus/postgresql
             ~/.cache/pg-embedded/binaries
-          key: ${{ runner.os }}-pg-embedded-${{ hashFiles('**/Cargo.lock') }}
-          restore-keys: ${{ runner.os }}-pg-embedded-
+          key: ${{ runner.os }}-pg-embedded-16.10.0
+          restore-keys: |
+            ${{ runner.os }}-pg-embedded-

Based on learnings: "Keep CI PostgreSQL binary cache locations separate ... do not co-locate inside Cargo registry."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/ci.yml around lines 148 - 151, The cache key for
PostgreSQL binaries currently uses a Cargo.lock hash (key: ${{ runner.os
}}-pg-embedded-${{ hashFiles('**/Cargo.lock') }}) which causes unrelated Rust
dep changes to evict the binary cache; change that key to be scoped to the
PostgreSQL version (for example use a fixed pg version token or an env var like
PG_VERSION in the key, e.g. ${{ runner.os }}-pg-embedded-${{ env.PG_VERSION }})
and update the corresponding restore-keys entry to match (restore-keys: ${{
runner.os }}-pg-embedded-) so the cache persists across Cargo.lock changes and
only rotates when PostgreSQL itself changes.

- name: Rust build
run: cargo check --locked --manifest-path backend/Cargo.toml --all-targets --all-features
- name: Rust fmt check
Expand All @@ -168,6 +175,16 @@ jobs:
run: |
make prepare-pg-worker

- name: Warm PostgreSQL embedded binary cache
env:
# Increase GitHub API rate limits for postgresql_embedded downloads.
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
# Pin the embedded PostgreSQL version so postgresql_archive skips release-listing queries.
# Use the semver "exact" prefix so no wildcard resolution is attempted.
POSTGRESQL_VERSION: "=16.10.0"
POSTGRESQL_RELEASES_URL: https://github.com/theseus-rs/postgresql-binaries/releases
run: bash scripts/warm-pg-embedded-cache.sh

- name: Rust tests
env:
# Increase GitHub API rate limits for postgresql_embedded downloads.
Expand All @@ -179,6 +196,8 @@ jobs:
PG_TEST_BACKEND: postgresql_embedded
# Root-path bootstrap requires the worker binary for privilege demotion.
PG_EMBEDDED_WORKER: ${{ github.workspace }}/target/pg_worker
# Match the repository Makefile and avoid concurrent first-use cluster bootstrap.
NEXTEST_TEST_THREADS: "1"
run: |
# Clean stale pg-embed directories that may conflict with new runs.
find target/ -maxdepth 1 -type d -name 'pg-embed-*' -exec rm -rf {} + 2>/dev/null || true
Expand Down
8 changes: 4 additions & 4 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -158,7 +158,7 @@ lint-actions:

PG_WORKER_PATH ?= $(CURDIR)/target/pg_worker
PG_WORKER_INSTALL_ROOT ?= $(CURDIR)/target/pg-worker-root
PG_EMBED_SETUP_UNPRIV_VERSION ?= 0.5.0
PG_EMBED_SETUP_UNPRIV_VERSION ?= 0.5.1
NEXTEST_TEST_THREADS ?= 1

test: test-rust test-frontend
Expand Down
2 changes: 1 addition & 1 deletion backend/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ clap = { version = "4", features = ["derive"] }
async-trait = "0.1.89"
color-eyre = "0.6.5"
pg_embedded_setup_unpriv = { package = "pg-embed-setup-unpriv", version = "0.5.1" }
postgresql_embedded = { version = "0.20.0", features = ["tokio"] }
postgresql_embedded = { version = "0.20.2", features = ["tokio"] }
postgres = { version = "0.19.12", features = ["with-uuid-1"] }
paste = "1.0.15"
thiserror = "2.0.17"
Expand Down
41 changes: 40 additions & 1 deletion backend/tests/support/atexit_cleanup.rs
Original file line number Diff line number Diff line change
Expand Up @@ -160,6 +160,19 @@ fn ensure_stable_password() {
std::env::set_var("PG_PASSWORD", "wildside_embedded_test");
}
}
if std::env::var_os("POSTGRESQL_RELEASES_URL").is_none() {
// Pin to Theseus binaries to avoid transient fetch failures in CI that
// reqwest misreports as "error decoding response body".
// SAFETY: called before the library spawns any threads. The shared-
// cluster singleton serialises access with a Mutex, so this runs at
// most once per process.
unsafe {
std::env::set_var(
"POSTGRESQL_RELEASES_URL",
"https://github.com/theseus-rs/postgresql-binaries/releases",
);
Comment on lines +170 to +173
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Use the repository URL for POSTGRESQL_RELEASES_URL

When local embedded-Postgres tests run with POSTGRESQL_RELEASES_URL unset and the binary cache is cold, this default makes postgresql_archive fail before it can select an asset: its GitHub matcher registry only supports the exact Theseus repository URL (https://github.com/theseus-rs/postgresql-binaries), while the /releases suffix leaves it with an unsupported matcher. CI currently masks this by setting the env var without the suffix in the workflow, but the repo default path now breaks the documented zero-config local test path.

Useful? React with 👍 / 👎.

}
}
}

/// Reads the postmaster PID from the `postmaster.pid` file in `data_dir`.
Expand Down Expand Up @@ -287,13 +300,39 @@ mod tests {

#[test]
fn ensure_stable_password_does_not_overwrite_existing_value() {
let _guard = env_lock::lock_env([("PG_PASSWORD", Some("custom_value"))]);
let _guard = env_lock::lock_env([
("PG_PASSWORD", Some("custom_value")),
(
"POSTGRESQL_RELEASES_URL",
Some("https://example.invalid/postgresql-binaries"),
),
]);
super::ensure_stable_password();
assert_eq!(
std::env::var("PG_PASSWORD").expect("PG_PASSWORD should be set"),
"custom_value",
"ensure_stable_password should not overwrite an existing PG_PASSWORD"
);
assert_eq!(
std::env::var("POSTGRESQL_RELEASES_URL")
.expect("POSTGRESQL_RELEASES_URL should be set"),
"https://example.invalid/postgresql-binaries",
"ensure_stable_password should not overwrite an existing release URL"
);
}

#[test]
fn ensure_stable_password_sets_release_url_when_missing() {
let _guard = env_lock::lock_env([
("PG_PASSWORD", Some("custom_value")),
("POSTGRESQL_RELEASES_URL", None),
]);
super::ensure_stable_password();
assert_eq!(
std::env::var("POSTGRESQL_RELEASES_URL")
.expect("POSTGRESQL_RELEASES_URL should be set"),
"https://github.com/theseus-rs/postgresql-binaries/releases"
);
}

#[test]
Expand Down
3 changes: 3 additions & 0 deletions docs/contents.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,9 @@
- [Wildside front-end roadmap](frontend-roadmap.md) – GIST-aligned
implementation roadmap for the Progressive Web Application (PWA) front-end.
_Audience: frontend developers and project planners._
- [Front-end source authority catalogue](frontend-source-authority-catalogue.md)
– ownership map for front-end requirements and reconciliation follow-ups.
_Audience: frontend developers, reviewers, and project planners._
- [Pure, accessible, and localizable React components](pure-accessible-and-localizable-react-components.md)
– building accessible, localizable components with Radix, TanStack, and
DaisyUI. _Audience: frontend developers._
Expand Down
84 changes: 59 additions & 25 deletions docs/developers-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,10 @@ decisions that have not yet been implemented.

Canonical front-end references:

- [Front-end source authority catalogue](frontend-source-authority-catalogue.md)
identifies the authoritative source or reconciliation follow-up for each
front-end platform, data, user experience, API, styling, accessibility,
localization, and testing topic.
- [v2a front-end stack](v2a-front-end-stack.md) documents the current package
state and the target v2a stack boundary.
- [Wildside front-end roadmap](frontend-roadmap.md) is the implementation task
Expand Down Expand Up @@ -86,7 +90,7 @@ Run front-end commands through workspace or Makefile targets unless debugging a
package-local failure:

```bash
make build-frontend
make fe-build
make test-frontend
make lint-frontend
make typecheck
Expand Down Expand Up @@ -227,6 +231,45 @@ test usage remains coherent:
- Use `CleanupMode::None` only for explicit debugging sessions where retained
files are required; keep deterministic cleanup defaults for normal runs.

### Embedded PostgreSQL CI bootstrap stability

Continuous Integration (CI) warms the `pg-embed-setup-unpriv` binary cache with
`scripts/warm-pg-embedded-cache.sh` before running `cargo nextest`. Keep this
warm-up step before `Rust tests`; it turns PostgreSQL binary acquisition into a
short, explicit CI step instead of letting the first integration test perform a
cold download inside `postgresql_embedded::setup()`.

The CI cache step must include both binary-cache locations used by the two
embedded PostgreSQL layers:

- `~/.theseus/postgresql` for `postgresql_embedded` runtime installations.
- `~/.cache/pg-embedded/binaries` for `pg-embed-setup-unpriv` release archives.

Do not co-locate those paths inside the Cargo registry/cache archive. Cargo
dependency updates and `Cargo.lock` churn otherwise evict the PostgreSQL
binary cache and force a fresh download during unrelated test changes.

The warm-up step pins:

- `POSTGRESQL_VERSION="=16.10.0"` so archive resolution does not need wildcard
release discovery.
- `POSTGRESQL_RELEASES_URL=https://github.com/theseus-rs/postgresql-binaries/releases`
so the binary source cannot drift when crate defaults change.
- `GITHUB_TOKEN` so GitHub release requests use the Actions token and avoid
anonymous rate limits.

Keep PostgreSQL-backed nextest binaries in the `pg-embedded` test group in
`.config/nextest.toml`, and keep that group serialised. First-use cluster
bootstrap is process-local and expensive; serial execution avoids concurrent
setup attempts competing for the same warmed cache, filesystem paths, and
worker process.

If CI reports `error decoding response body`, treat it as a likely download
stall or timeout from `reqwest` rather than as JSON/body corruption. Check the
`Cache PostgreSQL embedded binaries` and `Warm PostgreSQL embedded binary
cache` steps first, then verify that the `Rust tests` step is still exporting
`PG_EMBEDDED_WORKER`, `GITHUB_TOKEN`, and `NEXTEST_TEST_THREADS=1`.

## Rust behavioural tests with `rstest-bdd` v0.5.0

### Dependency contract
Expand Down Expand Up @@ -467,26 +510,15 @@ Related domain helpers:
- `RouteCacheKeyDerivationError` reports `Hash` and `Validation` failures from
key derivation.

### Test infrastructure

The Redis adapter test suite uses a dual-mode approach:

**Mock-based unit tests** (run by default):

- Located in `backend/src/outbound/cache/tests/mock_tests.rs`
- Use `FakeProvider` – an in-memory `ConnectionProvider` double
- Fast, deterministic, no external dependencies
- Run as part of the standard `cargo test` / `make test` gate
#### Test infrastructure

**Live Redis integration tests** (opt-in):
- `pg-embedded-setup-unpriv` – Embedded PostgreSQL cluster for BDD tests
- No feature flags required; BDD tests are in the `tests/` integration
harness and run unconditionally with `cargo test`

- Located in `backend/src/outbound/cache/tests/live_tests.rs`
- Require a `redis-server` binary on `PATH`
- Marked with `#[ignore = "requires redis-server binary..."]`
- Run explicitly with: `cargo test -- --ignored`
- Behavioural coverage for route-key canonicalization lives in
`backend/tests/route_cache_key_canonicalization_bdd.rs`.
To run BDD tests locally:

```bash
### RedisTestServer harness

Comment on lines +519 to 523
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Fix the broken fenced code block before the Redis test harness heading.

Close or remove the fence opened on Line 521; the current structure swallows subsequent Markdown content and breaks document parsing.

Triage: [type:syntax/md]

As per coding guidelines: "**/*.md: Validate Markdown files using make markdownlint."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/developers-guide.md` around lines 519 - 523, The Markdown has an open
fenced code block starting after the "To run BDD tests locally:" line which
swallows the following content (including the "RedisTestServer harness"
heading); fix this by closing the triple-backtick fence immediately after the
intended code sample or by removing the stray opening fence so the
"RedisTestServer harness" heading and subsequent Markdown are parsed normally
(ensure the fence uses ``` and, if present, a language tag, and that the opening
and closing fences match around any intended code block).

Integration tests use `RedisTestServer` from `backend/src/test_support/redis.rs`:
Expand Down Expand Up @@ -520,17 +552,19 @@ The cache adapter requires:

#### Production dependencies

- `bb8-redis` – Connection pooling for `redis-rs`
- `serde` / `serde_json` – Payload serialization
- `apalis-core` – Core Apalis job-queue primitives
- `apalis-postgres` – PostgreSQL storage backend for Apalis
- `sqlx` (features: `postgres`, `runtime-tokio-rustls`) – Async PostgreSQL
pool used by `ApalisPostgresProvider`
- `serde` / `serde_json` – Payload serialisation
Comment on lines +555 to +559
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Restore Redis-specific dependency documentation in the Redis adapter section.

Replace Apalis queue dependencies on Lines 555-559 with Redis adapter dependencies; this section currently documents the wrong subsystem and misdirects implementation work.

Triage: [type:docstyle]

🧰 Tools
🪛 LanguageTool

[typographical] ~555-~555: Consider using an em dash in dialogues and enumerations.
Context: ...roduction dependencies - apalis-core – Core Apalis job-queue primitives - `apa...

(DASH_RULE)


[style] ~559-~559: Would you like to use the Oxford spelling “serialization”? The spelling ‘serialisation’ is also correct.
Context: ...der-serde/serde_json– Payload serialisation #### Test infrastructure -pg-embedd...

(OXFORD_SPELLING_Z_NOT_S)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/developers-guide.md` around lines 555 - 559, The listed Apalis/Postgres
dependencies (`apalis-core`, `apalis-postgres`, `sqlx` entries and the `serde`
note) are for the wrong subsystem; in the Redis adapter section replace that
block with Redis-specific dependencies: mention the Redis Apalis adapter crate
(e.g. `apalis-redis`), the async Redis client crate (`redis` with the
appropriate async/tokio feature), a connection-pool crate if used (e.g.
`deadpool-redis` or the project’s chosen pool), and keep `serde`/`serde_json`
for payload serialization; update the prose to reference those symbols
(`apalis-redis`, `redis`, `deadpool-redis`, `serde`) so implementers know which
crates/features to add.


#### Test infrastructure

- `test-support` feature flag – Enables `RedisRouteCache::new()` constructor
and `RedisTestServer::pool()` for test injection
- `redis-server` binary – Required for live integration tests (not for unit
tests using `FakeProvider`)
- `pg-embedded-setup-unpriv` – Embedded PostgreSQL cluster for BDD tests
- No feature flags required; BDD tests are in the `tests/` integration
harness and run unconditionally with `cargo test`

To run live Redis tests locally:
To run BDD tests locally:

```bash
# Ensure redis-server is available
Expand Down
Loading
Loading