ci: add per-component docker startup integration tests - #63
Conversation
Adds a new 'integration-test' CI job that boots the built Docker image
for every AAS component (aas-repo, submodel-repo, aas-registry,
submodel-registry) in an isolated matrix leg, gating merges on all four
starting successfully.
Each leg runs scripts/ci/docker-startup-check.sh, which performs:
1. an MCP 'initialize' handshake and asserts
result.serverInfo.name == 'AAS MCP Server (<component>)';
2. an uptime check — starts a detached container and verifies it is
still running with exit code 0 after UPTIME_SECONDS (default 10).
The script is reusable locally against any freshly built
aas-mcp-server:test image.
Fixtures under tests/fixtures/integration/ are minimal, standalone
OpenAPI 3.0 documents (<=200 lines each): the aas-repo fixture is
hand-authored in the style of tests/fixtures/sample_official_spec.yaml;
the other three are derived by manual chunking from the official IDTA
specs (see aas-specs repo). Fixtures are isolated from unit-test
fixtures so changes to one do not force changes to the other.
The existing single-component smoke test in the 'docker' job is kept as
a fast-fail signal for the image build itself; a comment notes that
full four-component coverage now lives in 'integration-test'.
Local verification: all four components boot cleanly and pass the
two-part check against the actually-built image; existing pytest suite
unchanged (259 passed, 13 skipped, matches baseline).
Closes #62
ricogu
left a comment
There was a problem hiding this comment.
Nice addition — per-component startup coverage was a real gap and the matrix shape is clean. One blocking item before merge: the Phase-1 trap can mask a failed handshake as a green run (inline on the script). The rest are should-fix robustness (word-splitting, mktemp) plus an optional tools/list assertion so the gate proves tools actually loaded, and one doc nit.
For what it's worth, all four legs and test-summary are green today — but the trap bug means a real handshake failure might not have surfaced, so worth fixing before relying on it as a gate.
| INIT_REQUEST='{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{},"clientInfo":{"name":"docker-startup-check","version":"1.0"}}}' | ||
|
|
||
| HANDSHAKE_LOG="$(mktemp)" | ||
| trap 'rm -f "$HANDSHAKE_LOG"; cleanup' EXIT INT TERM |
There was a problem hiding this comment.
Blocking: this can turn a failed handshake into a green run. The trap runs rm -f "$HANDSHAKE_LOG" before cleanup, and cleanup() reads local rc=$? — so $? is rm's exit code (0), not the original exit 1 from lines 137/147. A broken handshake would exit 0 and pass the gate.
Capture the status first:
cleanup() { local rc="${1:-$?}"; ...; exit "$rc"; }
trap 'rc=$?; rm -f "$HANDSHAKE_LOG"; cleanup "$rc"' EXIT INT TERM| MOUNT_CONFIG="-v ${CONFIG_PATH}:/app/config/config.yaml:ro" | ||
| MOUNT_SPEC="-v ${SPEC_DIR}:/app/spec:ro" |
There was a problem hiding this comment.
These strings get expanded unquoted into docker run further down, so they word-split on spaces. Fine on the CI runner, but the script is documented as locally runnable and will break for a checkout under a path with spaces. Arrays are safer:
MOUNT_ARGS=(
-v "${CONFIG_PATH}:/app/config/config.yaml:ro"
-v "${SPEC_DIR}:/app/spec:ro"
)
# docker run ... "${MOUNT_ARGS[@]}" "${ENV_ARGS[@]}" ...| ) | ||
|
|
||
| # Temp file for uptime-phase container ID; cleaned up on any exit. | ||
| UPTIME_CIDFILE="$(mktemp -u)" |
There was a problem hiding this comment.
Minor: mktemp -u hands back a name without creating the file (TOCTOU). Low risk in CI, but easy to avoid — use a private dir and put the cidfile inside it:
CID_DIR="$(mktemp -d)"
UPTIME_CIDFILE="$CID_DIR/container.cid"
# ...clean up the dir in cleanup()| fi | ||
|
|
||
| if ! echo "$HANDSHAKE_LINE" | jq -e --arg name "$EXPECTED_NAME" '.result.serverInfo.name == $name' >/dev/null; then | ||
| echo "ERROR: handshake response did not match expected serverInfo.name=\"$EXPECTED_NAME\"" >&2 |
There was a problem hiding this comment.
The handshake only checks serverInfo.name, which is derived from AAS_COMPONENT — a component with zero exposed tools (e.g. a mis-parsed allowlist that curates everything away) would still pass here. Consider following up with tools/list and asserting it's non-empty, so the gate actually proves the spec + curation loaded something:
echo "$TOOLS_RESPONSE" | jq -e '.result.tools | length > 0'| Environment variables: | ||
| - `UPTIME_SECONDS` — how long the uptime phase waits (default `10`) | ||
| - `IMAGE_TAG` — image tag to test (default `aas-mcp-server:test`) | ||
|
|
There was a problem hiding this comment.
Nit: this points at docs/*.yaml, but the fixture README says the derived specs come from the external admin-shell-io/aas-specs repo (and aas-repo is hand-authored). Worth pointing this line at tests/fixtures/integration/README.md so future refreshers look in the right place.
Closes #62
Summary
Adds a new
integration-testCI job that boots the built Docker image for every AAS component (aas-repo,submodel-repo,aas-registry,submodel-registry) in an isolated matrix leg, gating merges on all four starting successfully. Before this change, onlyaas-repowas smoke-tested in CI — the other three components could silently regress until a downstream user hit the failure.What changes
New CI job:
integration-testdockerbuildsaas-mcp-server:test, beforetest-summary.strategy.fail-fast: false) over the AAS components — a broken component fails only its own leg.cache-from: type=gha(near-instant cache hit off the warmeddockerjob).test-summarynow gates onintegration-testin addition totest,lint,docker.dockerjob is kept as a fast-fail signal for the image build itself; an inline comment notes that full per-component coverage lives inintegration-test.New helper:
scripts/ci/docker-startup-check.shTwo-part startup check that CI invokes per matrix leg and any developer can run locally:
initializeJSON-RPC request intodocker run -i, asserts.result.serverInfo.name == "AAS MCP Server (<component>)"viajq -e. Printsdocker logson failure.-i, sleepsUPTIME_SECONDS(default 10 s), assertsState.Running == trueandState.ExitCode == 0.trap-based cleanup guarantees no leaked containers.Env vars:
UPTIME_SECONDS(default 10),IMAGE_TAG(defaultaas-mcp-server:test).--helpdocuments local usage.New integration fixtures:
tests/fixtures/integration/Five files, isolated from unit-test fixtures so they can evolve independently:
aas-repo-official-spec.yaml— hand-authored in the shape oftests/fixtures/sample_official_spec.yaml.submodel-repo-official-spec.yaml,aas-registry-official-spec.yaml,submodel-registry-official-spec.yaml— each derived by manual chunking from the corresponding official IDTA spec (aas-specs) into a small, valid standalone OpenAPI 3.0 doc (≤200 lines, ≥1GET, ≥1 path parameter, originaloperationIds preserved).config.yaml.template— one config declaring all four components withcuration.allowlist: [[get, "*"]].README.md— documents the chunking rule and local usage.Docs
tests/README.md— new "Docker Startup Integration Test (Local)" section with a one-liner developer usage pattern.Design decisions (why this shape)
docker. Keeps the image build as a single canonical step; each matrix leg pays only the cache-hit cost. Alternatives (bash loop, matrix ondockeritself) rejected — see commit body for rationale.initialize).AAS_BASE_URL. The server does not dial the backend at startup; nothing needs to listen on that port. If future changes start eagerly probing the backend the integration test will fail loudly — that's the correct signal.official_specon load, so mounting only the selected component's spec fails validation for the other three. The script mounts the wholetests/fixtures/integration/dir at/app/spec/; an inline comment documents why.Verification
Local (
docker build -t aas-mcp-server:test .+ Docker Desktop 28.3.3):Both the handshake and 10-second uptime sub-checks pass for every component against the actually-built image.
uv run pytest tests/ -m "not integration"→ 259 passed, 13 skipped (matches baseline; no unit-test regression).uv run reuse lint→ 73/73 files compliant; new script and config template carry explicit SPDX headers, YAMLs covered by the aggregatepath = "**"rule.yaml.safe_load, satisfy the structural check (openapi,info.title,info.version, non-emptypaths, ≥1GET, ≥1 path parameter), each ≤ 200 lines (81/67/67/67).Compatibility
Dockerfile, unit tests, production configs, release workflow.docker,jq,bash— all already onubuntu-latest.Non-goals
initialize.Follow-ups for reviewers
test-summarycorrectly fails — I did not do this in a pushed commit to keep history clean.