Proposal: opt-in lazy shared services for independent test-suite members

[yaroslavvoloshchuk-codaio]

Motivation

rules_itest intentionally gives every service_test a fresh service graph, as clarified in #27. That should remain the default.

For expensive service graphs, however, users may have many independent tests that can safely share one graph. Running one service_test per member repeats setup, while replacing the tests with one aggregate runner loses Bazel's per-test cache keys, reporting, retries, sharding, and XML results.

I'd like to propose an explicitly opt-in shared mode with these semantics:

native.test_suite(
    name = "_raw_storage_tests",
    tests = [
        ":reader_test",
        ":writer_test",
    ],
    tags = ["manual"],
)

service_test(
    name = "storage_tests",
    test = ":_raw_storage_tests",
    services = [":storage_environment"],
    shared_service_key = "storage-tests",
)

The exact public API is one of the decisions below.

Proposed behavior

• Expand an explicit suite into one ordinary Bazel test wrapper per member. Bazel continues to cache, schedule, shard, retry, and report every member independently.
• Coordinate compatible executing wrappers through a generic host-local broker keyed by output base/configuration and an opaque shared_service_key.
• The first cache-miss wrapper elects an owner and starts the existing itest_service graph. Other executing wrappers acquire authenticated IPC leases and receive its runtime environment.
• If every member is a Bazel test-cache hit, no wrapper executes, so no broker or service starts.
• Closing a wrapper's connection releases its lease. After zero leases for a configurable grace period—30 seconds by default—the owner runs the existing dependency-ordered shutdown.
• Startup failure is broadcast to waiting wrappers. Test exit, failure, cancellation, service failure, owner failure, SIGINT, and SIGTERM release leases and trigger bounded cleanup. Stale coordination state is recoverable.
• The mechanism remains resource-agnostic: it knows about itest_service graphs and opaque environment values, not databases, Redis, Docker, or downstream-specific policy.
• Existing executable-target service_test behavior remains unchanged when shared mode is not requested.

Deliberate limitations

This is an opt-in, non-hermetic execution mode:

• Initially Linux and macOS only, using Unix-domain IPC.
• Shared wrappers require no-sandbox and no-remote-exec, while retaining Bazel test-result caching.
• It is not a strict Bazel invocation identity. Bazel exposes no test-side "last suite member" callback. The lifecycle coalesces overlapping or nearby executing actions using leases plus the idle grace period.
• Cleanup cannot be guaranteed after SIGKILL, kernel failure, or host loss, although subsequent executions recover stale discovery state.
• Different output bases/configurations do not share a graph.

Decisions requested

1. Public Starlark API: should service_test(test = ":suite") be supported by turning the exported symbol into a symbolic finalizer macro and retaining the raw rule under another name, or would an additive API such as shared_service_test_suite(...) be preferable to avoid changing the existing symbol?

2. Shared-state model: is an authenticated host-local owner/broker with connection leases and idle shutdown acceptable as an explicitly non-hermetic mode, or would maintainers prefer a different ownership boundary?

3. Platform and execution scope: is Linux/macOS plus no-sandbox/no-remote-exec an acceptable initial contract, with Windows and remote execution rejected clearly?

4. Timeout semantics: the prototype separates acquisition/startup, underlying test-body, idle, and shutdown timeouts. Should Bazel's wrapper timeout cover the complete owner-supervision lifecycle while a separate test_timeout begins only after services are ready? Should generated wrappers inherit each member's timeout rather than defaulting to eternal?

A first implementation can deliberately support only concrete, direct, same-package suites with explicit members. Nested, implicit tag-based, configurable, and cross-package suites would fail clearly rather than acquire surprising semantics.

Prior art and evidence

I searched the existing issues and PRs. #27 is the closest related issue, but documents the current no-sharing behavior rather than proposing an opt-in model. #69 concerns external services and is orthogonal.

If the execution model and API direction are acceptable, I can split the implementation into reviewable prerequisite, broker/lifecycle, Starlark expansion, and generic end-to-end test changes.

[mkosiba]

I think what we're building in https://github.com/hermeticbuild/rules_systemtest might of interest. It's pretty much all that you described above but also designed to work with remote execution. Right now it's just a design doc but we have a few more PRs in the pipeline.

[yaroslavvoloshchuk-codaio]

Thanks, this looks very close to what we need. The runner/coordinator/lease model preserves independent Bazel test caching and reporting while lazily sharing an expensive fixture, which is the core requirement behind this proposal.

We have a working local prototype in a private enterprise repository, so unfortunately we cannot link to it publicly. One potentially reusable part is a generic local-mode adapter capable of starting an existing rules_itest service graph—including one-shot itest_task setup—and exposing it as a shared fixture. It contains no PostgreSQL, Redis, or application-specific policy.

We would be interested in contributing that adapter and the associated lifecycle tests to rules_systemtest. Which upcoming PR or branch should we wait for or build upon so we do not duplicate work already in your pipeline?

Our primary acceptance cases are:

• fully cached tests start neither the coordinator nor the fixture;
• independently cached/reported test actions share one fixture;
• one fixture admits a fixed number of concurrent leases and queues excess requests rather than provisioning another fixture;
• wrapper cancellation, startup failure, coordinator shutdown, and idle timeout clean up the complete service graph.

Happy to align our contribution with the APIs and ownership boundaries you are already implementing.

[mkosiba]

Thanks @yaroslavvoloshchuk-codaio. I didn't really expect to get folks interested in rules_systemtest this early! We'll be pushing changes to the main branch, so that's the one to watch, but it's difficult for me to make predictions as to progress as I'll be taking the bulk of my vacation time over the next two months.