research: channel service sizing is fixed at first open, which caps inbound links at 8 and makes tap reopen-only

[tato123]

What prompted this

Building a large graph on the streamlib-doom demo hit three separate walls that all trace back to one fact: an iceoryx2 service's participant counts are fixed when the service is created and verified on every reopen, and our services are created by whatever link happens to open first.

• A destination cannot hold more than 8 inbound links. MAX_INBOUND_LINKS_PER_DESTINATION is the max_notifiers the destination-keyed notify service is created with. The console pane compositor hit this twice; the second time it changed the design, because an event feed had to poll HTTP rather than take a link that had nowhere to go.
• tap cannot attach to an output nothing consumes. Tap is a reopen of an existing service onto a reserved subscriber slot, by design — runtime/streamlib-engine/src/core/runtime/tap.rs says so in its module doc. An output port with no destinations has no service in existence, so there is nothing to reopen. Proving a processor was alive cost a four-minute timeout in a setup script until it was special-cased to use that processor's own HTTP endpoint instead.
• Connection order decides the sizing, so which links are possible later depends on what attached first.

The owner's expectation, stated plainly, is that a tap should work on any channel at any time.

Upstream does not solve it

We are on iceoryx2 0.8.1; 0.9.3 is current. The 0.9.0 release notes contain no change to max_subscribers, max_notifiers, service resizing, dynamic participant counts, or reopening with a different configuration. The "resizable data segments" work in that release is payload memory, not participant slots, and will read like the answer without being it.

Separately: the 0.8.1 → 0.9.x bump looks unusually cheap for us. Of its ten breaking changes we use none — UnableToDeliverStrategy, DegradationCallback, service_id, NodeId, creation_timeout and iceoryx2_tunnel have zero occurrences in our tree; only allocation_strategy appears, twice. Two deadlock fixes and a chunk-leak fix ride along. That is hygiene, not a fix for this, and belongs in its own ticket.

The candidate answer, and why it needs a decision first

Open a source's output channel at setup with the full reserved sizing, rather than sizing it from the destinations that exist when the first link compiles. Tap then works on any port at any time, and the cap stops depending on connection order.

This is a change to how the engine sizes every channel, so it is docs/plan/ARCHITECTURE.md's call and not a ticket's. Filing as research to establish the facts a decision needs.

What to establish

1. What a reserved slot actually costs. Measure it, do not reason about it. open_or_create_service asks for max_publishers(1), max_subscribers(N) and subscriber_max_buffer_size(RING_DEPTH = 16). Establish the shared-memory delta per additional subscriber slot and per notifier slot, and whether the publisher's data segment grows with the subscriber count. Our bags are [u8] slices carrying surface descriptors rather than pixels, so the payload is small, but that should be measured rather than assumed.
2. Whether pre-sizing every channel at the maximum is affordable for a graph the size of the doom console — 25 to 29 processors, roughly 38 links — and for a plausibly larger one.
3. Whether the cap should rise at the same time. 8 predates how much live splicing the graph now does. If a slot is cheap, the question is what number is right, not whether to keep 8.
4. What breaks if a service is created before any link exists. In particular whether a later link's open still validates, and what happens to a source whose output nobody ever consumes.
5. Whether iceoryx2 offers any escape hatch we are not using — a second service, a re-create path that does not disturb attached participants, or a different messaging pattern for the notify service.

Not in scope

Implementation. This ticket ends in a recommendation with numbers behind it, for an /align decision.

Found while building github.com/tatolab/streamlib-doom.

[tato123]

A finding from the iceoryx2 0.9.3 upgrade (#2231 / PR #2244) that bears on questions 4 and 5.

0.9.3 adds global.service.cleanup_dead_nodes_on_open, and it defaults to true. 0.8.1 cleaned up dead nodes only when a node was created or destroyed.

• When a slot is freed: opening a service now scans for dead nodes and reclaims their resources. So a crashed participant's subscriber or notifier slot is freed the next time anyone opens that service, not only when a node is next created or destroyed. For a helper that dies mid-run, the slot it held can come back as soon as a relink or a tap reopens the channel.
• Cost: the scan blocks for up to creation_timeout (500 ms by default) when a dead node exists. A live service with no dead participants pays nothing.
• Capacity: participant counts are still fixed when a service is created and checked on every reopen. Nothing about max_subscribers or max_notifiers changed, so this doesn't change the recommendation this ticket is after.

Also relevant to question 5: open_or_create now retries ServiceInCorruptedState and IsMarkedForDestruction internally, up to 5 times, before reporting SystemInFlux. A service caught mid-teardown by a reopen is less likely to fail the open than it was on 0.8.1.

These are read from the 0.9.3 source, not measured. Treat them as claims to verify, to the best of our current knowledge.