Make flux-network production-ready across TCP and Unix sockets

[Bronek]

Summary

Flux's HTTP implementation needs a stronger transport foundation before it can serve as a production-ready, resilient server or client. The current design is tied closely to TCP, owns its polling loop, performs zero-timeout polling, and pushes all HTTP work through callbacks. This limits composition, makes idle operation inefficient, and leaves important failure and resource-management cases without explicit policy.

The implementation should separate stream transport from application protocols while keeping both layers small, predictable, and efficient. It should also support Unix-domain sockets alongside TCP. A single poll-driven StreamNetwork should manage both, while services such as HttpService own protocol state and policy for one ConnectionGroup.

Target model

Concept	Responsibility
StreamNetwork	Owns or integrates with polling, manages stream connections, performs transport I/O, and schedules services.
ConnectionGroup	Groups listeners, outbound endpoints, and connections that share transport configuration and one owner. A group may use TCP or Unix-domain sockets.
Service	Implements a stateful application-layer server or client over one connection group. HttpService is the first implementation and supports both HTTP clients and servers.
Unclaimed group	Leaves framing and event handling to the caller for protocols that do not need a service abstraction.

This separation should allow one network and one poll to host multiple independently configured services. For example, public and operator HTTP APIs can use separate groups and HttpService instances, with different limits, deadlines, and scheduling priorities, while sharing the same transport machinery.

Required capabilities

Transport and composition

• Support both TCP and Unix-domain listeners and outbound connections through the same connection lifecycle and state machine.
• Handle Unix socket paths safely, including stale socket cleanup without removing a path that no longer belongs to the running process.
• Allow multiple connection groups and services to share one network and one poll.
• Support both an owned polling mode and integration into a caller-owned mio::Poll loop.
• Block efficiently until I/O or the next deadline instead of repeatedly polling with a zero timeout.
• Provide a waker for work submitted from outside the polling thread.
• Return the actual bound endpoint when listening on an ephemeral port.
• Expose the compatible mio API used by public external-polling interfaces so callers do not have to guess the required version.

Resilience and bounded resource use

• Enforce a configurable connection admission cap and report refused connections.
• Bound read, write, parsing, and event work performed in one iteration so one busy connection or service cannot monopolise the tile.
• Preserve existing write-backlog warning and hard-cap behaviour.
• Apply idle and outbound-request deadlines explicitly and report failures with actionable reasons.
• Linger when rejecting malformed or oversized requests where necessary to avoid resetting a connection while the peer is still uploading.
• Support write-side half-close and complete the remaining read-side lifecycle correctly.
• Make shutdown and close behaviour explicit at the group, service, and connection levels.

HTTP behaviour

• Preserve HTTP/1.0 and HTTP/1.1 parsing, chunked and EOF-delimited response bodies, Expect: 100-continue, reconnect behaviour, and body suppression for HEAD, 204, and 304 responses.
• Deliver application events through a pull-based API so the caller controls when and how much work it accepts.
• Allow an immediate response through a scoped responder and a deferred response through a stable request token.
• Ensure a request can be answered at most once; dropping the scoped responder must leave deferred response by token available.
• Keep request bytes available without copying while an event is being handled, while representing retained parse state with owned byte ranges.
• Drain consumed request bytes lazily at the next pull so both response paths follow one invariant and memory accounting remains correct.
• Accept the full HTTP status-code range from 100 through 599, using an empty reason phrase for unknown codes.

Configuration boundaries

Transport and protocol policy should remain separate:

Configuration	Examples
ConnectionGroupConfig	Framing, TCP options, socket buffer sizes, reconnect policy, backlog limits, connection caps, and telemetry identity.
HttpConfig	Header and body limits, idle timeout, rejection linger, and outbound request timeout.

TCP-specific controls belong in an explicit nested TCP-options value rather than being scattered through generic configuration. Unix-domain groups should not pay for or expose irrelevant TCP machinery.

Required extension paths

The implementation should allow the following extensions without significant refactoring.

Streaming HTTP responses

The initial HTTP API may continue to produce complete responses, but its ownership and state model must leave room for bounded streaming. A future streaming API should be able to begin a response, emit data only when writable capacity is available, and finish or abort it without buffering the complete body.

Streaming must include explicit backpressure, high and low watermarks, bounded memory and per-iteration work, and a send deadline. The service should continue to own HTTP framing; application code should not assemble wire-format chunks itself.

TLS

The transport boundary should permit optional TLS without exposing encryption state to HttpService. A future per-group TLS configuration should allow the network to perform the handshake before reporting a usable connection, decrypt before framing, encrypt after framing, and send close_notify before transport shutdown.

TLS is an extension point rather than a requirement for the initial implementation. Deploying behind a TLS-terminating proxy must remain a simple, zero-cost alternative for applications that do not need in-process TLS.

Other application protocols

StreamNetwork and ConnectionGroup should remain independent of HTTP. Protocols with materially different state machines—such as gRPC or WebSocket-based application protocols—should be implementable as sibling services over the same transport foundation rather than being forced through HttpService.

Engineering constraints

• Keep the hot path allocation-free after warm-up where practical, including event queues and complete-response writes.
• Preserve zero-copy request handling within the event lifetime.
• Keep dynamic dispatch on scheduling and control paths, not on the byte-processing path.
• Make ownership, ordering, deadlines, and work limits visible in the API rather than hiding them behind async runtimes or broad abstractions.
• Keep transport lifecycle, application protocol state, and caller orchestration in separate modules with explicit interfaces.
• Prefer one shared connection lifecycle over duplicated TCP and Unix-domain state machines.

Acceptance criteria

• TCP and Unix-domain endpoints use one coherent connection lifecycle.
• Multiple independently configured services can share one network and poll loop.
• Both network-owned and caller-owned polling are supported with documented ordering.
• Idle operation blocks until I/O, wake-up, or the nearest deadline.
• Connection counts, buffers, backlogs, parsing, and per-iteration work are bounded.
• HTTP requests support immediate and deferred responses without copying request bodies.
• Timeouts, admission refusal, malformed input, half-close, and shutdown have explicit observable behaviour.
• Transport and HTTP configuration remain separate and narrowly scoped.
• The design preserves a bounded-backpressure path for streamed HTTP responses.
• The transport boundary can accommodate optional TLS and non-HTTP services without redesigning the core network lifecycle.