Skip to content

feat: pipe.broadcast cross-scope push + collect reads exposed#623

Merged
sini merged 4 commits into
mainfrom
feat/pipe-broadcast
Jun 26, 2026
Merged

feat: pipe.broadcast cross-scope push + collect reads exposed#623
sini merged 4 commits into
mainfrom
feat/pipe-broadcast

Conversation

@sini

@sini sini commented Jun 26, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds a cross-scope broadcast primitive to the pipe/quirk system, makes collect compose with expose, and fixes config-thunk resolution to honour the producing class + scope — then makes that resolution class-neutral so it no longer assumes NixOS.

pipe.broadcast pred — push primitive (dual of pipe.expose)

Where pipe.expose routes a value up to the parent applying source-side stages, pipe.broadcast routes the (transformed) value laterally to every other scope matching a receiver predicate, fleet-wide. Receivers read the pipe normally — no collect policy required.

den.policies.broadcast-dev = { host, user, ... }:
  [ (pipe.from "peer-dev" [
       (pipe.transform redact)                  # source-side, applied before send
       (pipe.broadcast ({ user, ... }: true))   # → every user scope, fleet-wide
     ]) ];
  • Receiver-only predicate (same signature as collectAll); selects receivers by their own context — kind, name, tags, host. Compound predicates like { host, user, ... }: host.name == "edge" target user scopes on a specific host.
  • Self-excluded (S≠R) — a source represents itself via its own base.
  • Implemented as collectAllBroadcast (Pass 1b) mirroring collectAllExposed; reuses findMatchingAll entity-kind filtering. bindsPipeLocally gains a broadcast clause so a pure receiver doesn't fall through to ancestor inheritance.

collect/collectAll read raw + exposed

collectTagged now reads resolveThunks(raw) ++ allExposed.${sid}, so a peer's collect sees data that children pipe.exposed up into a host. Fixes the expose-then-fleet-collect path. allExposed is Pass-1 data → no eval cycle.

Producer-class config-thunk resolution

A pipe config-thunk now resolves against the producing class module + scope, not the consuming one (the prior behaviour was a latent bug — a host-produced thunk consumed in a home resolved against the home config).

  • Cross-scope (broadcast/collect): the source's thunk resolves against the producer's class config.
  • Deferred (__configThunk): the marker carries the producer's class + name; the class-module wrapper resolves against the producing class's config and hands each thunk both config (producer class) and the owner config (the enclosing host), mirroring home-manager. The owner arg is requested from the module system only when a marker needs it, so standalone homes keep working.
  • isConfigDependent also detects owner-config-only thunks.

Class-neutral resolution (no host/osConfig hard-coding)

The host↔producer navigation is registry-driven, not baked into core fx — so den can describe a config-owner that isn't a NixOS host (e.g. terranix, nixidy) without core edits:

  • den.classes.<class>.parentPathname → path locating a member within the enclosing config-owner (the same route its content is delivered to); den.classes.<class>.parentArg — the module arg a nested member reaches the owner by. Both null for root classes that own a top-level config (nixos, darwin, …). The home-manager battery registers parentPath + parentArg = "osConfig" from the same value as its forward-delivery path.
  • producerConfigs/resolveEntry (assemble-pipes) and the class-module wrapper navigate via the registry, keyed by scopeEntityClass. isConfigDependent detects config or any registered parentArg. Zero host/osConfig/home-manager literals remain in core fx.
  • hostConfigs was already generic over instantiates (any entity with an intoAttr output), so the model already supported non-host owners; this removes the remaining NixOS-flavored literals. (Building actual terranix/nixidy entity/class defs is follow-on work this unblocks; the two soft or "nixos" class-default fallbacks are left intact.)

Tests

  • pipe-broadcast (10): all-to-all · user→remote-host · source-transform · predicate scoping (incl. { host, user } compound) · self-exclusion · targeted no-leak · pure-receiver binding · config-thunk from host / user source.
  • pipe-broadcast-isolation (4): cross-pipe-name · host-target-excludes-home (entity-kind via shared host context) · no-match predicate · broadcast↔collect boundary (a fleet collectAll counts each raw emit once, not the broadcast-amplified view).
  • pipe-config-scope (3): host→home, user→own-home, user→host producer-class resolution.
  • pipe-scope: the test-expose-then-fleet-collect witness now passes; the exposed-config-thunk witness reads the owner arg.

Full CI: 1042/1042. treefmt clean.

Commits

  1. pipe.broadcast cross-scope push + collect reads exposed
  2. producer-class config-thunk resolution + broadcast review fixes
  3. derive config-thunk host navigation from the class registry
  4. make config-thunk resolution class-neutral (no host/osConfig literals)

@sini sini requested a review from vic as a code owner June 26, 2026 02:42
@github-actions github-actions Bot added the allow-ci allow all CI integration tests label Jun 26, 2026
sini added 4 commits June 25, 2026 21:07
Add `pipe.broadcast pred` — a push routing primitive, the dual of
`pipe.expose`. A scope distributes its source-transformed pipe value to
every other scope whose context matches the receiver predicate,
fleet-wide; receivers read the pipe normally. Receiver-only predicate
(same signature as `collectAll`), self-excluded. Implemented as a
`collectAllBroadcast` pass (1b) mirroring `collectAllExposed`, reusing
`findMatchingAll` entity-kind filtering and `resolveThunks` cross-host
config resolution. `bindsPipeLocally` gains a broadcast clause so a pure
receiver does not fall through to ancestor inheritance.

Also make `collect`/`collectAll` read raw + exposed values at each
source scope rather than raw emits alone, so a peer's collect sees data
that children `pipe.expose`d up into a host. Fixes the
expose-then-fleet-collect path (witness test-expose-then-fleet-collect).

Tests: new pipe-broadcast suite (all-to-all, user->remote-host,
source-transform, predicate scoping incl. { host, user } compound,
self-exclusion) and pipe-broadcast-isolation suite (cross-pipe-name,
host-target-excludes-home entity-kind isolation, no-match predicate, and
the broadcast<->collect boundary: broadcast-injected values are not
re-collected). Full CI 1036/1036.
Resolve pipe config-thunks against the PRODUCING class module + scope,
not the consuming one. The prior behavior was a latent bug: a host-
produced thunk consumed in a home resolved against the home config.

- Cross-scope (broadcast/collect): resolveEntry resolves a source's
  config-thunk against the producer's class config — host -> nixos,
  user/home -> home-manager (hostConfigs.<host>.home-manager.users.<name>).
- Deferred (__configThunk): the marker carries the producer's class + name;
  the class-module wrapper resolves it against the producing class's
  config, handing each thunk both `config` (producer class) and `osConfig`
  (the enclosing host), mirroring home-manager. osConfig is requested only
  when a marker needs the host config, so standalone homes keep working.
- isConfigDependent now also detects osConfig-only thunks.

Review fixes: pin the bindsPipeLocally broadcast clause with a pure-
receiver test; extract a shared dedupEffectsByPolicy helper; document the
host-broadcaster raw-only boundary.

Tests: new pipe-config-scope suite (host->home, user->own-home,
user->host) plus broadcast config-thunk cases; the exposed-config-thunk
witness now reads osConfig. Full CI 1041/1041.
The producer-class config-thunk resolution hard-coded the home-manager
battery's delivery convention into core fx (the `home-manager.users.<name>`
path and the `"homeManager"` class literal). Replace that with a
registry-driven route.

- New `den.classes.<class>.hostPath` (name -> path): where a class's members
  nest inside the enclosing host config. The home-manager battery registers
  it from the same value as its forward delivery path (single source of
  truth); host-level classes (nixos, darwin) leave it null.
- assemble-pipes (producerConfigs / producerOf) and class-module
  (resolveMarkers) now navigate host<->producer config via the producer's
  registered hostPath and decide "is this class host-nested" from the
  registry, keyed by scopeEntityClass — no `home-manager`/`homeManager`
  literals in core fx. A new home-style class works by registering hostPath,
  no core edits.
- Marker carries `__producerClass` (was `__producerKind`); scopeEntityClass
  threaded into assemblePipes alongside scopeEntityKind.

The osConfig handed to deferred thunks remains den's own host-config link
(home.nix extraSpecialArgs.osConfig, derived from the host's intoAttr).
Behavior unchanged; full CI 1042/1042.
…g literals)

Generalize the producer-class config-thunk machinery so it no longer assumes
NixOS — den can describe a config-owner that isn't a host (e.g. terranix,
nixidy) without core edits.

- `den.classes.<class>.hostPath` -> `parentPath` (name -> path within the
  enclosing config-owner), and a new `parentArg` (the module arg by which a
  nested member reaches the owner config; home-manager registers "osConfig").
  Both null for root classes that own a top-level config.
- The home-manager battery registers parentPath + parentArg = "osConfig".
- assemble-pipes: `isConfigDependent` detects `config` or ANY registered
  parentArg (not the `osConfig` literal); `producerConfigs` returns
  { config, owner, parentArg } and navigates via the producer class's
  parentPath; resolveEntry hands a nested producer's thunk the owner config
  under its class's parentArg. Naming reframed host -> enclosing config-owner.
- class-module: the wrapper reads each class's parentPath/parentArg from the
  registry — a nested consumer fetches the owner via its own parentArg, and a
  thunk is handed the owner under its producer's parentArg. No osConfig literal.

`hostConfigs` was already generic over instantiates (any entity with an
intoAttr output), so the model already supported non-host owners; this removes
the remaining NixOS-flavored literals from the new config-thunk path. Behavior
unchanged; full CI 1042/1042.
@sini sini force-pushed the feat/pipe-broadcast branch from 68c8531 to a55556b Compare June 26, 2026 04:08

@drupol drupol left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not on a computer at the moment, can't read the code efficiently. That said, the feature looks nice!

@sini sini enabled auto-merge (squash) June 26, 2026 04:19
@sini sini merged commit 11866c1 into main Jun 26, 2026
41 of 43 checks passed
@sini sini deleted the feat/pipe-broadcast branch June 26, 2026 04:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

allow-ci allow all CI integration tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants