per sandbox networking by BenTheElder · Pull Request #1689 · agent-substrate/substrate

Benjamin Elder (BenTheElder) · 2026-09-16T22:52:55Z

This is part of #1266 , opening now for discussion.

Stacked on #1682 which was slightly orthogonal.

This is loosely based on the mini proposal by John Howard (@howardjohn) as discussed in the community meeting, and feedback from Bowei Du (@bowei) Eitan Yarmush (@EItanya) Lior Lieberman (@LiorLieberman) Antonio Ojea (@aojea).

https://docs.google.com/document/d/1TycfQ3iiEpbI3rveMIj0S2PpPuLecb8I5R--yTpt9Ig/edit?resourcekey=0-kJbtEZ-KGzuL5eCjHDvBhg&tab=t.0#heading=h.ga9bfaf55ptk

Roughly:

Actor sandboxes each get their own netns.
- gVisor grabs all interfaces in the netns, and tap currently requires running something like slipr, so for now we do two netns + a veth when gVisor.
In the netns we directly intercept TCP => atunnel for general traffic.
In the netns we serve a trivial TCP+UDP DNS relay to the pod resolution.
- In the future we can insert policy here.
We consistently inject a modified resolv.conf instead of bind-mounting it (gVisor) across both runtimes.
Readiness probe dialing happens in the actor netns.

Every actor gets the same fixed guest IP as before, which is only visible to the actor.
All inbound/outbound traffic comes from atunnel / the DNS relay.
The actor no longer has any direct use of the pod interface, so we can begin to consider ateom using the network itself.
When we add the rest of multi-actor changes, this greatly simplifies thing.

Full multi-actor requires further changes, but this diff is already large (suggest reading commit by commit) and can stand-alone. I'll file more stacked changes when we've got consensus on this one.

Eitan Yarmush (EItanya)

🤖 AI-generated review.

Eitan Yarmush (EItanya)

🤖 mostly AI generated, reviewed by me :)

Sorry for the double review, the agent initially didn't understand that I wanted to append the first review comment to these...

Eitan Yarmush (EItanya) · 2026-09-17T11:22:06Z

+	if err := s.releaseSandboxNetwork(ctx); err != nil {
+		return fmt.Errorf("while releasing the previous sandbox network: %w", err)
+	}
+	session, err := ateomnet.ServeSandbox(ctx, ateomnet.SandboxNetworkConfig{


Preserve the documented behavior when no egress gateway is configured.

gVisor setup and microVM setup now always redirect TCP into atunnel, but prepareActorEgress still returns nil for an absent gateway and activateActorNetworking then skips activating egress. The inactive handler closes every intercepted connection. The API server still defaults the gateway address to empty, and the protocol contract explicitly promises direct egress in that case.

Could we preserve this supported mode, or explicitly require a gateway and update the configuration contract? At present an actor can start successfully with all external TCP connectivity broken. This was traced through both runtimes; I did not run a complete boot without a gateway.

After some side discussions: I think we want to change this to clearly state that egress is not supported without a gateway, but a gateway is not strictly required (perhaps your actor does not need egress).

If we really need it later we can add a no-capture mode, but it greatly simplifies things to always use atunnel, and atunnel will need an egress gateway for egress.

plus 1 to ben

Bowei Du (bowei) · 2026-09-17T21:55:32Z

+	if err := root.Mkdir("etc", 0o755); err != nil && !errors.Is(err, fs.ErrExist) {
+		return fmt.Errorf("creating %q: %w", filepath.Join(rootfs, "etc"), err)
+	}
+	if err := root.Remove("etc/resolv.conf"); err != nil && !errors.Is(err, fs.ErrNotExist) {


Prefer write out file and then use atomic rename to replace file contents.

We're doing this while nothing is reading it? I don't see the benefit. This only happens when the actor is not yet running.

haiyanmeng · 2026-09-18T13:46:09Z

+	// TODO: we hope gVisor can take that same single-namespace shape soon,
+	// once runsc can be given one interface rather than claiming every
+	// interface in the namespace it runs in.
+	GatewayNetNS netns.NsHandle


Naming this GatewayNetNS is very confusing, since this is for atunnel. How about this?

Current Proposed

SandboxNetwork.GatewayNetNS SandboxNetwork.AtunnelNetNS

SandboxGatewayNetNSName SandboxAtunnelNetNSName

setupGatewaySide setupAtunnelSide

gatewayVethName atunnelVethName

It's not just atunnel, so I'm not sure that's clearer. We're also doing the DNS listener, and possibly soon an "upward API" socket.

I do hope we can align gVisor soon so there's just one netns and no veth.

Point is that its not "Gateway" and using the word "gateway" is confusing? What is it beyond Atunnel?

Bowei Du (@bowei) , WDYT?

keep in mind we will not have both of these once gvisor can adapt. RuntimeNetNS will be the only one. GatewayNetNS is only if we have to split them across the veth, we drop this one later.

I don't think Atunnel vs Runtime is clearer personally but I'll rename to a consensus.

Lior Lieberman (LiorLieberman)

LGTM overall

Lior Lieberman (LiorLieberman) · 2026-09-18T21:18:40Z

+	// TODO: we hope gVisor can take that same single-namespace shape soon,
+	// once runsc can be given one interface rather than claiming every
+	// interface in the namespace it runs in.
+	GatewayNetNS netns.NsHandle


Point is that its not "Gateway" and using the word "gateway" is confusing? What is it beyond Atunnel?

Lior Lieberman (LiorLieberman) · 2026-09-18T21:38:43Z

+	if err := s.releaseSandboxNetwork(ctx); err != nil {
+		return fmt.Errorf("while releasing the previous sandbox network: %w", err)
+	}
+	session, err := ateomnet.ServeSandbox(ctx, ateomnet.SandboxNetworkConfig{


plus 1 to ben

Use fixed sandbox addresses in private namespaces. gVisor uses a veth pair across two namespaces; microVMs use a tap in one. Redirect TCP egress to atunnel and provide namespace-scoped listeners and dialers. The dialer takes TCP and UDP IP literals only, and restores the worker namespace once the socket exists so a pending connect does not pin a native thread.

Forward UDP and TCP DNS unchanged through the worker pod. Bound concurrency and connection lifetime, and close TCP connections on cancellation. DNS bypasses the actor egress policy.

A sandbox now lives in its own network namespace, so a readiness probe has to be dialed from there rather than from the worker's. Take the dialer as a parameter, and let a caller name the resolv.conf bound into the sandbox.

Wire both runtimes to isolated namespaces, namespace-aware ingress and readiness, and gateway DNS. Preserve fixed addresses across restore. Replace worker-wide routing and microVM TC mirroring with per-sandbox TCP redirects.

Benjamin Elder (BenTheElder) · 2026-09-19T00:02:26Z

One naming bikeshed outstanding. We can revisit that monday.

Resolved the others and did some more passes for bugs / minor nits.

Benjamin Elder (BenTheElder) added area/gvisor area/microVM area/network area/node labels Sep 16, 2026

Benjamin Elder (BenTheElder) force-pushed the per-sandbox-networking branch 8 times, most recently from 36d971a to 554332a Compare September 17, 2026 00:36

Eitan Yarmush (EItanya) reviewed Sep 17, 2026

View reviewed changes

Comment thread internal/ateomnet/sandbox.go Outdated

Eitan Yarmush (EItanya) reviewed Sep 17, 2026

View reviewed changes

Benjamin Elder (BenTheElder) force-pushed the per-sandbox-networking branch from 554332a to 096646d Compare September 17, 2026 21:34

Bowei Du (bowei) reviewed Sep 17, 2026

View reviewed changes

Comment thread internal/ateomnet/net.go

Bowei Du (bowei) added the kind/feature An enhancement / feature request or implementation label Sep 17, 2026

Benjamin Elder (BenTheElder) force-pushed the per-sandbox-networking branch from 096646d to 2c90bf9 Compare September 17, 2026 22:33

Bowei Du (bowei) reviewed Sep 17, 2026

View reviewed changes

Bowei Du (bowei) self-assigned this Sep 17, 2026

haiyanmeng reviewed Sep 17, 2026

View reviewed changes

Comment thread internal/proto/ateletpb/atelet.proto

Benjamin Elder (BenTheElder) force-pushed the per-sandbox-networking branch from 2c90bf9 to a04fc54 Compare September 17, 2026 23:34

Benjamin Elder (BenTheElder) mentioned this pull request Sep 17, 2026

Allow actors to self-suspend #483

Open

Benjamin Elder (BenTheElder) force-pushed the per-sandbox-networking branch 4 times, most recently from faf330a to 57f05ee Compare September 18, 2026 01:24

haiyanmeng reviewed Sep 18, 2026

View reviewed changes

Lior Lieberman (LiorLieberman) self-requested a review September 18, 2026 15:54

Benjamin Elder (BenTheElder) mentioned this pull request Sep 18, 2026

Proposal: Add a shim-managed Kata backend for Ateom checkpoint/restore #1738

Open

Benjamin Elder (BenTheElder) force-pushed the per-sandbox-networking branch from 57f05ee to b477fbe Compare September 18, 2026 18:05

Benjamin Elder (BenTheElder) mentioned this pull request Sep 18, 2026

ateapi: require an egress gateway address #1717

Open

2 tasks

Lior Lieberman (LiorLieberman) reviewed Sep 18, 2026

View reviewed changes

This was referenced Sep 18, 2026

Atunnel <> PEP contract documentation #1726

Merged

Document the egress traffic supported for GA #1749

Open

Benjamin Elder (BenTheElder) force-pushed the per-sandbox-networking branch from b477fbe to da4ba43 Compare September 18, 2026 23:29

Benjamin Elder (BenTheElder) added 4 commits September 18, 2026 16:44

atunnel: relay sandbox DNS through pod resolvers

7e2f7f0

Forward UDP and TCP DNS unchanged through the worker pod. Bound concurrency and connection lifetime, and close TCP connections on cancellation. DNS bypasses the actor egress policy.

readyz, ocispec: support sandbox-specific networking

e60919f

A sandbox now lives in its own network namespace, so a readiness probe has to be dialed from there rather than from the worker's. Take the dialer as a parameter, and let a caller name the resolv.conf bound into the sandbox.

ateom: use per-sandbox network namespaces

fc3897e

Wire both runtimes to isolated namespaces, namespace-aware ingress and readiness, and gateway DNS. Preserve fixed addresses across restore. Replace worker-wide routing and microVM TC mirroring with per-sandbox TCP redirects.

Benjamin Elder (BenTheElder) force-pushed the per-sandbox-networking branch from da4ba43 to fc3897e Compare September 18, 2026 23:45

Current	Proposed
`SandboxNetwork.GatewayNetNS`	`SandboxNetwork.AtunnelNetNS`
`SandboxGatewayNetNSName`	`SandboxAtunnelNetNSName`
`setupGatewaySide`	`setupAtunnelSide`
`gatewayVethName`	`atunnelVethName`

Conversation

Benjamin Elder (BenTheElder) commented Sep 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Eitan Yarmush (EItanya) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Eitan Yarmush (EItanya) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Lior Lieberman (LiorLieberman) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Benjamin Elder (BenTheElder) commented Sep 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Benjamin Elder (BenTheElder) commented Sep 16, 2026 •

edited

Loading