Atunnel <> PEP contract documentation

docs/glossary.md

@@ -64,7 +64,8 @@ for etcd.
 
 - **ateom**: the coordinator that runs inside each worker pod and drives the
   sandbox runtime on behalf of atelet. This decouples the physical pod
-  lifecycle from the sandboxed agent process.
+  lifecycle from the sandboxed agent process. It embeds a networking service
+  called `atunnel` that handles network traffic for the sandboxed Actor.
 
 - **atenet**: the networking stack. Its router resumes suspended Actors on
   demand and routes traffic to the right worker pod.

docs/network-egress.md

@@ -0,0 +1,67 @@
+# Network Egress Contract
+
+Last updated: 09/18/2026
+
+## Overview
+
+Agent Substrate enforces that all outbound traffic from an actor, except traffic sent to TCP or UDP destination port 53, MUST go through an egress policy enforcement point (PEP). That PEP can (optionally) inspect, filter, or redirect the traffic according to substrate-native egress policies. This document describes the contract between substrate and egress PEPs.
+
+## Transport
+
+At present, actor TCP traffic exits the sandbox through [`atunnel`](./glossary.md#components) and is forwarded on to the PEP, except traffic to destination port 53; traffic using TCP or UDP destination port 53 is allowed to bypass the PEP for DNS, while all other protocols are blocked. `atunnel` is configured with the PEP's network address and port at startup and uses that address to open an mTLS HTTP/1.1 CONNECT tunnel to the PEP. The CONNECT request target and `Host` header are both set to the original destination IP and port that `atunnel` derived from the actor's TCP connection. Actor networking is currently IPv4-only.
+
+## Trust Boundary
+
+In substrate, we consider everything originating from the actor to be untrusted. The egress PEP MUST NOT trust any decision or attestation coming from the actor itself; it MUST instead trust only information carried through substrate-controlled channels (e.g. the actor certificate and the CONNECT request produced by atunnel). For example, the PEP MUST NOT treat an actor-provided HTTP hostname or TLS SNI as proof that the original destination has that name. The PEP MAY use the hostname as a policy input, but when authorization is based on that hostname, it MUST route the traffic to the authorized hostname rather than use the hostname to authorize an arbitrary IP address from the CONNECT request.
+
+## Server-side TLS
+
+The egress PEP MUST serve TLS using a certificate that is valid for the configured PEP hostname and chains to a CA trusted by `atunnel`. `atunnel` uses the configured PEP hostname as the TLS server name and MUST reject the connection if it cannot verify the PEP's certificate.
+
+In the default Kubernetes deployment, the PEP obtains its certificate and private key through a projected `podCertificate` volume using substrate's Service DNS signer (`servicedns.podcert.ate.dev/identity`). The signer derives the certificate's DNS SANs from the Services that select the PEP Pod, so the PEP MUST be selected by a Service and the hostname configured in `atunnel` MUST match one of those Service DNS names. For the default PEP, that name is `atenet-egress.ate-system.svc`. The projected `credential-bundle.pem` contains the serving certificate chain and private key, while the `ClusterTrustBundle` for the same signer supplies `atunnel` with the corresponding trust anchors.
+
+## Client Authorization
+
+`atunnel` MUST present an actor-specific client certificate when connecting to the PEP. At actor activation, `atunnel` generates a private key and requests a short-lived certificate from substrate through the node-local atelet. The private key remains in `atunnel`. The certificate identifies the actor by its atespace, name, and UID and is scoped to the `atunnel` purpose.
+
+The PEP MUST require and verify the client certificate before accepting a CONNECT request. It MUST verify the certificate chain and lifetime against the Actor Identity CA, require the client authentication usage, and require exactly one valid `ActorIdentity` extension whose atespace, actor name, actor UID, and purpose are present. The current extension OID is `1.3.6.1.4.1.11129.2.12.2`; it is allocated under Google's enterprise number and will change after the CNCF donation is complete. The purpose MUST be `atunnel`, and the certificate's actor URI SAN MUST identify the same actor as the extension. The PEP MUST then confirm with ate-api-server that the actor still exists, that its current UID matches the certificate, and that it is running. The PEP MUST reject the CONNECT request if any of these checks fail.
+
+```text
+CURRENT CONNECT EGRESS PATH (one tunnel per actor TCP connection except destination port 53)
+
+  Actor sandbox                 Worker pod / ateom                  Egress PEP                     Authorized upstream
+  +------------------+          +--------------------------+        +-------------------------+    +------------------+
+  | Actor process    |          | nftables                 |        | mTLS listener           |    | Target selected  |
+  | (untrusted)      |          | TCP REDIRECT             |        | + CONNECT terminator    |    | under policy     |
+  +--------+---------+          +------------+-------------+        +------------+------------+    +--------+---------+
+           | connect(IP:port)                |                                   |                          |
+           +-------------------------------->|                                   |                          |
+                                             | SO_ORIGINAL_DST                   |                          |
+                                             v                                   |                          |
+                                +------------+-------------+                     |                          |
+                                | atunnel                  |                     |                          |
+                                | (trusted tunnel client)  |                     |                          |
+                                +------------+-------------+                     |                          |
+                                             |                                   |                          |
+                                             | TLS 1.2+ mutual authentication    |                          |
+                                             | actor-specific client certificate |                          |
+                                             +---------------------------------->| verify certificate and   |
+                                             |                                   | authorize current actor  |
+                                             |                                   |                          |
+                                             | HTTP/1.1 CONNECT IP:port          |                          |
+                                             | Host: IP:port                     |                          |
+                                             +---------------------------------->| validate destination;    |
+                                             |                                   | evaluate egress policy   |
+                                             |                                   |                          |
+                                             | HTTP/1.1 2xx                      |                          |
+                                             |<----------------------------------+                          |
+                                             |                                   | CIDR/all policy:         |
+                                             |                                   | dial now;                |
+                                             |                                   | hostname policy:         |
+                                             |                                   | inspect inner request    |
+                                             |                                   | before dialing           |
+                                             |                                   +------------------------->|
+                                             |                                   |                          |
+           |<================ raw bidirectional bytes inside CONNECT ===========>|<========================>|
+           |             (PEP may pass through, inspect, redirect, or MITM according to policy)             |
+```

internal/ateomnet/net.go

@@ -44,6 +44,7 @@ const (
 	ActorVethGateway  = "169.254.17.1"
 	ActorVethIP       = "169.254.17.2"
 	ActorNftTableName = "ateom_actor"
+	dnsPort           = 53
 
 	// ActorVethSubnet is the point-to-point /30 the actor veth lives on.
 	ActorVethSubnet = "169.254.17.0/30"
@@ -212,10 +213,12 @@ func InstallActorNftablesRules(egressPort uint16) error {
 	//
 	// The rules do three things:
 	//
-	//   * prerouting: redirect new actor TCP connections to atunnel's local
-	//     listener. REDIRECT preserves SO_ORIGINAL_DST for the CONNECT authority.
+	//   * prerouting: redirect new actor TCP connections, other than traffic to
+	//     destination port 53, to atunnel's local listener. REDIRECT preserves
+	//     SO_ORIGINAL_DST for the CONNECT authority.
 	//   * postrouting: masquerade traffic not handled by the TCP tunnel, notably
-	//     DNS over UDP, so hostname resolution continues to work.
+	//     traffic to TCP or UDP destination port 53, so hostname resolution
+	//     continues to work.
 	//   * forward: drop actor UDP egress to any port but DNS, and accept the rest
 	//     of the packets forwarded between the actor veth and pod eth0.
 	if err := RemoveActorNftablesRules(); err != nil {
@@ -334,14 +337,28 @@ func l4ProtocolEqual(proto byte) []expr.Any {
 }
 
 // ActorEgressRedirectRule returns the prerouting rule that redirects actor TCP
-// egress to the local atunnel egress listener on port, or nil when port is zero
-// (tunneled egress disabled, so actor egress stays on the masquerade path).
+// egress, except traffic to [dnsPort], to the local atunnel egress listener on
+// port, or nil when port is zero (tunneled egress disabled, so actor egress
+// stays on the masquerade path).
 func ActorEgressRedirectRule(table *nftables.Table, chain *nftables.Chain, port uint16) *nftables.Rule {
 	if port == 0 {
 		return nil
 	}
 	exprs := append(IPSourceEqual(ActorVethIP), l4ProtocolEqual(unix.IPPROTO_TCP)...)
 	exprs = append(exprs,
+		// Traffic to destination port 53 bypasses atunnel and follows the direct
+		// masquerade path.
+		&expr.Payload{
+			DestRegister: 1,
+			Base:         expr.PayloadBaseTransportHeader,
+			Offset:       2,
+			Len:          2,
+		},
+		&expr.Cmp{
+			Op:       expr.CmpOpNeq,
+			Register: 1,
+			Data:     binaryutil.BigEndian.PutUint16(dnsPort),
+		},
 		&expr.Immediate{
 			Register: 1,
 			Data:     binaryutil.BigEndian.PutUint16(port),
@@ -358,9 +375,6 @@ func ActorEgressRedirectRule(table *nftables.Table, chain *nftables.Chain, port
 // up as a rising counter in `nft list table ip ateom_actor` rather than as an
 // unexplained timeout.
 func actorNonDNSUDPDropRule(table *nftables.Table, chain *nftables.Chain) *nftables.Rule {
-	// dnsPort is the only destination port on which actor UDP egress is forwarded.
-	const dnsPort = 53
-
 	exprs := append(IPSourceEqual(ActorVethIP), l4ProtocolEqual(unix.IPPROTO_UDP)...)
 	exprs = append(exprs,
 		// Destination port, at offset 2 of the UDP header.

internal/ateomnet/net_linux_test.go

@@ -17,6 +17,7 @@
 package ateomnet
 
 import (
+	"bytes"
 	"context"
 	"errors"
 	"net"
@@ -26,6 +27,7 @@ import (
 
 	"github.com/agent-substrate/substrate/internal/roottest"
 	"github.com/google/nftables"
+	"github.com/google/nftables/binaryutil"
 	"github.com/google/nftables/expr"
 	"github.com/vishvananda/netlink"
 	"github.com/vishvananda/netns"
@@ -307,10 +309,41 @@ func sendUDP(t *testing.T, addr string) {
 	}
 }
 
+func TestActorEgressRedirectRuleExcludesDNS(t *testing.T) {
+	table := &nftables.Table{}
+	chain := &nftables.Chain{Table: table}
+
+	if rule := ActorEgressRedirectRule(table, chain, 0); rule != nil {
+		t.Fatal("ActorEgressRedirectRule returned a rule when tunneled egress is disabled")
+	}
+
+	rule := ActorEgressRedirectRule(table, chain, 15001)
+	if rule == nil {
+		t.Fatal("ActorEgressRedirectRule returned nil when tunneled egress is enabled")
+	}
+	if len(rule.Exprs) != 8 {
+		t.Fatalf("redirect rule has %d expressions, want 8", len(rule.Exprs))
+	}
+	payload, ok := rule.Exprs[4].(*expr.Payload)
+	if !ok {
+		t.Fatalf("redirect expression 4 is %T, want *expr.Payload", rule.Exprs[4])
+	}
+	if payload.Base != expr.PayloadBaseTransportHeader || payload.Offset != 2 || payload.Len != 2 {
+		t.Errorf("redirect destination-port payload = %+v, want transport-header offset 2 length 2", payload)
+	}
+	cmp, ok := rule.Exprs[5].(*expr.Cmp)
+	if !ok {
+		t.Fatalf("redirect expression 5 is %T, want *expr.Cmp", rule.Exprs[5])
+	}
+	if cmp.Op != expr.CmpOpNeq || !bytes.Equal(cmp.Data, binaryutil.BigEndian.PutUint16(dnsPort)) {
+		t.Errorf("redirect destination-port comparison = %+v, want destination port != %d", cmp, dnsPort)
+	}
+}
+
 // TestActorNonDNSUDPIsDropped covers the forward-chain rule behaviorally: only
-// TCP is redirected into atunnel, so UDP on any port but 53 must not reach the
-// masquerade, and DNS must still get through or the sandbox cannot resolve
-// anything.
+// TCP not destined for port 53 is redirected into atunnel, so UDP on any port
+// but 53 must not reach the masquerade, and DNS must still get through or the
+// sandbox cannot resolve anything.
 func TestActorNonDNSUDPIsDropped(t *testing.T) {
 	roottest.Require(t, "creating network namespaces, veth pairs, and nftables rules")
 	ctx := context.Background()