multipart uploads, tool-server routes, and the General Legal submission

appstore-meta/data/index.json

@@ -23,7 +23,7 @@
     {
       "id": "infra",
       "name": "Infrastructure",
-      "blurb": "Containers, microVMs, and deploys — the compute layer for agents.",
+      "blurb": "Containers, microVMs, and deploys \u2014 the compute layer for agents.",
       "hue": 30
     },
     {
@@ -41,13 +41,13 @@
     {
       "id": "work",
       "name": "Work & Research",
-      "blurb": "Put real-world work in motion and get real answers back — human hands on a task, and pricing research with your own customers.",
+      "blurb": "Put real-world work in motion and get real answers back \u2014 human hands on a task, and pricing research with your own customers.",
       "hue": 45
     },
     {
       "id": "comms",
       "name": "Communications",
-      "blurb": "Give an agent its own phone number or email inbox — voice, SMS/iMessage, email, and threaded conversations.",
+      "blurb": "Give an agent its own phone number or email inbox \u2014 voice, SMS/iMessage, email, and threaded conversations.",
       "hue": 315
     }
   ],
@@ -57,6 +57,7 @@
     "io.pilot.docker"
   ],
   "app_order": [
+    "io.pilot.generallegal",
     "io.pilot.dial",
     "io.pilot.deadsimple",
     "io.pilot.kinetic",
@@ -86,4 +87,4 @@
     "io.pilot.tldr",
     "io.pilot.insforge"
   ]
-}
+}

docs/MULTIPART-UPLOADS.md

@@ -0,0 +1,150 @@
+# Multipart uploads — sending a file to a partner API
+
+Some partner endpoints take a file, not JSON. General Legal's document endpoint is
+the shape: `POST /api/v1/documents` is `multipart/form-data` carrying a DOCX or PDF
+plus a few scalar fields, one of which names the matter the document belongs to.
+
+Declare `multipart:` on the route and the generator handles the rest.
+
+```yaml
+- name: gl.document_upload
+  summary: "Upload a document for attorney review (20 MiB max)."
+  duration: slow
+  timeout: 180s
+  http:
+    verb: POST
+    path: /api/v1/documents
+    multipart:
+      file_field: file        # the form field the partner reads (default "file")
+      max_bytes: 20971520     # match the partner's own limit
+  params:
+    blob_id: "string (required) — from gl.upload_begin"
+    deal_id: "string (optional) — attach to an existing matter"
+    context_for_legal: "string (optional)"
+```
+
+Every param except `blob_id` and the path placeholders becomes a form field, so the
+shape an agent reads in `<ns>.help` is the shape the partner receives.
+
+## Why the file is staged instead of sent inline
+
+The obvious design — base64 the file into the method's JSON payload — does not fit,
+and this is worth understanding before reaching for it anyway.
+
+Pilot IPC is JSON in, JSON out over a framed unix socket, and `ipc.MaxFrameSize`
+caps a single envelope at **1 MiB**; an oversize frame is refused and the connection
+dropped. base64 inflates by 4/3, so inline encoding tops out around **740 KiB** of
+real file. Partner limits are much larger (General Legal: 20 MiB). That is not a
+constant to tune — a 20 MiB document needs a 27 MiB envelope, 27× the frame.
+
+So the file travels in chunks that each fit, and the adapter reassembles it on disk
+under `$APP/blobs` before building one multipart body. The IPC layer never carries
+more than a chunk and no platform limit has to move.
+
+`internal/multipartkit` asserts this premise directly
+(`TestBase64InOneEnvelopeExceedsIPCFrame`) rather than leaving it as a comment, so
+if the frame size ever changes the trade-off gets re-examined instead of silently
+becoming wrong.
+
+## What an agent does
+
+Three generated methods appear automatically on any app with a multipart route —
+`<ns>.upload_begin`, `<ns>.upload_chunk`, `<ns>.upload_abort`. Do not author them.
+
+```bash
+# 1. Declare the file. sha256 is the integrity contract over the whole reassembly.
+pilotctl appstore call io.pilot.generallegal gl.upload_begin \
+  '{"file_name":"nda.docx","content_type":"application/vnd.openxmlformats-officedocument.wordprocessingml.document","total_bytes":3145728,"sha256":"<64 hex>"}'
+# -> {"blob_id":"…","max_chunk_bytes":524288,"next_seq":0}
+
+# 2. Push the bytes, in order, at most max_chunk_bytes of RAW file per call.
+pilotctl appstore call io.pilot.generallegal gl.upload_chunk \
+  '{"blob_id":"…","seq":0,"data_base64":"…"}'
+# -> {"received":524288,"next_seq":1,"complete":false}
+#    …the last chunk returns "complete":true once the sha256 verifies.
+
+# 3. Send it.
+pilotctl appstore call io.pilot.generallegal gl.document_upload \
+  '{"blob_id":"…","deal_id":"…","context_for_legal":"Standard mutual NDA."}'
+```
+
+The blob is **single-use**: once the partner has the bytes the adapter drops the
+local copy, so a replayed `blob_id` fails rather than uploading twice. Staged
+uploads that are never sent are reclaimed on a TTL, and an unfinished staging does
+not survive an adapter respawn (its rolling hash and chunk cursor die with the
+process, so resuming it could splice a gap into the middle of a document).
+
+## Rules the store enforces
+
+Chunked reassembly is a place to get integrity wrong, so the store is strict:
+
+| Rule | Why |
+|---|---|
+| `blob_id` is minted, never caller-supplied | a chosen id overwrites someone else's in-flight upload |
+| ids are 32 hex chars, validated | an id becomes a filename; nothing that could hold a separator is admitted |
+| chunks must be strictly sequential | tolerating a gap or a replay reassembles something the caller never sent |
+| declared size is a hard cap as bytes land | not just checked at the end |
+| sha256 must match at finalize | the integrity contract over the whole reassembly |
+| the file name is reduced to its base name | it is metadata for the form part, never a path |
+
+## Managed apps: two broker steps
+
+A multipart app behind the managed-key broker needs two things in its registry
+entry beyond the usual (see [`MANAGED-KEY.md`](MANAGED-KEY.md)):
+
+```json
+"forward_content_types": ["multipart/form-data"],
+"max_body_bytes": 25165824,
+"tenancy": {
+  "body_refs": {"deal_id": "deal"},
+  ...
+}
+```
+
+- **`forward_content_types`** — the broker forces `application/json` by default,
+  which strips the boundary and makes the body undecodable. This is an allow-list
+  rather than a passthrough on purpose: the request media type selects which parser
+  the partner runs, and letting a caller choose that freely is the same lever as the
+  duplicate-key parser differential tenancy already refuses.
+- **`max_body_bytes`** — the broker-wide default (8 MiB) is tuned for JSON calls and
+  is smaller than an upload partner's limit. Without this an upload the partner
+  would have accepted is refused with `413`.
+- **`tenancy.body_refs`** — an upload names the resource it acts on in a **form
+  field**, never in the path. An app that forwards multipart while declaring no
+  `body_refs` would ownership-check nothing on exactly the route that needs it most,
+  so the registry **fails the boot** rather than serve it.
+
+The broker parses the multipart form to check those refs, with the same stance as
+the JSON path: unparseable bodies, missing boundaries, and over-budget part counts
+all deny; a repeated **ref** field is refused as a parser differential (repeats of
+fields nobody checks are fine — they are legal multipart); and a ref arriving as a
+file part is denied outright, since file parts are not inspected and that shape
+would route a ref past the check.
+
+## Testing an upload app
+
+`docs/PUBLISHING-PLAYBOOK.md` Step 4 applies unchanged, plus:
+
+- Upload a file **larger than 1 MiB**. Anything smaller would fit an inline
+  encoding and proves nothing about the transport this design exists for.
+- Verify the partner received the bytes **unchanged** — compare sha256, not size.
+- If the app is managed, run it through a real broker, not just socket mode: the
+  Content-Type forwarding and the form-field ownership check only exist there.
+
+The reference tests are `internal/scaffold/zz_multipart_e2e_test.go` (generated
+adapter, real socket, real chunking) and `zz_multipart_broker_e2e_test.go` (the full
+adapter → broker → partner topology, including an upload into an unowned resource
+being refused).
+
+## Limits
+
+| | |
+|---|---|
+| chunk | 512 KiB raw per call (`max_chunk_bytes`, reported by `upload_begin`) |
+| staged upload | 24 MiB default, or `multipart.max_bytes` |
+| parts the broker will parse | 64 |
+| ref field value | 4 KiB |
+| staging TTL | 30 minutes |
+
+One file per request. A partner endpoint taking several files at once would need
+the route to carry several blob ids; nothing we ship requires it yet.

docs/PUBLISHING-PLAYBOOK.md

@@ -4,7 +4,8 @@ The end-to-end runbook for getting an app live on the Pilot app-store, for **eve
 backend and auth mode. It ties together the focused docs
 ([`PUBLISHING.md`](PUBLISHING.md), [`CLI-ADAPTER.md`](CLI-ADAPTER.md),
 [`NATIVE-APPS.md`](NATIVE-APPS.md), [`R2-ARTIFACT-REGISTRY.md`](R2-ARTIFACT-REGISTRY.md),
-[`MANAGED-KEY.md`](MANAGED-KEY.md), [`CI-AB-REPORT.md`](CI-AB-REPORT.md),
+[`MANAGED-KEY.md`](MANAGED-KEY.md), [`MULTIPART-UPLOADS.md`](MULTIPART-UPLOADS.md),
+[`CI-AB-REPORT.md`](CI-AB-REPORT.md),
 [`PRODUCT-DEMOS.md`](PRODUCT-DEMOS.md),
 [`UPDATING.md`](UPDATING.md), [`UPDATING-BUNDLES.md`](UPDATING-BUNDLES.md),
 [`APP-PUBLISHING-SPEC.md`](APP-PUBLISHING-SPEC.md)) into one
@@ -47,6 +48,14 @@ Two orthogonal choices. Get these right first; everything else follows.
 | a CLI tool **already on every host** | `cli` | methods → subprocess argv |
 | a CLI tool **not** on the host | `cli` + `assets[]` | the adapter fetches the binary from the R2 registry at install ([`NATIVE-APPS.md`](NATIVE-APPS.md)) |
 
+> **Edge case — an endpoint takes a file.** A `multipart/form-data` route cannot send
+> the file inside the JSON payload: `ipc.MaxFrameSize` caps an envelope at 1 MiB, so
+> base64 tops out near 740 KiB of real file. Declare `multipart:` on the route and the
+> generator adds the staging methods that push the bytes in frame-sized chunks
+> ([`MULTIPART-UPLOADS.md`](MULTIPART-UPLOADS.md)). Managed apps additionally need
+> `forward_content_types` and `tenancy.body_refs` in the broker registry, or uploads
+> are either undecodable on arrival or ownership-checked against nothing.
+
 **Auth** (`backend.auth`) — only relevant when the backend needs a key:
 
 | Key situation | `backend.auth` | What ships |

internal/broker/broker.go

@@ -247,14 +247,28 @@ func (b *Broker) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 	}
 
 	maxBody := b.MaxBody
+	if app.MaxBodyBytes > 0 {
+		maxBody = app.MaxBodyBytes
+	}
 	if app.Provision != nil && mpath == app.Provision.PushPath {
 		maxBody = app.Provision.ArtifactMaxBytes
 	}
-	body, err := io.ReadAll(io.LimitReader(r.Body, maxBody))
+	// Read ONE byte past the cap so an oversize body is detectable. Reading
+	// exactly maxBody silently truncates it instead, and a truncated body is far
+	// worse than a refused one: the signature no longer matches (401) or, for a
+	// multipart upload, the closing boundary is missing and the request dies as
+	// an opaque tenancy refusal — neither of which tells the caller their file
+	// was too big.
+	body, err := io.ReadAll(io.LimitReader(r.Body, maxBody+1))
 	if err != nil {
 		writeJSON(w, http.StatusBadRequest, map[string]string{"error": "read body"})
 		return
 	}
+	if int64(len(body)) > maxBody {
+		writeJSON(w, http.StatusRequestEntityTooLarge, map[string]string{
+			"error": fmt.Sprintf("request body exceeds %d bytes", maxBody)})
+		return
+	}
 
 	// 1. WHO is calling — verified, not asserted. Signed over the full request.
 	caller, sigErr := b.Verify.Verify(r.Header.Get, r.Method, r.URL.Path, body)
@@ -300,7 +314,7 @@ func (b *Broker) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 	//     indistinguishable, otherwise the broker is an oracle for enumerating
 	//     other tenants' resource ids.
 	if app.Tenancy != nil {
-		if _, ok := app.Tenancy.EnforceRequest(b.ownerStore(), appID, app.allowSegs, r.Method, mpath, r.URL.RawQuery, body, string(caller)); !ok {
+		if _, ok := app.Tenancy.EnforceRequest(b.ownerStore(), appID, app.allowSegs, r.Method, mpath, r.URL.RawQuery, r.Header.Get("Content-Type"), body, string(caller)); !ok {
 			writeJSON(w, http.StatusNotFound, map[string]string{"error": "not found"})
 			return
 		}
@@ -411,7 +425,7 @@ func (b *Broker) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 		b.internalError(w, http.StatusBadGateway, appID, "build upstream", err)
 		return
 	}
-	ureq.Header.Set("Content-Type", "application/json")
+	ureq.Header.Set("Content-Type", app.forwardContentType(r.Header.Get("Content-Type")))
 	app.injector.Inject(ureq, app.master)
 
 	resp, err := b.Client.Do(ureq)

internal/broker/meter_test.go

@@ -100,7 +100,13 @@ func TestMeter_PartialChargeKeepsRunning(t *testing.T) {
 
 func TestRunMeter_OneTick(t *testing.T) {
 	st := NewMemStore()
-	st.Provision("io.pilot.smol", "alice", "ip", 100, 0, 0, time.Unix(1, 0))
+	// One micro-dollar of credit against a 4 cpu / 8 GB machine (302_400
+	// micro-$/hour) is exhausted by the very first tick, whatever the tick
+	// interval. Seeding enough credit to survive ~1.2s of metering instead made
+	// this a race between that arithmetic and the 2s deadline below, with under
+	// a second of headroom — so any unrelated work added to the package could
+	// flake it, which is not something a test should be sensitive to.
+	st.Provision("io.pilot.smol", "alice", "ip", 1, 0, 0, time.Unix(1, 0))
 	fp := &fakeProvider{stopped: map[string]bool{},
 		machines: []MachineInfo{{ID: "m1", Owner: "alice", State: "started", Cpus: 4, MemoryMb: 8192}}}
 	app := meterApp(fp)

internal/broker/registry.go

@@ -3,6 +3,7 @@ package broker
 import (
 	"encoding/json"
 	"fmt"
+	"mime"
 	"os"
 	"sort"
 	"strings"
@@ -54,11 +55,37 @@ type AppEntry struct {
 	// otherwise just a self-minted keypair.
 	RequireAccessKey bool `json:"require_access_key"`
 
+	// ForwardContentTypes allow-lists request media types the broker may forward
+	// to the partner VERBATIM instead of forcing application/json.
+	//
+	// This exists for multipart uploads: a multipart/form-data body is
+	// meaningless without its boundary parameter, which lives in the
+	// Content-Type header. Forcing application/json (the historical behaviour)
+	// makes the partner reject every upload.
+	//
+	// It is an ALLOW-LIST, not a passthrough, and it is deliberately opt-in.
+	// The broker is the only thing between an untrusted caller and the master
+	// key, and the request media type selects which PARSER the partner runs.
+	// Letting a caller choose that freely is the same class of lever as the
+	// duplicate-key parser differential tenancy already refuses: a body the
+	// broker validated as one thing could be re-read by the partner as another.
+	// Unlisted types are forced to application/json exactly as before.
+	ForwardContentTypes []string `json:"forward_content_types,omitempty"`
+
+	// MaxBodyBytes overrides the broker-wide body cap for this app (0 = use it).
+	//
+	// The default is tuned for JSON calls and is smaller than what an upload
+	// partner accepts, so an app with a multipart route needs to raise it to at
+	// least the partner's own limit — otherwise the broker refuses (413) uploads
+	// the partner would have taken.
+	MaxBodyBytes int64 `json:"max_body_bytes,omitempty"`
+
 	master        string          // resolved from KeyEnv at load (managed: partner key; provisioned: cloud master, e.g. smk_)
 	injector      AuthInjector    // built from AuthHeader/Scheme
 	allowSet      map[string]bool // key = costKey(method, path); method "" = any method
 	allowPatterns []allowPattern  // templated allow entries ("{x}" matches any one segment)
 	allowSegs     [][]string      // every templated allow path (method-independent), for tenancy param extraction
+	fwdCT         map[string]bool // normalised ForwardContentTypes (media type only, lowercased)
 	breaker       *Breaker
 
 	creditSeed         int            // Credit.SeedCredits (0 ⇒ no budget)
@@ -376,6 +403,17 @@ func ParseRegistry(raw []byte, getenv func(string) string) (*Registry, error) {
 				a.allowSet[costKey(method, p)] = true
 			}
 		}
+		a.fwdCT = map[string]bool{}
+		for _, ct := range a.ForwardContentTypes {
+			mt, _, err := mime.ParseMediaType(strings.TrimSpace(ct))
+			if err != nil {
+				return nil, fmt.Errorf("registry: app %s: forward_content_types %q is not a media type: %w", a.ID, ct, err)
+			}
+			if mt == "application/json" {
+				continue // already the default; listing it is a no-op, not an error
+			}
+			a.fwdCT[strings.ToLower(mt)] = true
+		}
 		if a.CostField == "" {
 			a.CostField = "cost_cents"
 		}
@@ -518,3 +556,29 @@ func (r *Registry) AppsRequiringAccessKey() []string {
 	sort.Strings(out)
 	return out
 }
+
+// forwardContentType decides the Content-Type the broker sends upstream.
+//
+// Default (and the historical behaviour) is application/json: the caller does
+// not get to choose the partner's parser. An app may opt specific media types
+// in via forward_content_types, and only then is the caller's header forwarded
+// VERBATIM — parameters included, because a multipart body without its
+// boundary= parameter is undecodable.
+func (a *AppEntry) forwardContentType(incoming string) string {
+	if incoming == "" || len(a.fwdCT) == 0 {
+		return "application/json"
+	}
+	mt, _, err := mime.ParseMediaType(incoming)
+	if err != nil {
+		return "application/json" // unparseable → do not let it through
+	}
+	if !a.fwdCT[strings.ToLower(mt)] {
+		return "application/json"
+	}
+	return incoming
+}
+
+// forwardsMultipart reports whether this app may forward multipart bodies.
+func (a *AppEntry) forwardsMultipart() bool {
+	return a.fwdCT["multipart/form-data"]
+}