Roadmap: one-day multimodal chat app — assembled vertical + gates

[drewstone]

Definition of done: create-agent-app → fill agent.config.ts + profile → wrangler deploy = a working multimodal chat product, in one day, with CI that makes the known incident classes unshippable. Everything below is derived from a measured audit (2026-07-15) of agent-app, agent-dev-container, legal-agent, and tax-agent — each item cites the gap it closes. Nothing speculative.

Status update (2026-07-23): Phase 1 shipped via the #191–#220 series and is adopted by legal (legal-agent#263/#264), tax (tax-agent#378/#379), and gtm (gtm-agent#601–#603) — the migration acceptance test below is satisfied. Remaining shared-code work and fleet-wide adoption are tracked in the consolidation epic #228; the Phase-2 gates + nightly live-e2e are implemented by #226 — work them there, not here. This issue stays open only for the /preflight bin wiring and Phase 3.

Ground truth today: every primitive exists and is production-proven (client streamChatTurn w/ resume, D1 turn-buffer, PromptInputPart[] dispatch as of #187, interactions contract as of #176, teams/access/seat-billing, harness-native images on all 3 product harnesses). What's missing is the assembly: legal wrote 2,190 lines of chat orchestration, tax wrote 1,058 — differently (legal skips the turn-buffer, tax skips onInteraction, both fork the question-card stack ~1,000 lines each, neither delivers an image part). The scaffold emits a JSON tool-loop worker, not a chat app.

Phase 1 — the assembled chat vertical (kills ~3,500–4,500 lines of per-app glue)

• createChatTurnRoutes() (M): one exported factory assembling body-parse → sandbox producer → turn-buffer tap → SSE Response, plus replay + interaction-answer endpoints. Consumers: replace legal api.chat.ts (895 lines) and tax chat-turn.ts (1,058 lines) with it — the migration is the acceptance test. — shipped in feat(chat-routes): assembled chat-turn vertical + multimodal upload→parts (#188 Phase 1) #194 (+ product seams feat(chat-routes): product turn-lifecycle seams (heartbeat, beforeTurn, lifecycle, context-gate, turn-lock) #200, hardening feat(sandbox): optional billingOwnerId on create + keyless model metadata #204–feat(chat-routes): let authorize suppress the user-row insert for a dispatched turn #207); legal-agent#263 and tax-agent#378 replaced their orchestration with it
• /chat-store (S–M): threads+messages drizzle table factory (same injection pattern as createTeamTables) + thread/message CRUD incl. batched + bulk delete. Legal/tax each hand-roll incompatible schemas today; tax re-declares the framework's own turn_events DDL. — shipped in feat(chat-store): shared chat thread/message persistence (issue #188 Phase 1) #191
• Multimodal middle (M): upload helper (inline ≤~750 KB, else fs.uploadData + path-ref part — the 1 MiB gateway cap makes this a required two-step), file→PromptInputPart assembly, and ChatComposer emitting parts from onSend (today it emits onSend(string) and a disjoint FileList; both products drop attachments on the floor). — upload→parts shipped in feat(chat-routes): assembled chat-turn vertical + multimodal upload→parts (#188 Phase 1) #194, durable object-store in feat(object-store): portable signed-URL object store for durable attachments #211/fix(web-react,object-store): guard tool-detail render + reject interior slash in key segments #217, composer attach UX in sandbox-ui 0.84+; client-half completeness across products continues in [FEAT] Uplift the attachment client half — structured dispatch + eager file-part promotion #224 / [EPIC] One chat vertical across the fleet — consolidate agent-app, adopt in every app #228
• Interaction UI + answer forwarding (M): shared question-card component + sidecar-answer server helper. Legal (1,059 lines) and tax (844 lines) each maintain a forked copy of the contract agent-app already ships. — shipped in feat(interactions): human-in-the-loop ask channel — contract subpath, answer route, question/plan cards #192; forked copies deleted in legal-agent#264, tax-agent#379, gtm-agent#602
• createAppAuth() (S): better-auth config factory (drizzle adapter, providers, cookie prefix, SSO wiring). Both apps hand-roll; tax re-implements cookie signing. — shipped and adopted by all three migrated apps (e.g. gtm-agent#602)

Phase 2 — provable (works, tests, ships)

• Five S-cost gates in agent-app (each an assertion at an existing choke point): provision-payload size (<240 KiB, per-section breakdown) in ensureWorkspaceSandbox; composed-prompt byte budget (default 40 KB) in profile composition; onErrorEvent fail-loud when unhandled (silent-empty-answer class); generalize the /catalog import-graph walker over a declared browser-safe manifest (all ~12 browser subpaths); env-var size check (E2BIG class) piggybacked on the payload gate. — implemented by [INFRA] Ship the Phase-2 CI gates + nightly live-e2e from the #188 roadmap #226
• Nightly live-e2e workflow in agent-dev-container (S): one workflow flipping the existing RUN_*_LIVE_E2E flags — the test content exists; the flags appear in zero of 30 workflows. This is the class behind every "green CI, dead in prod" event this month.
• Chat-app scaffold variant (M once Phase 1 lands): create-agent-app --chat assembles Phase 1; the existing scaffold test (which already typechecks generated apps against real dist) extends to run one fake-backend turn + the prompt/theme budget checks. — shipped via feat(sandbox+chat-routes): durable interactive turn — detach + running discovery #209/feat(create-agent-app): scaffold chat turns durable by default (detach) #210 — create-agent-app --chat exists, durable by default, with a scaffold test that runs the generated app's own fake-backend turn e2e; the prompt/theme budget-check extension rides with [INFRA] Ship the Phase-2 CI gates + nightly live-e2e from the #188 roadmap #226 — implemented by [INFRA] Ship the Phase-2 CI gates + nightly live-e2e from the #188 roadmap #226
• /preflight secret-liveness bin (M): declared probes (router chat ping, sandbox auth ping, platform HEAD) run as a deploy step — the one class that can't be a CI gate (4 dead secrets in one day last incident). — library half shipped (/preflight subpath: runPreflight + router/sandbox/HEAD probe builders); the agent-app-preflight bin entry and deploy-step wiring are still missing — this is the live remainder

Phase 3 — platform closers (agent-dev-container)

• #3539 E2BIG (S–M): OPENCODE_CONFIG_CONTENT is still an unbounded env var (sdk-provider-opencode/src/server.ts:357); write config to the on-disk opencode.jsonc it already points at. #3621 covered file mounts only.
• Attachment workflow docs + opencode live image test (S): the upload-then-path two-step is mechanically solid and documented nowhere; opencode's native image path has no live proof that the model sees the image.
• Codex images × interactions — blocked upstream (#3586); until then apps must check runtime getCapabilities().multimodal (registry imageInput: true is not the whole truth on app-server configs). Document the check in the SDK README.

Deliberately excluded (not "necessary to ship")

User-facing AgentProfile customization UI (the only true L — seams exist via userSkills/ProfileOverlay, zero producers; build when a product needs it), additional harness attachment support beyond the 3 product harnesses, balance-manager adoption (S, cleanup not blocker).

Sequencing / effort

Phase 1 items are independent → parallelizable; ~1 focused engineer-week total, or 2–3 orchestrated days. Phase 2 gates are hours each. Phase 3 is Lin's lane and mostly S. After Phase 1 + scaffold variant, the one-day claim is real and the scaffold test enforces it forever.

[drewstone]

Hard evidence for Phase 1 priority — the fleet fixes every chat-shell bug ~3×, ~10 days apart, by different people, because the shell lives in N copies:

Bug class	gtm-agent (Lin)	tax-agent	legal-agent
Optimistic user message wiped on failed turn	#525, Jul 7	#374, Jul 14	#261, Jul 14
Stream errors invisible / no retry affordance	#479 + #492, Jul 3	#374, Jul 14	#261, Jul 14
Interaction-question dedupe	#489, Jul 3	— (pending traffic)	ships in the forked card stack
Duplicate in-flight turn guard	#487, Jul 3	—	—
Seat-paywall double-render	#478, Jul 3	shipped separately	shipped separately

None of these fixes propagated because there is no shared component to fix. gtm-agent is effectively the lead implementation of the chat shell — createChatTurnRoutes() + the shared interaction/question UI should be lifted FROM gtm's current state (it has absorbed the most fixes), not written fresh.

[drewstone]

Fleet Migration Brief — 3 apps onto @tangle-network/agent-app assembled chat vertical

Migrating gtm-agent, legal-agent, and tax-agent (apps/web) off hand-rolled chat orchestration onto the shared createChatTurnRoutes + createSandboxChatProducer + chat-store + app-auth + /web-react cards + /interactions modules.

1. Fleet net-LOC

App	Deletions	Additions	Net
gtm-agent	1,895 (12 files)	283	−1,612
legal-agent	1,689 (11 files)	330	−1,359
tax-agent (apps/web)	1,902 (16 files)	460	−1,442
Fleet total	5,486	1,073	−4,413

2. Recommended cross-app order

Blocking prereq (all apps): land PR #188 (createChatTurnRoutes, branch 46c2054), cut an agent-app release exposing ./chat-store ./app-auth ./interactions ./chat-routes, bump each app's dep + lockfile. Today all three are on 0.43.23, which exports only ./sandbox + ./stream. No turn-route step starts until this ships. Auth / schema / store steps depend only on origin/main and can start now.

1. gtm-agent FIRST — proves the most. Its chat-interactions.ts, question/plan cards, and interactions route ARE the byte-source the shared modules were forked from (feat(interactions): human-in-the-loop ask channel — contract subpath, answer route, question/plan cards #192). Migrating it validates the shared components against their own origin with near-zero behavior surprise; a clean gtm run de-risks the other two.
2. legal-agent SECOND — cleanest additive. No at-rest encryption, single sandbox producer lane, columns already byte-match the factory. Gains durable turn-buffer/replay it never had. Lower blast radius than tax.
3. tax-agent LAST — riskiest. AES-256-GCM at-rest encryption (dedup correctness hazard), dual producer lanes (sandbox + router/openai-compat), a full chat_messages→message backfill + DROP TABLE. Do it once the shared path is proven twice.

Within each app: auth swap standalone → schema/store adoption → turn-route assembly LAST (highest risk, depends on the prior two).

3. GAP punch-list — what the shared components lack (owning subpath)

#	Gap	Owning subpath	Hit by
1	Encryption at rest — message.content is plaintext text col, no field crypto. Adapter MUST decrypt in listMessages or non-deterministic ciphertext breaks resolveChatTurn retry-dedup → double-insert. Highest risk.	/chat-store (+ /crypto adapter product-side)	tax
2	Dead-model auto-switch on 503 — MODEL_DOWN_PATTERN /503|upstream|unavailable/ + switchToModel(DEFAULT) absent; onErrorEvent only surfaces the string (#261/#375). Keep chat-turn-error.ts as product glue.	/web-react (chat-stream)	legal, tax
3	One-row-per-turn error latch — onErrorEvent fires per error line, no dedupe; a flaky turn renders duplicate error rows (#261/#376). Keep client alreadyErrored/turnErrorRef, reset on send.	/web-react	legal, tax
4	#374 refetch-suppression latch — suppressing the onComplete history-refetch that would wipe a failed turn is product history mgmt, not shared.	/web-react (product glue)	tax
5	API-key + sandbox-internal-secret auth — createAppAuth gives session/getSession + guards only; SHA-256 API-key + constant-time X-Sandbox-Internal bypass stay product. Refactor only the session branch.	/app-auth	legal (tax similar)
6	declineInteraction seam — non-renderable ask kinds auto-decline ONLY if the product supplies declineInteraction; without it runs block until broker timeout.	/chat-routes (sandbox-producer)	legal
7	#495 composer-open-during-questions — contract helpers exist (fieldAcceptsFreeText/composerAnswerDeliveries) but no shared composer consumes them (zero non-test consumers). Product composer must keep calling them.	/web-react (needs shared composer)	gtm
8	#496 plan-MODE machinery — card covers Approve/Request-changes, but harness-pin / harnessSupportsPlanMode / composer toggle / schema planMode are not in the vertical; stay product-side.	/web-react + /chat-store schema	gtm
9	#497 late-answer affordance is an optional prop — onLateAnswer optional; if product omits it the affordance silently hides. Preserve send-new-turn wiring.	/web-react	gtm
10	Forked question-card parity unverified — hasSecretField masking, lateAnswerMessage, INTERACTION_SUBMIT_TIMEOUT_MS, option-list rendering not confirmed in shared card. Gate deletion on a real click-through.	/web-react (+ /interactions)	legal, tax
11	Turn-buffer NDJSON {type:'turn',turnId} first-line marker changes the exact byte stream the client parses; legal/tax hand-rolled readers must move to shared consumeChatStream or be re-verified.	/chat-routes + /web-react	legal, tax

4. Shared data-migration pattern (the three plans rhyme)

All three are one forward-only, additive, reversible migration — no destructive rewrite of existing chat rows:

• Threads: adopt createChatTables({ workspaceTable, threadExtraColumns }). Base columns (id, workspace_id, title, category, is_pinned, created_at, updated_at) byte-match each app's hand-rolled table → drizzle-kit generate must emit an empty thread diff. Divergent columns (status, scopeKind/scopeKey/harness) ride the /missions extras passthrough — written verbatim in the same insert, never validated. Existing rows untouched.
• Messages: factory ADDs 7 nullable usage columns (model, input_tokens, output_tokens, reasoning_tokens, cache_read_tokens, cache_write_tokens, cost_usd) — one ALTER TABLE … ADD COLUMN, no backfill, old rows read NULL. App-specific columns (toolInput/toolOutput/vaultFiles) → messageExtraColumns. parts widens Array<Record>→ChatMessagePart[] — type-only, on-disk JSON identical, no DDL.
• Turn buffer: createD1TurnEventStore(env.DB) provisions turn_events/turn_status from TURN_EVENTS_MIGRATION_SQL; legal has none today (fresh CREATE), tax's DDL is byte-identical (source-of-truth swap only).
• Verify gate (all apps): run pnpm drizzle-kit generate against prod schema and assert the ONLY statements are the 7 message ADD COLUMNs.

Where they diverge from the pattern: tax alone does a content-preserving backfill (INSERT INTO message … SELECT … FROM chat_messages mapping session_id→thread_id, content stays AES-256-GCM ciphertext, parts='[]') then DROP TABLE chat_messages, and keeps taxSessions as the canonical thread table (~15 dependent FKs) — factory thread adoption is deferred/optional. gtm and legal migrate in place with empty thread diffs.