AI Observer Panel: per-handler test scaffolding

[droarty]

AI Observer Panel: per-handler test scaffolding

Context

The "AI Requests" tab currently shows a read-only logTreePanel listing handler names
that have an ai-route step (via aiHandlersView, added in #259 and generalized in a
follow-up using the $map/$where transform directive). This extends it into a real
testing tool: each ai-routed handler gets a persisted tests array on the artifact's
own state, the tree becomes editable (handler → its tests), and clicking "Add Test"
scaffolds a new test object (seeded from the $state.*/$message.* paths the handler
actually references) that's immediately editable in a JSON editor. Per your framing,
this PR stops at "display the test JSON editor" — saving edits and actually running
tests against a handler is separate follow-up work.

Key technical findings from exploration

• DatabasePersistor.ts (server-side action persistence) never reads a document
  into Node before writing — every actionType is either batched into one trailing
  native $set/$push/$pull, or (for upsert/update-in/slice) issues its own
  Mongo aggregation-pipeline updateOne that references the field's current value
  via $<mongoPath> inside the pipeline itself. Critically, merge does not do a
  safe object-merge — it decomposes into one $set per key in value, so an existing
  key in value would clobber existing data. This means "add missing handler names
  to tests without touching ones that already have tests" needs a new action type,
  not merge.
• JSONata ~{ } pitfall (already documented from the prior PR, re-confirmed): use
  bare item/message/state, never $item/$message/$state, inside ~{ }.
• Dynamic action path via ~{ } works today, unmodified. resolveValue evaluates
  ~{ } before checking for a literal $-prefix, so a JSONata expression that
  itself returns a string starting with "$state." (e.g.
  "~{ '$state.tests.' & message.handlerName }") passes DatabasePersistor.ts's
  path.startsWith('$state.') guard and works with the existing, unmodified
  append action — no engine change needed for the "add test" append. (No existing
  config does this today, so it's new usage, not a new capability.)
• TreeView (apps/web/src/components/ui/tree-view.tsx) is semi-controlled: no
  value prop, but it re-syncs its internal selection whenever initialSelectedItemId
  changes across renders — good enough to drive "select the newly created test."
• No editable JSON component exists (JsonView.tsx is a read-only <pre>, no
  editor library is installed). Build a small controlled-<textarea> editor following
  WritingArea.tsx's existing style — no new dependency.
• Tree-node selection should be pure local React state — LogTreePanel's
  select-log today round-trips through the server for a click that only ever does a
  client-side update-state; there's no reason to repeat that for this component.
• $map's object-source extension (added in the prior PR) isn't needed here — the
  tree is built client-side in the new component directly from @state.tests (a plain
  dot-path prop, already fully supported), not server-side via $map.

Design

1. New defaults action type — upsert-if-absent on an object

Server — apps/api/src/app/websocket/DatabasePersistor.ts, new case (mirrors
upsert/slice's "own aggregation-pipeline updateOne" pattern, no document read):

case 'defaults': {
  const fieldRef = `$${mongoPath}`;
  await db.collection('artifacts').updateOne(
    { _id: artifactId },
    [{ $set: { [mongoPath]: { $mergeObjects: [value, { $ifNull: [fieldRef, {}] }] } } }] as any
  );
  break;
}

$mergeObjects merges value (candidates) then the field's current value on top —
later arguments win on key collision, so existing keys are preserved untouched and
only genuinely-missing keys from value get added.

Client — apps/web/src/app/services/documentModelStore.ts, applyAction, new case
(same precedence, existing wins):

case 'defaults': {
  const existing = (getAtPath(next, resolvedPath) as Record<string, unknown>) ?? {};
  return setAtPath(next, resolvedPath, { ...(value as Record<string, unknown>), ...existing });
}

Shared type — libs/shared-types/src/message.types.ts: add 'defaults' to
ActionItem['actionType'].

2. workflow-builder.json — upsert tests defaults, render the new panel

Replace the body of ai-handlers-view-render (the handler that currently renders
logTreePanel) with two steps:

"ai-handlers-view-render": {
  "steps": [
    {
      "route": ["client", "database"],
      "transform": {
        "clientMessageType": "update-state",
        "actions": [
          {
            "actionType": "defaults",
            "path": "$state.tests",
            "value": "~{ $merge($each(message.document.state.draftConfig.handlers, function($v, $k){ $v.steps[route='ai' or ($type(route)='array' and 'ai' in route)] ? {($k): []} : {} })) }"
          }
        ]
      }
    },
    {
      "route": "client",
      "transform": {
        "clientMessageType": "initialize-view",
        "viewHandler": "aiHandlersView",
        "layoutConfig": [
          { "componentType": "aiObserverPanel", "props": { "tests": "@state.tests" }, "emits": { "addTest": "add-test" } }
        ]
      }
    }
  ]
}

The candidate value computes "all currently ai-routed handler names → []" freshly
every time (no need to pre-filter against existing state.tests — defaults handles
that). Step 2's @state.tests is a plain client-resolved dot-path — by the time the
client renders it, step 1's update-state has already been processed (messages are
handled in order), so tests is already up to date. No second get-channel-document
fetch needed.

Add two new handlers for "Add Test":

"add-test": {
  "steps": [
    { "route": "database-query", "query": { "name": "get-handler-test-skeleton", "responseType": "handler-test-skeleton-ready" } }
  ]
},
"handler-test-skeleton-ready": {
  "steps": [
    {
      "route": ["client", "database"],
      "transform": {
        "clientMessageType": "update-state",
        "actions": [
          { "actionType": "append", "path": "~{ '$state.tests.' & message.handlerName }", "value": "$message.skeleton" }
        ]
      }
    }
  ]
}

3. New query — apps/api/src/app/websocket/QueryExecutor.ts

get-handler-test-skeleton: fetch draftConfig (same getArtifactIdForChannel +
artifacts.findOne pattern as get-workflow-builder-context), look up
draftConfig.handlers[handlerName], scan its steps' transform/condition/query
fields (deliberately excluding ai.systemPrompt, which is free-form prose most
likely to produce false-positive matches, not real data references) for
$state.x/state.x/$message.x/message.x patterns, and build a nested skeleton:

if (queryName === 'get-handler-test-skeleton') {
  const channel = context.message['channel'] as string | undefined;
  const handlerName = context.message['handlerName'] as string | undefined;
  const artifactId = channel ? await getArtifactIdForChannel(channel) : null;
  const doc = artifactId ? await db.collection('artifacts').findOne({ _id: artifactId }, { projection: { state: 1 } }) : null;
  const state = (doc?.['state'] as Record<string, unknown> | undefined) ?? {};
  const draftConfig = state['draftConfig'] as
    | { handlers?: Record<string, { steps?: Record<string, unknown>[] }> }
    | null | undefined;
  const handler = handlerName ? draftConfig?.handlers?.[handlerName] : undefined;

  const skeleton: Record<string, unknown> = { createdAt: new Date().toISOString() };
  const scanTargets = (handler?.steps ?? []).map((s) => ({
    transform: s['transform'], condition: s['condition'], query: s['query'],
  }));
  const text = JSON.stringify(scanTargets);
  const pattern = /\$?(state|message)\.([a-zA-Z0-9_]+(?:\.[a-zA-Z0-9_]+)*)/g;
  const seen = new Set<string>();
  let m: RegExpExecArray | null;
  while ((m = pattern.exec(text))) {
    const [, root, subPath] = m;
    const key = `${root}.${subPath}`;
    if (seen.has(key)) continue;
    seen.add(key);
    skeleton[root] = skeleton[root] ?? {};
    let curr = skeleton[root] as Record<string, unknown>;
    const parts = subPath.split('.');
    parts.forEach((p, i) => {
      if (i === parts.length - 1) curr[p] = '';
      else { curr[p] = curr[p] ?? {}; curr = curr[p] as Record<string, unknown>; }
    });
  }

  return { skeleton, handlerName };
}

This is a best-effort heuristic scaffold, not a precise static analyzer — matches your
description ("so if there are references to state.something... the test object will
have...") and the fact that the result lands directly in an editable JSON editor for
manual correction.

4. New component — apps/web/src/components/layout/AiObserverPanel.tsx

Adapted from LogTreePanel.tsx (same TwoColumnPanel shell), but with local state
instead of server round-trips for selection, and an editable right pane:

• Props: { tests?: Record<string, unknown[]>, onAddTest?: (payload: { handlerName: string }) => void } (onAddTest is the resolveEmits-injected prop from the emits: {addTest: "add-test"} config).
• useMemo builds TreeDataItem[] from Object.entries(tests ?? {}): one parent node
  per handler (id/name = handler name), each with child nodes per test entry
  (id/name derived from the test's createdAt), every node carrying
  rawData: { handlerName, test? } so the selected node always resolves back to its
  owning handler regardless of whether a handler or a test node is selected.
• Local useState for selectedId (no emit — matches the exploration finding that
  mere selection should stay client-side).
• Right panel: selected handler's name, an "Add Test" button (onClick calls
  onAddTest({ handlerName })), and the new JsonEditor component (see below) bound
  to the selected test's data (or empty/disabled if a handler node, not a test node, is
  selected).
• A small useEffect/ref-based "pending add" flag: when "Add Test" is clicked, remember
  the handler; once tests[handlerName] grows by one entry, select the newest one
  (last array element) and clear the flag. Ties together "Add Test" → auto-select
  without needing the server to echo back a specific test id.

5. New component — apps/web/src/components/layout/JsonEditor.tsx

Following WritingArea.tsx's established style: a controlled <textarea> seeded via
JSON.stringify(value, null, 2), onChange attempts JSON.parse and tracks a parse-
error state (shown inline, doesn't block typing). No save/persist wiring yet — purely
local component state, per your stated scope for this PR.

6. Registration and docs

• apps/web/src/app/registry/layoutRegistry.ts — register aiObserverPanel (lazy
  import, same pattern as every other entry).
• docs/workflow-reference/registered-component-types.md — document aiObserverPanel
  (props, example JSON, brief note on the tree/detail/editor behavior).

Files touched

• libs/shared-types/src/message.types.ts — add 'defaults' action type.
• apps/api/src/app/websocket/DatabasePersistor.ts — defaults case.
• apps/web/src/app/services/documentModelStore.ts — defaults case in applyAction.
• apps/api/src/app/websocket/QueryExecutor.ts — new get-handler-test-skeleton query.
• apps/api/src/app/config/workflows/workflow-builder.json — upsert step, new
  add-test/handler-test-skeleton-ready handlers, swap logTreePanel →
  aiObserverPanel.
• apps/web/src/components/layout/AiObserverPanel.tsx — new (duplicated from
  LogTreePanel.tsx, then adapted).
• apps/web/src/components/layout/JsonEditor.tsx — new.
• apps/web/src/app/registry/layoutRegistry.ts — register aiObserverPanel.
• docs/workflow-reference/registered-component-types.md — document it.

LogTreePanel.tsx, TwoColumnPanel.tsx, VerticalSplitPanel.tsx, TreeView, and the
existing $map/$where transform machinery are all reused/untouched — this is
additive.

Verification

1. pnpm run restart:both (or restart from this worktree).
2. Open a workflow-builder document, seed a synthetic draftConfig via mongosh
   (mixing ai-route and non-ai-route handlers, same approach as prior verification
   passes), open the "AI Requests" tab.
3. Confirm state.tests now has one key ([]) per ai-routed handler — check via
   mongosh directly against the artifact.
4. Manually seed one handler's state.tests.<name> with a fake existing test before
   reopening the tab, then reopen it — confirm that handler's existing test is not
   clobbered back to [] (the core defaults correctness check).
5. Click a handler node → confirm right panel shows its name + "Add Test". Click
   "Add Test" → confirm a new test appears as a child node, is auto-selected, and the
   JSON editor shows a skeleton with createdAt plus any state.*/message.* paths
   referenced by that handler's transform/condition/query (seed a handler with a known
   reference to make this concrete and checkable).
6. Edit the JSON in the editor — confirm it's freely editable locally (no crash on
   invalid JSON, no attempt to save/persist — out of scope for this PR).
7. npx nx build web and npx nx build api to catch type errors, especially around
   the new ActionItem union member touching both client and server switch statements.
8. Clean up all test-only data afterward (mongo edits, test users/artifacts), scoped by
   exact ID, per this project's established convention.