Skip to content

perf(intake): the funnel stops paying for work it already has - #1483

Open
yanekyuk wants to merge 1 commit into
devfrom
perf/funnel-signal-latency
Open

perf(intake): the funnel stops paying for work it already has#1483
yanekyuk wants to merge 1 commit into
devfrom
perf/funnel-signal-latency

Conversation

@yanekyuk

Copy link
Copy Markdown
Contributor

Three cuts on the live /i/new path, between a person's last answer and the
proposal they see. Per signal: 5 model calls → 3, blocking 3 → 2.

1. The profile bridge folds into the follow-up planner

generateFollowUps ran the planner, then made a second serial call to append at
most one profile-derived option per question. Both calls already received the
same inputs. They are now one structured call whose question carries a nullable
profileBridgeOption.

Both prompts' content survives the merge — the core rules, the bridge's
"what makes a bridge useful", and the scuba-divers-plus-pianist example. Only the
call count changed.

Two properties the two-call split enforced structurally are now prompt
constraints, so both are asserted:

  • The bridge never touches a core option. A new test drives three bridge
    shapes (null, a real bridge, a bridge that duplicates a core label) and asserts
    2–3 answer-grounded options survive normalizeOption / normalizeFollowUpQuestion
    on every question, never displaced and never replaced by the fallback.
  • Bridge silence is not failure. The old code swallowed a bridge-model error
    on purpose. There is no separate failure to swallow now, so the field is
    genuinely nullable: a response with every bridge null returns the questions
    normally, in one call, with no retry. A bridge offered when no brief was
    supplied is dropped in code rather than trusted.

The $ref trap this uncovered

Merging made optionSchema appear twice in one schema, so the JSON Schema
converter emitted the second occurrence as a $ref into definitions. Gemini
rejects that document.
The merged call was failing twice on
gemini-2.5-flash, then answering from the gpt-4o-mini fallback — 1.5s became
6.5s, on a model nobody chose, with no error anywhere. Captured request bodies:

orchestrator 1474ms  model: google/gemini-2.5-flash   ← rejected
orchestrator 1433ms  model: google/gemini-2.5-flash   ← rejected (retry)
orchestrator 6698ms  model: openai/gpt-4o-mini        ← fallback answered

The fix is one extra zod instance for the bridge option so both shapes inline.
A spec guard converts the schema with LangChain's own toJsonSchema and fails on
any $ref or definitions — verified to fail when the instance is shared again.

2. The graph stops re-inferring what synthesis just wrote

invokeIntentGraphProduction ran prep → inference → verification. Its input is
the output of synthesize: one clean, self-contained, first-person signal. There
is no messy text left to pull candidates out of, and the call passes
userProfile: '', so inference's other half — enrichment — was switched off
anyway.

Chosen shape: supplying a stage's output skips that stage. shouldRunInference
routes straight to verification when state.inferredIntents is already
non-empty on entry, and the funnel seeds it with the synthesized signal. No new
operation mode. Seeded state does not collide with prep: prep writes only
activeIntents / activeIntentIds / trace, and the inferredIntents reducer
is last-write-wins on a channel prep never returns.

This generalises the way the later cut wants — pre-verified intents can skip to
the reconciler by the same rule.

Chat and MCP are untouched: they feed raw utterances in and still route through
inference.

Known loss, confirmed in code: tombstone extraction disappears from this path.
It was never reachable here — runSynthesis reads only description, score,
and verification from the result, and propose mode exits at
routeAfterVerification before the reconciler, which is the only thing that acts
on a tombstone. So nothing that ran is lost; the capability simply stops being
theoretically present.

3. Authority stops scoring signals against a profile that was never supplied

Not a speed fix. authority asks whether the speaker's profile supports the
speech act. The funnel passes userProfile: '', and the graph took
score = min(authority, sincerity, clarity) — so every funnel-created signal had
its confidence capped by a number the verifier guessed from nothing.

combineFelicityScores now leaves authority out of the minimum when no profile
was supplied. This honours the comment above invokeIntentGraphProduction rather
than reversing it: an intent derives only from the person's answers, so it should
not be scored against a profile they were never asked for. The alternative —
passing the global context paragraph getGlobalContextProduction already
fetches — changes what a signal is derived from, which is a product decision, not
a perf one.

This changes stored confidence values. New funnel signals get a
combinedScore on intent_proposals.analysis that is no longer capped by
authority; it can only rise. Existing rows keep their old numbers — there is no
backfill. Nothing gates on this score (isVague reads clarity directly), so no
signal that used to pass now fails or vice versa; only the recorded number
changes. The raw felicity_scores.authority is still stored untouched, and the
verifier still emits SKILL_MISMATCH when it is low — both left alone
deliberately, so the verdict stays a faithful record of what the model said.

Measured

Live gemini-2.5-flash, this branch vs 5b1878732, five interleaved samples
each, medians (same fixture: a two-round interview plus a profile brief):

stretch baseline branch
between answers (planner + bridge → one call) 2674ms 1684ms
last answer → proposal (synthesis + inference + verification → synthesis + verification) 4748ms 3629ms

Baseline samples: 2674/3025/2866/2656/2612 and 5686/4610/5081/4312/4748.
Branch samples: 1684/1639/1877/1736/1616 and 3629/4713/3797/3399/3515.

That is the claimed 5 → 3 calls and blocking 3 → 2. The per-call saving is
smaller than a whole call on the post-answer stretch because verification is the
slowest of the three, and it stays.

Testing

Baselines re-derived on this worktree, not quoted from anywhere:

  • protocolbun run test: 2388 pass / 0 fail / 211 files before,
    2399 pass / 0 fail / 212 files after. The runner excludes six live-model
    specs; none of them cover the intake orchestrator or the verifier, so nothing
    relevant is hidden by that exclusion. The two touched specs were also run in
    isolation.
  • services/apibun run test: 16 fail before, 16 after, 1801 pass both
    runs.
    The named set is identical apart from one timing-sensitive rate-limit
    guard (throws RateLimiterError past the limit, 5001ms in the baseline) that
    flipped to passing on a later run. That set is the stable pre-existing dev
    failure set — MCP owner-proof, CLI credentials, ToolController, conversation
    sessions, rate-limit guard, legacy-negotiation archive — and nothing in it is
    reachable from this diff.
  • apps/web — 3 failures in negotiation-presence and chat-sidebar-unread.
    Neither file, nor the module under test, imports @indexnetwork/protocol;
    they are the pre-existing web set and cannot be reached by this diff.
  • bun run lint (0 errors), typecheck, typecheck:specs all clean.

New coverage: intent.graph.profile-blind.spec.ts (seeded state skips inference,
unseeded still infers, delete still wins over the seed, and the authority
exemption at both the helper and the node), plus the bridge-property and
$ref-guard tests in intake.orchestrator.spec.ts.

packages/protocol 23.6.4 → 23.7.0; neither open PR (#1400, #1072) touches
packages/protocol/package.json.

Not in this PR

The other four cuts — merge inference+verification, batch the network scoring,
have the verifier emit the clarifier's repair, cap the verifier's reasoning
tokens — are deliberately separate branches and are untouched here.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Ay9d5zypTYuunAUZX5Vpsw

Three cuts between a person's last answer at /i/new and the proposal they
see: 5 model calls per signal become 3, and the blocking stretch 3 becomes 2.

The profile bridge folds into the follow-up planner as a nullable field on
each question, so personalization is one prompt section rather than a second
serial call — and silence on every question is an ordinary success, not a
failure to swallow. Merging put optionSchema in the schema twice, which made
the converter emit a $ref Gemini rejects: the call was burning its retry and
answering from the fallback model. One extra zod instance fixes it, and a
spec guard fails on any $ref.

The graph stops re-inferring what synthesis just wrote. A caller that
supplies a stage's output now skips that stage: seeded inferredIntents route
prep straight to verification, and the funnel seeds the synthesized signal.
Chat and MCP still infer, because they feed raw utterances in.

And authority stops capping the score of signals it was never given a
profile to judge. The propose path attaches no profile on purpose; honouring
that means leaving authority out of the minimum rather than guessing it.

Claude-Session: https://claude.ai/code/session_01Ay9d5zypTYuunAUZX5Vpsw
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant