Skip to content

plan_interactive_tutor.md

Latest commit

 

History

History
332 lines (271 loc) · 14.9 KB

plan_interactive_tutor.md

File metadata and controls

332 lines (271 loc) · 14.9 KB
Raw

Interactive Tutor — Implementation Plan (Clicky-inspired)

Bring Clicky's three interactive mechanics into the PDF Explainer lesson:

  1. The Pointing Tutor — the tutor's answer physically points at the right regions on the PDF (Clicky's [POINT:...] + flying cursor).
  2. Voice push-to-talk + spoken answers — hold a key to ask out loud; the tutor pauses the lesson, answers in voice, and points (Clicky's push-to-talk
    • ElevenLabs TTS → our Chirp TTS).
  3. Ask-about-this — click a region on the PDF and ask about exactly that (Clicky's screen-awareness → our stable page targets).

Why this maps cleanly onto what exists

Clicky primitive PDF Explainer reuse
Screen it points at Rendered PDF page + TargetAnchorLayer (DOM anchors per target)
[POINT:x,y] pixels Stable target IDs with known boxes (p2_formula_001) — no coord guessing
Cursor bezier animation OverlayLayer already flies spotlight/laser to any target
Streaming voice answers The per-action Chirp TTS path (synthesizeCloudTts)
"What to point at" aiTutor already returns referencedTargetKeys (currently unused in UI)
Interrupt / pause / resume ActionEngine already has pause/resume/waiting_* states

Current gap: AskTutorPanel and ScenePlayer are siblings under LessonPlayer with no shared state, and the tutor reply is text-only.

Shared foundation (built once, used by all three features)

S1. LessonInteractionContext (new)

A React context provided by LessonPlayer, consumed by AskTutorPanel, ScenePlayer, and TargetAnchorLayer. It is the single coordination point so the tutor (right panel) can drive the PDF stage (center).

interface TutorPoint {
  targetKey: string;       // must be a target on the currently-rendered page
  say: string;             // short spoken/explanatory chunk for this region
  audioUrl?: string;       // filled by Feature 2 (per-point Chirp clip)
  duration?: number;
}

interface TutorInterjection {
  messageId: string;
  pageNumber: number;      // the page these points live on
  points: TutorPoint[];
  activeIndex: number;     // which point is showing
}

interface LessonInteractionState {
  interjection: TutorInterjection | null;
  focusedTargetKey: string | null;   // Feature 3
  explainMode: boolean;              // Feature 3 (targets clickable)
  startInterjection(i: Omit<TutorInterjection, "activeIndex">): void;
  advancePoint(): void;
  endInterjection(): void;
  setFocusedTargetKey(key: string | null): void;
  setExplainMode(on: boolean): void;
}

Rule: while interjection !== null, the lesson ActionEngine is paused; when it ends, the lesson resumes from exactly where it was.

S2. Reusable interjection overlay rendering

ScenePlayer's PdfPageSceneRenderer already renders OverlayLayer + TargetAnchorLayer. When an interjection is active for the current page, render its active point as a normal TimedOverlay (no engine changes needed):

const interjectionOverlays: TimedOverlay[] = interjection
  ? [{ id: `tutor_${interjection.activeIndex}`, overlayType: "spotlight",
       start: 0, end: 1e9, targetBlockKeys: [activePoint.targetKey] },
     // + a callout overlay carrying activePoint.say next to the target
    ]
  : engineOverlays;

The engine is paused, so its overlays are inert; we swap in the interjection's overlays. This keeps the tutor pointing decoupled from the lesson engine (simpler than feeding transient actions into the engine; can be unified later).

Feature 1 — The Pointing Tutor

Tutor answers about the current page drive the spotlight to each referenced region, stepping through the explanation. No voice required (that's Feature 2); F1 points + shows text and advances on a read-time timer or a "Next" control.

Backend

convex/aiTutor.ts — structured output + POINT protocol. Replace the plain-text return with a Zod-validated JSON shape. Keep the rich markdown answer for the chat bubble; add an ordered pointing sequence restricted to current-page targets (visual pointing only works on the rendered page).

const TutorAnswerSchema = z.object({
  answerMarkdown: z.string(),
  sources: z.array(z.number()).default([]),
  points: z.array(z.object({
    targetId: z.string(),   // MUST be a current-page targetKey from the prompt
    say: z.string(),        // one short spoken/explanatory chunk
  })).default([]),
});

Prompt additions:

  • "You are given the targets on the CURRENT page (page X) with their IDs. If the answer involves regions on page X, return points: an ordered walkthrough where each entry names a target ID on page X and a short say explaining it, in the order you'd point at them."
  • "Use ONLY target IDs that appear in the page-X target list. If the best answer is on a different page, set points: [] and tell the student which page to open." (cross-page auto-nav is a later enhancement — see Risks.)
  • Switch the call to responseMimeType: "application/json" + responseSchema, and validate with TutorAnswerSchema + a one-shot repair (mirror the jsonRepair/repair pattern already used in actionGeneration.ts).
  • Drop the incidental text.includes(key) echo detection (replaced by points).
  • Keep the "Sources:" extraction as a fallback if sources is empty.

convex/chat.ts + convex/schema.ts. Add to chatMessages:

pointingSequenceJson: v.optional(v.string()), // JSON TutorPoint[]

Thread it through _insertMessage and listMessages. Keep referencedTargetKeys for back-compat (derive it from points).

convex/chatActions.ts — sendMessage. Persist the points:

const result = await askTutor({ ... });
await ctx.runMutation(internal.chat._insertMessage, {
  ...,
  content: result.answerMarkdown,
  sourcePages: result.sources,
  referencedTargetKeys: result.points.map(p => p.targetId),
  pointingSequenceJson: JSON.stringify(result.points),
});

Validate every point.targetId against the current page's target keys server-side (drop any the model invented) so the frontend never spotlights a non-existent target.

Frontend

  1. LessonPlayer provides LessonInteractionContext (S1) and renders the center/right columns inside it. It owns the interjection state and, via a ref to the active ActionEngine (lifted from ScenePlayer, or exposed through a callback), pauses on startInterjection and resumes on endInterjection.
  2. AskTutorPanel — when an assistant message has pointingSequence whose pageNumber === currentPageNumber, render a "Show me on the page" button on that bubble (and optionally auto-start once). Clicking calls startInterjection({ messageId, pageNumber, points }).
  3. ScenePlayer — consume the context. When an interjection is active:
    • pause the engine,
    • render interjection overlays (S2): spotlight to activePoint.targetKey via OverlayLayer, plus a CalloutOverlay carrying activePoint.say anchored beside the target,
    • show a small interjection control strip ("◀ ▶ point 2/4 ✕ resume lesson").
    • advance: F1 auto-advances after a read-time delay (Math.max(2.2, words/2.6) s) or on the ▶ control; on the last point, endInterjection() resumes the lesson.
  4. Chat bubble polish — when a point is active, highlight the matching say text in the bubble (scroll-to + emphasis) so panel and page stay in sync.

Edge cases

  • Points reference the current page only; if pageNumber !== currentPageNumber (user navigated away), show "Open page N to see this" instead of pointing.
  • Empty points → behaves exactly like today (text answer only).
  • Target measured box missing (anchor not mounted) → OverlayLayer already falls back to the stored normalized box.

Acceptance

Ask a question about the current page → the lesson pauses, the spotlight flies to each referenced region in turn with its explanation, then the lesson resumes where it left off.

Feature 2 — Voice push-to-talk + spoken answers

Adds Clicky's "real teacher next to you" loop: hold to talk, the tutor speaks back while pointing. Composes with Feature 1: each point.say becomes its own Chirp clip, so pointing is audio-synced with the same drift-proof model the lesson already uses (visual drawn → its clip plays start-to-finish → advance).

Speech-to-text (input)

Phase 2a (ship fast): browser SpeechRecognition. Zero infra; works in Chrome/Edge. Hold-to-talk starts recognition, release finalizes the transcript. Gate behind feature detection.

Phase 2b (robust, cross-browser): server STT. Mirror Clicky's proxy model (keys server-side). Capture with MediaRecorder, POST the blob to a new Convex action convex/transcription.ts → transcribeAudio that proxies to an STT provider (OpenAI Whisper or AssemblyAI — Clicky uses AssemblyAI; key in Convex env, same pattern as synthesizeCloudTts). Returns the transcript.

Push-to-talk in the player

  • Add a mic button + hold-key (e.g. hold V) handler. Reuse ScenePlayer's existing global keydown/keyup infrastructure (it already manages player shortcuts and ignores typing in inputs).
  • Press → engine.pause() + show a recording indicator (reuse the waveform idea from Clicky's overlay, or a simple pulsing mic). Release → transcribe → call sendMessage({ question, currentPageNumber, ... }).
  • This is the browser analog of Clicky's GlobalPushToTalkShortcutMonitor + BuddyDictationManager.

Text-to-speech (spoken answer)

Reuse synthesizeCloudTts (Chirp). Synthesize one clip per point.say at answer time inside sendMessage, storing audioUrl + duration on each point:

for (const point of result.points) {
  const tts = await synthesizeCloudTts({ text: point.say });
  const storageId = await ctx.storage.store(new Blob([tts.audioBytes], {type: tts.mimeType}));
  point.audioUrl = await ctx.storage.getUrl(storageId);
  point.duration = tts.durationSeconds;
}

(Extract the existing storeAudioBytes helper from processing.ts into a shared module so both pipelines use it.) Only point.say chunks get audio — not every text answer — to control TTS quota. The full markdown answer keeps a separate optional "🔊 play" button that synthesizes on demand.

Interjection player becomes audio-driven

The F1 interjection player now: for each point, draw the spotlight, play point.audioUrl, advance on the audio ended event (exactly like ActionEngine.notifySpeechEnded). Reuse a small inter-beat pause for the same "point, then talk" feel. On completion, resume the lesson.

Barge-in & resume

  • Asking again (or pressing the hotkey) during an answer cancels the current interjection audio and starts the new turn.
  • After the tutor finishes, the lesson resumes from its paused cursor.

Phases

  1. STT: 2a browser SpeechRecognition; later 2b transcribeAudio action.
  2. Push-to-talk hotkey + mic button → pause lesson, capture, transcribe, send.
  3. Per-point Chirp TTS in sendMessage; store audioUrl/duration on points.
  4. Audio-driven interjection player (advance on ended); optional "play answer".
  5. Barge-in + resume.

Acceptance

Hold the mic key during a lesson → it pauses → speak a question → the tutor answers in voice while the spotlight points at each region → the lesson resumes.

Feature 3 — Ask-about-this (page-aware selection)

Clicky sees exactly what you're looking at; here you click a region and the tutor centers its answer (and pointing) on that target.

Frontend

  1. TargetAnchorLayer interactive mode. Today anchors are non-interactive. Add an "Explain mode" toggle in PlayerControls (or hold a modifier). When on, anchors become clickable with a hover affordance (cursor + a faint outline + a "?" badge). This avoids fighting the viewer's click-to-pause (which ScenePlayer.handleViewerClick owns) — pointing/explain is opt-in.
  2. Selection → context. Clicking a target calls setFocusedTargetKey(targetKey) (S1), switches the right panel to the Ask tab, and pre-seeds the input: Explain this: "<label>".
  3. The answer comes back with points (Feature 1) and naturally spotlights the selected region; with Feature 2 it also speaks.

Backend

chatActions.sendMessage + aiTutor.askTutor accept an optional focusedTarget?: { targetKey, label, kind, text, pageNumber }. Inject into the prompt: "The student is pointing at <targetKey> (<label>, a <kind>): <text>. Center your answer and your first points entry on this region."

Phases

  1. Explain-mode toggle + clickable anchors with hover affordance.
  2. Thread focusedTargetKey through the context → panel pre-seed + sendMessage.
  3. askTutor focuses the answer/pointing on the selected target.

Acceptance

Turn on Explain mode, click a formula on the PDF → "Explain this" is pre-filled → the tutor answers about that formula and points right at it.

Cross-cutting

Data model

  • chatMessages.pointingSequenceJson (F1); per-point audioUrl/duration live inside that JSON (F2). Add stored tutor-audio cleanup to documents.ts deleteDocument and on chat-session reset.
  • Shared storeAudioBytes helper extracted from processing.ts.

Build order (each builds on the previous)

  1. Feature 1 — backend POINT protocol + LessonInteractionContext + interjection overlay player. This is the foundation; nothing else works without "tutor can point".
  2. Feature 2 — voice in/out; per-point Chirp TTS makes F1's pointing audio-synced. This is the biggest "wow", but depends on F1's interjection.
  3. Feature 3 — selection; smallest change, reuses F1's pointing and the shared context.

Risks & decisions

  • Cross-page pointing. v1 points only on the current page; if the answer is elsewhere, the tutor names the page. Later: auto-navigate to that page, then point (the player already supports onSelectChapter / page jumps).
  • STT browser support. SpeechRecognition is Chrome/Edge-only; ship that first, add server STT (2b) for Safari/Firefox + quality.
  • TTS quota. Only point.say chunks are synthesized, not full answers; one tutor turn ≈ a handful of short clips. Reuse the Chirp path/quota notes.
  • Click conflict. The viewer surface uses click-to-pause; target clicking is gated behind Explain mode so the two never fight.
  • Latency. Synthesizing per-point audio at answer time adds a beat before the spoken walkthrough starts; show the text answer immediately and let audio catch up (text is ready before clips finish).

Testing

  • Unit: TutorAnswerSchema parse + repair; server-side targetId validation (invented IDs dropped); read-time advance math.
  • Integration: ask → points stored → interjection plays + pauses/resumes lesson; focusedTarget biases the answer.
  • Manual matrix: current-page question, cross-page question, empty-points (chit-chat) answer, voice round-trip, Explain-mode selection.