plan_imp.md

PDF Explainer Agentic Classroom Implementation Plan

Goal

Build the PDF Explainer into an agentic classroom system where a learner uploads PDF notes and the AI teaches the notes page by page. The original PDF remains the visual source of truth. The AI teaches on top of each page with accurate focus effects, marker highlights, arrows, circles, zooms, callouts, whiteboard derivations, quizzes, and optional interactive visualizations.

This plan adapts the OpenMAIC mechanism to a PDF-first product:

• OpenMAIC uses generated slide elements with stable IDs.
• PDF Explainer must create stable visual targets from each PDF page.
• OpenMAIC generates action sequences that target element IDs.
• PDF Explainer should generate action sequences that target PDF page target IDs.
• OpenMAIC plays actions through an action/playback engine.
• PDF Explainer should move from passive overlay timelines to a scene/action playback model.

All implementation, prompts, UI copy, examples, fallback text, generated narration, quiz text, and widget text should be English only.

References From OpenMAIC

Use these files as behavioral references, not as copy-paste sources:

• F:/OpenMAIC/lib/prompts/templates/interactive-outlines/system.md
• F:/OpenMAIC/lib/prompts/templates/slide-actions/system.md
• F:/OpenMAIC/lib/prompts/snippets/whiteboard-reference.md
• F:/OpenMAIC/lib/generation/scene-generator.ts
• F:/OpenMAIC/lib/generation/action-parser.ts
• F:/OpenMAIC/lib/playback/engine.ts
• F:/OpenMAIC/lib/action/engine.ts
• F:/OpenMAIC/components/slide-renderer/Editor/SpotlightOverlay.tsx
• F:/OpenMAIC/components/slide-renderer/Editor/LaserOverlay.tsx

Current Project State

The current project already has a useful foundation:

• Convex schema for documents, pages, page explanations, audio, overlays, quizzes, jobs, and chat.
• Upload and dashboard flow.
• PDF canvas renderer using PDF.js in the browser.
• AI page analysis through Vertex Gemini.
• Per-page TTS generation.
• Overlay rendering for pointer, spotlight, laser, highlight, circle, arrow, callout, zoom, dim background, and quiz popup.
• Transcript, quiz card, ask tutor panel, and player controls.

The main limitations to fix:

• The AI currently guesses regions from the PDF page image instead of using a reliable target list.
• Processing is page-analysis-first, not scene/action-first.
• Audio is page-level, while the OpenMAIC mechanism works better with speech actions.
• Overlays are timeline records, but there is no action engine that can run speech, visual effects, whiteboard actions, quiz gates, and widget actions in one sequence.
• There is no document-level lesson planner that decides where to insert quizzes, whiteboard explanations, or interactive visualizations.
• There is no coverage validator proving that all important PDF content is taught.

Target Architecture

The finished system should have these layers:

1. PDF ingestion
2. Page target extraction
3. Document lesson planning
4. Per-page action generation
5. Activity scene generation
6. TTS generation for speech actions
7. Playback/action engine
8. Accurate overlay rendering
9. Whiteboard teaching layer
10. Quiz and visualization interludes
11. Tutor chat grounded in the PDF
12. Coverage and quality validation

Core Data Model

Lesson Scene

Add a scene model that can represent the original PDF pages plus inserted activities.

type LessonSceneType =
  | "pdf_page"
  | "whiteboard"
  | "quiz"
  | "interactive"
  | "summary";

type LessonScene = {
  id: string;
  documentId: Id<"documents">;
  pageId?: Id<"documentPages">;
  pageNumber?: number;
  order: number;
  type: LessonSceneType;
  title: string;
  objective: string;
  sourcePageNumbers: number[];
  contentJson: string;
  createdAt: number;
  updatedAt: number;
};

PDF pages become pdf_page scenes. Quizzes, widgets, and whiteboard-only derivations are inserted between PDF page scenes when needed.

Page Target

Create stable target IDs for anything the AI may focus on.

type PageTargetKind =
  | "heading"
  | "paragraph"
  | "formula"
  | "diagram"
  | "table"
  | "image"
  | "caption"
  | "example"
  | "key_point"
  | "region";

type NormalizedBox = {
  x: number;
  y: number;
  width: number;
  height: number;
};

type PageTarget = {
  id: string;
  documentId: Id<"documents">;
  pageId: Id<"documentPages">;
  pageNumber: number;
  kind: PageTargetKind;
  label: string;
  text?: string;
  box: NormalizedBox;
  order: number;
  source: "pdf_text" | "vision" | "ocr" | "fallback";
  confidence: number;
  createdAt: number;
};

This is the PDF equivalent of OpenMAIC slide element IDs. Every spotlight, laser, arrow, highlight, and narration reference should target these IDs.

Teaching Action

Move toward OpenMAIC-style actions.

type TeachingAction =
  | {
      type: "speech";
      id: string;
      text: string;
      targetIds?: string[];
      audioStorageId?: Id<"_storage">;
      audioUrl?: string;
      duration?: number;
    }
  | { type: "spotlight"; id: string; targetId: string; dimness?: number }
  | { type: "laser"; id: string; targetId: string; color?: string }
  | { type: "highlight"; id: string; targetId: string; color?: string }
  | { type: "circle"; id: string; targetId: string; color?: string }
  | { type: "arrow"; id: string; fromTargetId?: string; toTargetId: string }
  | { type: "callout"; id: string; targetId: string; text: string }
  | { type: "zoom"; id: string; targetId: string; scale?: number }
  | { type: "quiz_gate"; id: string; quizId: Id<"quizzes"> }
  | { type: "whiteboard_open"; id: string }
  | { type: "whiteboard_draw_text"; id: string; params: object }
  | { type: "whiteboard_draw_latex"; id: string; params: object }
  | { type: "whiteboard_draw_shape"; id: string; params: object }
  | { type: "whiteboard_draw_line"; id: string; params: object }
  | { type: "whiteboard_draw_chart"; id: string; params: object }
  | { type: "whiteboard_draw_table"; id: string; params: object }
  | { type: "whiteboard_draw_code"; id: string; params: object }
  | { type: "whiteboard_edit_code"; id: string; params: object }
  | { type: "whiteboard_clear"; id: string }
  | { type: "whiteboard_close"; id: string }
  | { type: "widget_highlight"; id: string; selector: string }
  | { type: "widget_set_state"; id: string; payload: object }
  | { type: "widget_annotation"; id: string; selector: string; text: string }
  | { type: "widget_reveal"; id: string; selector: string };

Store these in a new sceneActions table or as small bounded scene-level JSON if the action count is guaranteed to stay small. Prefer a table for scale and debuggability.

Processing Pipeline

Step 1: Upload PDF

Keep the existing upload flow.

Changes:

• Store the original PDF in Convex storage.
• Create a document row.
• Schedule a durable processing action.
• Use local PDF page counting with pdf-lib, not Gemini.

Step 2: Create Page Records

For each page:

• Create or update documentPages.
• Store page dimensions from PDF metadata if available.
• Render or prepare browser-side page rendering for the player.
• Create thumbnails later if needed.

Step 3: Extract Page Targets

This is the most important improvement.

For each page, build a reliable target list:

1. Extract PDF text items with bounding boxes.
2. Merge nearby text items into readable blocks.
3. Detect headings, paragraphs, formulas, captions, tables, and examples.
4. Render the page image for vision analysis.
5. Ask the vision model to find diagrams, formulas, tables, image regions, and layout groups.
6. Merge text-derived targets and vision-derived targets.
7. Deduplicate overlapping boxes.
8. Assign stable IDs such as p3_heading_001, p3_formula_002, p3_diagram_001.
9. Store targets in pageTargets.

Acceptance criteria:

• Debug mode can show every target box over the PDF page.
• Boxes align with visible PDF content.
• All target coordinates are normalized from 0 to 1.
• Target IDs remain stable across regeneration when the page content is unchanged.

Step 4: Generate Document Lesson Plan

Before generating per-page scripts, run a document-level planner.

Input:

• Document title
• Page count
• Per-page summaries
• Page target inventory
• Extracted page text

Output:

{
  "languageDirective": "Teach everything in English using clear, student-friendly explanations.",
  "scenes": [
    {
      "type": "pdf_page",
      "pageNumber": 1,
      "title": "Page 1: Introduction",
      "objective": "Explain the main idea and orient the learner."
    },
    {
      "type": "quiz",
      "afterPageNumber": 1,
      "title": "Quick Check",
      "objective": "Confirm the learner understands the first concept."
    },
    {
      "type": "interactive",
      "afterPageNumber": 2,
      "widgetType": "simulation",
      "title": "Explore the Concept",
      "objective": "Let the learner manipulate the key variables."
    }
  ]
}

Rules:

• Every original PDF page must appear as a pdf_page scene.
• Insert activities only when they improve learning.
• Prefer quizzes for recall checks.
• Prefer whiteboard scenes for derivations or step-by-step problem solving.
• Prefer interactive widgets for concepts that benefit from manipulation or visualization.
• Keep the plan English only.

Step 5: Generate Per-Page Teaching Actions

For each pdf_page scene, use a prompt based on OpenMAIC's slide action generator.

Input:

• Page number and document context
• Previous page summary
• Next page preview
• Page target list with IDs, labels, text snippets, kinds, and boxes
• Scene objective

Output:

A JSON array of interleaved actions and speech.

[
  { "type": "action", "name": "spotlight", "params": { "targetId": "p1_heading_001" } },
  { "type": "text", "content": "First, look at the heading. It tells us the main topic of this page." },
  { "type": "action", "name": "laser", "params": { "targetId": "p1_formula_001" } },
  { "type": "text", "content": "This formula is the key relationship. We will read each part slowly." }
]

Generation rules:

• Use only target IDs provided in the target list.
• Point first, then speak.
• Each important target should be explained or intentionally skipped with a reason.
• Use spotlight for sustained focus.
• Use laser for brief pinpoint emphasis.
• Use highlight for text marker behavior.
• Use circle for formulas, values, labels, or small visual regions.
• Use arrow to show relationships between two targets.
• Use callout only for short clarifications.
• Use whiteboard actions when the PDF needs step-by-step expansion.
• Do not greet after the first page.
• Do not say "slide"; say "page".
• Do not invent content outside the PDF. If extra explanation is necessary, label it as extra explanation.
• Output valid JSON only.
• Output English only.

Step 6: Validate, Repair, and Normalize Actions

Create convex/aiActionSchemas.ts with Zod schemas for:

• Page target
• Teaching action
• Activity outline
• Widget teacher action
• Whiteboard action
• Quiz action

Validation must enforce:

• Every action target ID exists.
• Every speech action has non-empty English text.
• Discussion-like activities are not inserted on every page.
• Whiteboard coordinates fit inside the board.
• LaTeX strings are correctly escaped.
• No action references decorative or margin-only targets.
• No page has zero teaching speech.
• Every page has enough coverage.

If validation fails:

1. Run a repair prompt with the validation error.
2. If repair fails, generate a deterministic fallback action script from the page targets.
3. Mark the page as partially generated but still playable.

Step 7: Generate TTS Per Speech Action

Change from one page audio file to one audio file per speech action, or store per-action timing inside a page audio file. The OpenMAIC-style approach is per speech action.

Recommended:

• Each speech action stores audioStorageId, audioUrl, and duration.
• The playback engine waits for speech audio to finish before continuing.
• If TTS fails for one speech action, use browser speech synthesis or estimated timing fallback.
• Never reuse an audio URL across pages unless it is the exact same action by ID.

This directly fixes the "same audio on every slide" class of bugs.

Step 7A: Google Cloud TTS SSML Mark Timepoints

Add Google Cloud Text-to-Speech as a timestamp-capable TTS provider alongside the current Vertex Gemini TTS provider.

Important distinction:

• Vertex Gemini TTS currently used in convex/vertexClient.ts returns audio through generateContent.
• Google Cloud Text-to-Speech v1beta1/text:synthesize can return timepoints for SSML <mark> tags.
• These are not automatic word timestamps. They are timestamps for marks we insert into the SSML.
• Segment-level marks should be implemented first. Word-level marks can be added later for karaoke transcript highlighting.

Authentication:

• Reuse the same Google service-account JSON already used for Vertex AI.
• The existing cloud-platform OAuth scope is sufficient for Cloud TTS.
• The GCP project must have texttospeech.googleapis.com enabled.
• Billing must be enabled.
• The service account must have permission to call Cloud Text-to-Speech.

Environment variables to add:

GOOGLE_CLOUD_TTS_ENABLED=true
GOOGLE_CLOUD_TTS_VOICE=en-US-Neural2-F
GOOGLE_CLOUD_TTS_LANGUAGE=en-US
GOOGLE_CLOUD_TTS_AUDIO_ENCODING=MP3

Provider request shape:

{
  "input": {
    "ssml": "<speak><mark name=\"seg_001\"/>First, look at the heading. <mark name=\"seg_002\"/>Now focus on the formula.</speak>"
  },
  "voice": {
    "languageCode": "en-US",
    "name": "en-US-Neural2-F"
  },
  "audioConfig": {
    "audioEncoding": "MP3"
  },
  "enableTimePointing": ["SSML_MARK"]
}

Expected response shape:

{
  "audioContent": "base64 audio",
  "timepoints": [
    { "markName": "seg_001", "timeSeconds": 0 },
    { "markName": "seg_002", "timeSeconds": 3.42 }
  ]
}

Implementation plan:

1. Add synthesizeCloudTtsWithMarks in convex/vertexClient.ts or a new convex/googleCloudTts.ts.
2. Build SSML from narration segments by inserting one <mark> before each segment.
3. Escape SSML text safely.
4. Call https://texttospeech.googleapis.com/v1beta1/text:synthesize.
5. Decode audioContent into a Blob and store it in Convex storage.
6. Convert returned timepoints into SegmentTiming[].
7. Store exact timings in pageAudio.segmentTimingsJson.
8. Use exact mark timings in buildTimedOverlays.
9. Fall back to proportional timing when Cloud TTS returns missing or invalid marks.

Recommended first version:

• Use one page-level audio file with segment-level marks.
• Keep the current pageAudio table.
• Use returned segment timepoints for overlay sync.
• Later, when the action engine is implemented, move from page-level audio to per-speech-action audio.

SSML mark rules:

• Do not place consecutive marks with no spoken text between them.
• Do not add marks around empty segments.
• Keep mark names stable and ASCII-only, such as seg_001.
• Strip or escape XML-sensitive characters in narration text.
• If the source PDF has non-English terms, explain them in English and quote only the minimal visible term when necessary.

Why this matters:

• Current overlay timing is estimated from character count.
• SSML mark timepoints provide real audio offsets for each narration segment.
• This makes spotlight, laser, highlight, and transcript sync feel much more natural.

Step 8: Build Playback Engine

Add a real action playback engine.

Responsibilities:

• Load current scene actions.
• Execute actions in order.
• Fire and continue for visual effects such as spotlight, laser, highlight, circle, and arrow.
• Wait for synchronous actions such as speech, quiz_gate, widget interactions, video, and whiteboard sequences.
• Support play, pause, resume, stop, replay, next page, previous page, and speed control.
• Persist current scene/action index for resume.

Engine modes:

• idle
• playing
• paused
• waiting_for_quiz
• waiting_for_widget
• complete

Step 9: Build Action Engine

Create an action engine similar to OpenMAIC.

Responsibilities:

• speech: play action audio and update transcript bubble.
• spotlight: set focused target and dim the PDF page.
• laser: animate a pointer to target center.
• highlight: show marker-style rectangle over target.
• circle: draw a hand-marked circle around target.
• arrow: draw relationship arrow from one target to another.
• callout: show a short anchored explanation.
• zoom: scale or magnify the target area.
• quiz_gate: show quiz and wait for answer.
• whiteboard_*: update whiteboard state.
• widget_*: post messages into the widget iframe.

Accurate Overlay Mechanism

OpenMAIC measures real DOM elements by ID. PDF Explainer should create DOM target anchors over the PDF page.

Implementation:

1. Render the PDF page canvas.
2. Render invisible target anchors over it:

<div
  id={`pdf-target-${target.id}`}
  data-target-id={target.id}
  style={{
    position: "absolute",
    left: target.box.x * pageWidth,
    top: target.box.y * pageHeight,
    width: target.box.width * pageWidth,
    height: target.box.height * pageHeight,
  }}
/>

3. Spotlight, laser, highlight, circle, and arrows measure those anchors with getBoundingClientRect.
4. Overlays render from measured DOM boxes instead of trusting stale pixel math.

This gives the same natural feel as OpenMAIC while keeping the original PDF as the background.

Whiteboard Teaching Layer

Add a whiteboard overlay or side-stage for concepts that need step-by-step teaching.

Use the OpenMAIC whiteboard action shapes, adapted to English-only prompts.

Board rules:

• Canvas size: 1000 by 563 virtual pixels.
• Safe zone: keep content inside x 20 to 980 and y 20 to 543.
• Use text for plain explanations.
• Use LaTeX only for formulas.
• Use lines and arrows for relationships.
• Use charts and tables when the PDF has data or comparisons.
• Use code blocks for programming notes.
• Do not close the whiteboard immediately after drawing; let learners read it.

Whiteboard action examples:

{ "type": "action", "name": "wb_open", "params": {} }

{
  "type": "action",
  "name": "wb_draw_text",
  "params": {
    "content": "Step 1: identify the known values.",
    "x": 60,
    "y": 60,
    "width": 620,
    "height": 48,
    "fontSize": 20,
    "color": "#333333"
  }
}

{
  "type": "action",
  "name": "wb_draw_latex",
  "params": {
    "latex": "\\\\frac{a}{b}",
    "x": 100,
    "y": 130,
    "height": 80
  }
}

Important:

• In JSON, every LaTeX backslash must be written as a double backslash.
• For example, the LaTeX source \frac{a}{b} must appear in JSON as "\\\\frac{a}{b}" if the prompt text is itself inside a TypeScript string, and as "\\frac{a}{b}" in the final JSON emitted by the model.
• Never put LaTeX commands inside plain text actions.

Quiz and Knowledge Check System

The AI should insert checks only when useful.

Quiz scene types:

• Multiple choice
• True/false
• Fill in the blank
• Short answer
• Visual identification on the PDF page

Rules:

• Do not add a quiz after every page by default.
• Add a quiz after dense definitions, formulas, procedures, or sections.
• Keep quiz text English only.
• Ground every question in source pages.
• Give immediate feedback.
• Explain why the answer is correct.
• Save attempts and progress.

Interactive Visualization System

Add optional interactive scenes when the PDF concept benefits from manipulation.

Widget types:

• simulation: variables, physics, chemistry, math, systems.
• diagram: process flows, maps, hierarchies, system relationships.
• code: programming exercises and algorithm tracing.
• game: skill-based practice, not a disguised quiz.
• visualization3d: geometry, anatomy, molecules, astronomy, spatial topics.

Implementation:

• Generate widget outline at document-plan time.
• Generate widget HTML at scene generation time.
• Sandbox the widget in an iframe.
• Use postMessage for teacher actions.
• Validate that the widget has visible content on load.
• Verify it works on mobile.
• Provide a fallback quiz if widget generation fails.

Prompt Files To Add

Create these prompt files under convex/prompts or convex/prompts/templates:

convex/prompts/templates/pdf-lesson-outline/system.md
convex/prompts/templates/pdf-page-actions/system.md
convex/prompts/templates/pdf-whiteboard-actions/system.md
convex/prompts/templates/pdf-widget-outline/system.md
convex/prompts/templates/pdf-widget-teacher-actions/system.md
convex/prompts/snippets/whiteboard-reference.md

English-only prompt rule to include in all of them:

All generated text must be English only. This includes narration, titles, labels, quiz questions, answer choices, explanations, widget text, fallback text, and UI-facing strings. If the source PDF contains non-English terms, quote only the minimal visible term when necessary and explain it in English.

Convex Schema Additions

Add or migrate toward these tables:

• pageTargets: stable visual targets for every page.
• lessonScenes: ordered PDF page and activity scenes.
• sceneActions: ordered action records for each scene.
• interactiveWidgets: generated widget HTML/config per interactive scene.
• whiteboardSnapshots: optional persisted whiteboard state per scene.
• quizAttempts: learner answers and scores.
• lessonProgress: current scene/action progress per learner.

Keep existing tables while migrating:

• documentPages
• pageExplanations
• pageAudio
• pageOverlays
• quizzes
• chatSessions
• chatMessages

During migration, the app can support both:

• legacy page overlays
• new scene actions

Once the new action engine is stable, overlays can be derived from actions instead of stored as the primary source.

Update existing audio fields:

• Keep pageAudio.segmentTimingsJson, but prefer Google Cloud TTS SSML mark timings when available.
• Add pageAudio.timingProvider later if needed, with values such as estimated, cloud_tts_ssml_marks, or provider_word_timestamps.
• Store provider as google-cloud-tts for Cloud TTS audio and vertex-ai-gemini-tts for the current Gemini path.
• If using per-action audio later, add timing fields directly to sceneActions for speech actions.

Backend Modules To Add Or Refactor

Add convex/pageTargets.ts

Functions:

• extract targets from PDF text blocks
• merge vision targets
• save targets
• query targets by page
• debug target output

Add convex/lessonPlanner.ts

Functions:

• create document-level lesson plan
• insert quiz/widget/whiteboard scenes
• validate every PDF page is included

Add convex/actionGeneration.ts

Functions:

• generate actions for PDF page scene
• generate actions for quiz scene
• generate actions for whiteboard scene
• generate widget teacher actions
• repair invalid action JSON
• fallback action generation

Add convex/actionSchemas.ts

Zod schemas:

• PageTargetSchema
• TeachingActionSchema
• SceneActionListSchema
• WhiteboardActionSchema
• WidgetTeacherActionSchema
• LessonPlanSchema

Refactor convex/processing.ts

New flow:

1. read PDF
2. count pages locally
3. create pages
4. extract page targets
5. create document lesson plan
6. create lesson scenes
7. generate actions for each scene
8. generate TTS per speech action
9. mark document ready when enough scenes are playable

For the intermediate page-level player, generate one Cloud TTS audio file per page using SSML marks before each narration segment. Save the returned mark offsets as exact segment timings.

Refactor convex/aiPageAnalysis.ts

Split the current all-in-one page analysis into:

• page target vision analysis
• page summary generation
• page action generation
• fallback generation

Add convex/googleCloudTts.ts

Functions:

• create SSML from narration segments
• escape SSML text
• call Cloud TTS v1beta1/text:synthesize
• decode MP3 or LINEAR16 audio
• map timepoints into SegmentTiming[]
• validate all expected segment marks are present
• fallback to estimated timing when marks are missing

Frontend Components To Add Or Refactor

Add ScenePlayer

Replaces the page-only lesson player over time.

Responsibilities:

• load current lesson scene
• load scene actions
• initialize playback engine
• render scene renderer by type
• handle navigation

Add PdfPageScene

Renders:

• PDF canvas
• invisible target anchors
• overlay effects
• transcript bubble

Add TargetAnchorLayer

Renders DOM anchors for page targets.

Refactor OverlayLayer

Split into focused components:

• SpotlightOverlay
• LaserOverlay
• MarkerHighlightOverlay
• CircleOverlay
• ArrowOverlay
• CalloutOverlay
• ZoomOverlay

These should measure target anchors by DOM ID.

Add WhiteboardLayer

Renders:

• virtual 1000 by 563 whiteboard canvas
• whiteboard elements
• draw animations
• optional history/debug

Add InteractiveWidgetRenderer

Renders:

• sandboxed iframe
• postMessage bridge
• loading and fallback states

Refactor QuizCard

Support:

• quiz scenes
• in-page quiz gates
• visual target questions
• saved attempts

Coverage Validator

Add a quality gate so the system actually teaches the PDF.

For each page:

• Identify key targets.
• Check that each key target appears in at least one speech action or visual action.
• Check that every speech action is grounded in one or more targets.
• Check that there are no huge generic boxes unless the whole page is being summarized.
• Check that narration follows visual order.
• Check that formulas/tables/diagrams receive extra explanation.
• Check that page-level summary exists.
• Check that optional quiz/widget insertion is justified.

If coverage is poor:

• repair the action script
• add missing explanation actions
• add a whiteboard action sequence for hard concepts
• add a quiz after the page if needed

Development Phases

Phase 1: Target Extraction and Debug Overlay

Tasks:

• Add pageTargets table.
• Extract text blocks with coordinates.
• Generate target IDs.
• Add debug target overlay in the lesson player.
• Add vision target detection for diagrams, formulas, tables, and images.

Done when:

• Every page has target boxes.
• Debug overlay aligns with visible PDF content.
• Target IDs can be referenced by AI actions.

Phase 2: Lesson Scene Graph

Tasks:

• Add lessonScenes.
• Convert existing document pages into pdf_page scenes.
• Add scene queries for the frontend.
• Add page strip that understands inserted activity scenes.

Done when:

• Lesson navigation is scene-based.
• PDF pages still display exactly as before.
• Inserted activity placeholders can appear between pages.

Phase 3: Action Generation

Tasks:

• Add sceneActions.
• Add English-only page action prompt.
• Add JSON parser and repair pass.
• Validate target IDs.
• Generate fallback scripts.

Done when:

• Each PDF page has an action sequence.
• Every visual action targets a valid pageTarget.
• The action sequence reads like a natural teacher explanation.

Phase 4: Playback Engine

Tasks:

• Add PlaybackEngine.
• Add ActionEngine.
• Execute speech and visual actions in order.
• Support pause, resume, replay, next, previous, and speed.

Done when:

• Page actions run in order.
• Visual focus happens before speech.
• Moving between pages does not reuse wrong audio.

Phase 5: Per-Action TTS

Tasks:

• Generate audio for every speech action.
• Store audio IDs and URLs on action records.
• Add TTS fallback.
• Add audio cleanup on regeneration.
• Add Google Cloud TTS SSML mark support for exact segment timings.
• Reuse the existing Google service-account JSON auth for Cloud TTS.
• Store returned SSML mark timepoints in segmentTimingsJson.

Done when:

• Every speech segment has its own correct audio.
• No page plays another page's narration.
• Playback waits for speech before continuing.
• Page-level overlays can sync from real Cloud TTS mark offsets instead of estimated character timing.

Phase 6: Accurate Overlays

Tasks:

• Add target anchors over the PDF canvas.
• Refactor overlay components to measure anchors.
• Add natural spotlight, laser, marker, circle, arrow, callout, and zoom effects.

Done when:

• Spotlight tightly frames the actual content.
• Laser lands on the target center.
• Highlights and circles align after resize.
• Debug mode can show target and overlay boxes together.

Phase 7: Whiteboard Teaching

Tasks:

• Add whiteboard state and renderer.
• Add whiteboard actions.
• Add English-only whiteboard prompt snippet.
• Add bounds and overlap validation.

Done when:

• AI can open the whiteboard and explain a formula or process step by step.
• LaTeX renders correctly.
• Whiteboard content stays inside bounds.

Phase 8: Quizzes and Activity Gates

Tasks:

• Generate quiz scenes and quiz gates.
• Save attempts.
• Show feedback.
• Block or continue playback based on activity settings.

Done when:

• AI inserts checks after hard concepts.
• Learners get immediate feedback.
• Quiz content cites source page numbers internally.

Phase 9: Interactive Widgets

Tasks:

• Add widget outline prompt.
• Generate sandboxed HTML widgets.
• Add iframe renderer.
• Add widget teacher actions.
• Add widget fallback.

Done when:

• At least one generated widget can be inserted after a relevant PDF page.
• Teacher actions can highlight or reveal widget parts.
• Widget failure does not break the lesson.

Phase 10: Tutor Chat and Review

Tasks:

• Ground chat in current page targets, nearby pages, and summaries.
• Let the learner ask for simpler explanations.
• Let the learner request extra practice.
• Let the tutor reference visible page regions.

Done when:

• Tutor answers from the PDF.
• Tutor can point the learner back to exact page targets.
• Tutor avoids unrelated explanations.

Quality Rules

1. The PDF is always the visual source of truth.
2. Do not regenerate the PDF as slides.
3. Every focus action must target a valid page target ID.
4. Do not use giant boxes unless the whole page is being summarized.
5. Use normalized page coordinates for PDF targets.
6. Use virtual 1000 by 563 coordinates for whiteboard actions.
7. Generate English only.
8. Validate every AI JSON output with Zod.
9. Repair invalid AI output once, then fallback.
10. Page failure must not kill the whole document.
11. Quizzes and widgets should support learning, not interrupt randomly.
12. Whiteboard actions should be used for step-by-step explanation, not decoration.
13. Store enough debug data to inspect why an overlay appeared.

Testing Plan

Unit Tests

• target coordinate normalization
• target merge and dedupe
• action schema validation
• invalid target ID rejection
• coverage scoring
• action to overlay geometry
• whiteboard bounds validation
• LaTeX escaping checks
• per-action audio mapping

Integration Tests

• upload PDF -> create page records
• process PDF -> create page targets
• targets -> generate lesson scenes
• scenes -> generate actions
• speech actions -> generate audio
• playback -> execute actions in order
• quiz gate -> save attempt
• widget failure -> fallback quiz

Manual Test PDFs

• text-heavy notes
• math formula notes
• diagram-heavy science notes
• table-heavy notes
• programming notes
• scanned notes
• long PDF
• PDF with mixed layouts

First Working Demo

Build a 3-page demo that proves the mechanism:

1. Upload a short PDF.
2. Process pages into targets.
3. Open page 1.
4. Spotlight the heading.
5. Narrate the heading.
6. Laser a formula or key label.
7. Highlight a definition.
8. Draw a circle around an important value.
9. Open whiteboard for a short derivation.
10. Ask a quiz question.
11. Move to page 2 automatically.

The demo is successful when the learner feels the AI is looking at the same exact PDF content they are looking at.

Implementation Priority

Build in this order:

1. pageTargets extraction and debug overlay
2. scene graph tables and queries
3. page action prompt and parser
4. action validation and fallback
5. Google Cloud TTS SSML mark timings for current page-level audio
6. playback engine
7. per-action TTS
8. target-anchor overlay system
9. whiteboard layer
10. quiz gates and attempts
11. interactive widgets
12. tutor chat target awareness
13. coverage validator and quality dashboard

Do not start with fancy animations. First make the AI target the correct content, speak naturally, and play the correct audio for the correct page.