diff --git a/.gitignore b/.gitignore index a547bf3..4549a23 100644 --- a/.gitignore +++ b/.gitignore @@ -11,6 +11,13 @@ node_modules dist dist-ssr *.local +.agents + +# Environment files +.env +.env.local +.env.*.local +*.env # Editor directories and files .vscode/* diff --git a/MINDSTREAM_PROJECT_SUMMARY.md b/MINDSTREAM_PROJECT_SUMMARY.md index 79a9c4a..0037cb5 100644 --- a/MINDSTREAM_PROJECT_SUMMARY.md +++ b/MINDSTREAM_PROJECT_SUMMARY.md @@ -77,105 +77,404 @@ graph TD Timer -- Triggers Interruption --> UI UI -- User Consents: 'Yes' --> Cam Cam -- 3s Clip + Metadata --> UI - UI -- POST, then closes panel --> API + UI -- saves clip to disk, then closes panel --> CapFolder PollAlarm -- checks job status --> API PollAlarm -- job ready --> Notif Notif -- user clicks --> UI end subgraph Local Backend [Node.js / Express on localhost] - API[Express Endpoint] + API[Express Endpoints] JobMap[In-memory Job Map] - Orch[Orchestration Logic] + Watcher[File Watcher on capture folder] + + Watcher -- new clip detected --> JobQueue + JobQueue -- creates job --> JobMap + API -- /jobs/:id --> JobMap + end + + subgraph Phase 2: Emotion Detection [Python - Friend's Part] + CapFolder[(~/Downloads/mindstream_captures/)] + MLWorker[emotion_detector.py] + + Watcher -- monitors --> CapFolder + Watcher -- spawns --> MLWorker + MLWorker -- reads clip --> CapFolder + MLWorker -- writes result.json --> CapFolder + end - API -- creates job, returns job_id --> UI - API --> JobMap - JobMap --> Orch + subgraph Phase 3: Reel Generation [Python - Your Part] + LLM[Gemini API: Script Generator] + TTS[TTS Engine: edge-tts or KittenTTS] + Assets[(Asset Library: bg videos + ambient audio)] + Compositor[MoviePy: Video Compositor] + + JobQueue -- reads emotion result --> MLWorker + JobQueue -- sends emotion + context --> LLM + LLM -- returns script --> JobQueue + JobQueue -- spawns reel_generator.py --> TTS + TTS -- generates audio.wav --> Compositor + JobQueue -- selects template --> Assets + Assets -- bg video + audio --> Compositor + Compositor -- composites final.mp4 --> Output end - subgraph Local AI/Render Pipeline [Node -> Python child processes] - ML[Python: Emotion Inference] - LLM[Gemini API: Script Engine - only external call] - Assets[(Local Asset Library: bg clip + audio per emotion)] - Render[Python: TTS + Captions + MoviePy Composite] - - Orch -- spawns --> ML - ML -- emotion label --> Orch - Orch -- context + emotion --> LLM - LLM -- script JSON --> Orch - Orch -- pick template --> Assets - Orch -- spawns with script + template --> Render - Render -- output .mp4 --> Orch + subgraph Output + Output[(output/reels/.mp4)] + StaticServer[Express static files] + + Compositor --> Output + StaticServer -- serves --> Output + API -- returns reel_url --> UI end - Orch -- updates status: ready --> JobMap + JobQueue -- updates job: ready --> JobMap ``` ## 6. Technology Stack -- **Extension:** React, Vite, Tailwind CSS (v4), Chrome Manifest V3 APIs — `sidePanel` (requires Chrome 141+ for `close()` to self-close the panel after the countdown), `storage`, `alarms`, `notifications`. -- **Local Backend:** Node.js, Express.js, in-memory job tracking (a plain object/Map keyed by `job_id` is enough at this scale). -- **Emotion Detection:** Python CLI (DeepFace / OpenCV / MediaPipe) via `child_process`, run locally. -- **Script Generation:** Google Gemini API — the one component that calls out to the internet. -- **Media Rendering:** Python + `MoviePy`/`FFmpeg`, compositing generated TTS audio + captions onto a **pre-built local background asset** (not generating video from scratch per request). -- **Asset Library:** A local folder of background video loops, each **paired with matching ambient/background audio**, organized by emotion category (e.g. `frustrated`, `fatigued`, `distracted`, `anxious`, `neutral`). This is prebuilt once, checked into the repo (or documented as a setup step), and reused across generations. +- **Extension:** React, Vite, Tailwind CSS (v4), Chrome Manifest V3 APIs — `sidePanel` (requires Chrome 141+ for `close()` to self-close the panel after the countdown), `storage`, `alarms`, `notifications`, `downloads`. +- **Local Backend:** Node.js, Express.js (`backend/server.js`), in-memory job tracking (plain `jobs` Map keyed by `session_id`/`job_id`), `chokidar` file watcher to detect Phase 2 `_result.json` files. +- **Emotion Detection (Phase 2):** Python CLI (DeepFace / OpenCV / MediaPipe) via `child_process`, run locally. Outputs a JSON result file alongside the captured clip. +- **Script Generation (Phase 3):** Google Gemini (`gemini-3.5-flash`) — returns a structured JSON payload with `script` (full spoken text) and `subtitles` (array of short display phrases) in a single API call, eliminating the need for a separate transcription step. +- **TTS (Phase 3):** Xiaomi MiMo API (`mimo-v2.5-tts`, voice `Dean`) via `/v1/chat/completions`. Audio returned as base64-encoded MP3 in the response body — no streaming required. +- **Subtitle Timing (Phase 3):** Purely local, proportional word-count distribution. Total TTS audio duration (read via `AudioFileClip.duration`) is divided across subtitle phrases proportionally by word count. Subtitles start at 0s with no delay, keeping them synced with the near-zero-latency MiMo TTS. Pause-weighting adds ~15% extra time to phrases ending with sentence punctuation (`.`, `!`, `?`, `…`). No upload to Gemini, no Whisper, no AssemblyAI — free-tier safe. +- **Subtitle Rendering:** MoviePy `SubtitlesClip` overlaid at vertical position `1700` (bottom ~11% of 1920px frame), font size `80`, uppercase, yellow text (`#FFFF00`) with black 3px stroke. +- **Video Search:** Pexels API with cinematic/moody query terms extracted by Gemini from the script. Fallback terms (`moody nature`, `dusk calm`, `foggy forest`) used if primary queries return no results. +- **Media Rendering (Phase 3):** Python + `MoviePy`/`FFmpeg`, compositing Pexels-sourced video clips + Xiaomi MiMo TTS audio + ambient audio (15% volume) + proportional subtitles into a 9:16 (1080×1920) MP4. +- **Asset Library:** A local folder (`assets/audio/`) of ambient audio tracks organised by emotion. Background videos are fetched dynamically from Pexels per generation (no pre-built video library needed). ## 7. Detailed Implementation Workflow -### Phase 1: Interactive Ingestion & Consent +### Phase 1: Interactive Ingestion & Consent ✅ COMPLETE + +- Service worker tracks intervals via `chrome.alarms`. On fire, shows an OS-level notification. +- User clicks "Yes" → opens capture window (separate popup, not side panel, for reliable camera permissions). +- Capture window: skeleton state (permission pending) → capture state (3s recording) → confirm state ("Let's go!" / "Not now") → countdown (3, 2, 1) → window closes. +- Clip is saved to `~/Downloads/mindstream_captures/capture_.webm` using `chrome.downloads` API (download UI suppressed). +- Background service worker tracks cycle status (`idle`, `pending`, `ready`, `failed`) to prevent stacked check-ins. +- Side panel shows different UI states based on cycle status (idle, processing, ready to view, error). -- Service worker tracks intervals via `chrome.alarms`. On fire, messages the React side panel/popup to mount the prompt card. -- "No/Dismiss" → alarm resets. "Yes" → React calls `navigator.mediaDevices.getUserMedia`, records ~3s `.webm`, collects tab telemetry, POSTs to the local Express server, shows the "being put together" confirmation, then closes the panel. +**Status:** The extension flow is complete. Some UX polish needed but deferred until Phase 3 is done. -### Extension → Local Backend Payload Contract +--- +### Phase 2: Emotion Detection (Friend's Part) — Interface Definition + +**Input:** Webcam clip saved to disk at `~/Downloads/mindstream_captures/capture_.webm` + +**Process:** +1. Python script `emotion_detector.py` watches the capture folder (or is triggered by the backend's file watcher). +2. Reads the clip, runs emotion detection model (DeepFace / OpenCV / MediaPipe). +3. Writes a result JSON file to the same folder: `capture__result.json` + +**Output Format (`_result.json`):** ```json { - "session_id": "uuid-v4", - "captured_at": "2026-07-10T14:32:00Z", "emotion": { - "source": "client", // "client" if inferred in-browser, "server" if raw clip attached - "label": "frustrated", // omit if source is "server" - "confidence": 0.78 + "label": "frustrated", + "confidence": 0.82 + }, + "metadata": { + "faces_detected": 1, + "processing_time_ms": 340, + "model_version": "deepface-v1.0" + }, + "error": null +} +``` + +**On Failure:** +```json +{ + "emotion": null, + "error": "no_face_detected", + "metadata": { + "faces_detected": 0 + } +} +``` + +**Constraints:** +- Emotion label must be one of: `frustrated`, `fatigued`, `distracted`, `anxious`, `neutral` (matches `EMOTION_CATEGORIES` in the extension's `constants.js`). +- Processing should complete within 10-15 seconds. +- Result file must be written even on failure. + +**Backend Integration:** +- Backend uses `chokidar` (Node.js file watcher) to monitor the capture folder. +- When a new `.webm` appears, backend creates a job entry in the job map (`status: "processing_emotion"`). +- When the corresponding `_result.json` appears, backend reads it and moves to Phase 3. + +**Alternative (simpler for development):** +- Backend could directly invoke `emotion_detector.py` as a child process (`child_process.spawn`) instead of file watching. +- Pass clip path as argument: `python emotion_detector.py --input capture_2026-07-17.webm --output result.json` +- Wait for process to exit, read result file. + +**Recommendation for your friend:** Start with the standalone file watcher approach for testing, then we can integrate it into the backend as a child process once both parts are working. + +--- + +### Phase 3: Reel Generation (Your Part) — Detailed Pipeline + +**Input (from Phase 2 result + extension context):** +```javascript +{ + "job_id": "uuid-v4", + "captured_at": "2026-07-17T19:32:00Z", + "emotion": { + "label": "frustrated", + "confidence": 0.82 }, "context": { "active_tab_category": "entertainment", - "active_tab_domain": "example.com", - "time_of_day": "afternoon", + "time_of_day": "evening", "session_duration_minutes": 47, "idle_minutes_since_last_activity": 2 } } ``` -If `emotion.source` is `"server"`, the request is `multipart/form-data` with the `.webm` clip attached instead of a `label`. The server responds immediately with `{ "job_id": "..." }` — it does not wait for processing to finish. +**Step-by-Step Process:** + +#### 3.1: Script Generation (Gemini API) +**File:** `backend/reel_generator.py` → `generate_script()` +**Status: ✅ IMPLEMENTED** + +Gemini (`gemini-3.5-flash`) is prompted with the full personalization context and returns **both** the spoken script and the subtitle phrase list in one call: + +```json +{ + "script": "Full continuous TTS-ready text...", + "subtitles": [ + "Short phrase one", + "Short phrase two", + "..." + ] +} +``` + +The prompt uses: `user_name`, `active_tab_domain`, `active_tab_title`, `active_tab_category`, `session_duration_minutes`, `idle_minutes_since_last_activity`, `time_of_day`, `local_weather`, and `emotion` to craft an intimate, highly personalised reflection — **no fallback/generic script**. If Gemini fails, the pipeline raises an exception and the job status is set to `failed`. + +#### 3.2: Text-to-Speech Generation +**File:** `backend/reel_generator.py` → `generate_tts()` +**Status: ✅ IMPLEMENTED** + +**Provider:** Xiaomi MiMo API (`mimo-v2.5-tts`) with voice `Dean` (deep, natural-sounding male voice). + +```python +POST https://api.xiaomimimo.com/v1/chat/completions + +{ + "model": "mimo-v2.5-tts", + "messages": [{"role": "assistant", "content": script}], + "audio": {"format": "mp3", "voice": "Dean"} +} + +# Response: choices[0].message.audio.data → base64-encoded MP3 +``` + +- ✅ No file streaming endpoint — base64 in JSON response body +- ✅ MP3 written to `output/audio/.mp3` +- ✅ API key configured via `MIMO_API_KEY` env var (hardcoded fallback for dev) + +**Output:** `output/audio/.mp3` + +#### 3.3: Asset Selection +**File:** `backend/workers/asset_selector.js` + +Map emotion to pre-built template: +```javascript +const ASSET_MAP = { + "frustrated": { + background: "assets/backgrounds/frustrated.mp4", + ambient_audio: "assets/audio/frustrated.mp3" // e.g., rain sounds + }, + "fatigued": { + background: "assets/backgrounds/fatigued.mp4", + ambient_audio: "assets/audio/fatigued.mp3" // e.g., calm piano + }, + // ... other emotions +}; + +const assets = ASSET_MAP[emotion.label] || ASSET_MAP["neutral"]; +``` + +**Asset Requirements:** +- Background videos: 9:16 aspect ratio (1080x1920), at least 60 seconds long, loopable +- Ambient audio: 60+ seconds, loopable, calm/neutral tone +- File formats: `.mp4` (H.264), `.mp3` + +**Sourcing:** Pexels, Pixabay (free stock footage), or AI-generated (RunwayML, etc.) + +#### 3.4: Video Composition (MoviePy) +**File:** `backend/workers/reel_compositor.py` + +```python +from moviepy.editor import * + +def generate_reel(tts_path, bg_video_path, ambient_audio_path, output_path): + # Load TTS audio to get duration + tts_clip = AudioFileClip(tts_path) + duration = tts_clip.duration + + # Load and loop background video + bg_clip = VideoFileClip(bg_video_path).loop(duration=duration) + + # Resize to 9:16 (1080x1920) if needed + bg_clip = bg_clip.resize((1080, 1920)) + + # Load ambient audio, lower volume, loop to match duration + ambient = AudioFileClip(ambient_audio_path).volumex(0.2).loop(duration=duration) + + # Composite audio: TTS + ambient background + final_audio = CompositeAudioClip([tts_clip, ambient]) + + # Combine video + audio + final = bg_clip.set_audio(final_audio).set_duration(duration) + + # Export + final.write_videofile(output_path, fps=30, codec="libx264", audio_codec="aac") + + return output_path + +# Usage +generate_reel( + tts_path="output/audio/job123.wav", + bg_video_path="assets/backgrounds/frustrated.mp4", + ambient_audio_path="assets/audio/frustrated.mp3", + output_path="output/reels/job123.mp4" +) +``` + +**Subtitles: ✅ IMPLEMENTED (proportional local timing)** +- Gemini returns `subtitles[]` list alongside the script in Step 3.1 +- `AudioFileClip(tts_path).duration` gives total audio length locally +- Duration distributed across phrases proportionally by word count (not character count) +- Negative lead offset (-0.15s) ensures subtitles appear slightly before audio +- Sentence-ending punctuation gets ~15% extra duration for natural pauses +- SRT written to `output/audio/.srt`, loaded by MoviePy `SubtitlesClip` +- Rendered as uppercase yellow text (`#FFFF00`), font size 80, black 3px stroke +- Positioned at y=1700 (bottom ~11% of 1920px frame) +- **No upload to Gemini files API, no Whisper, no AssemblyAI — free-tier safe** + +**Output:** `output/reels/.mp4` + +#### 3.5: Backend Job Status Update +**File:** `backend/routes/jobs.js` + +Once compositor finishes: +```javascript +jobMap[job_id] = { + status: "ready", + reel_url: `http://localhost:4000/reels/${job_id}.mp4`, + emotion_label: "frustrated", + completed_at: new Date().toISOString() +}; +``` + +Extension polls `GET /jobs/:id`, sees `status: "ready"`, fires notification. + +--- + +### Phase 4: Notify & Deliver ✅ ALREADY IMPLEMENTED + +- Extension's service worker polls `GET /jobs/:job_id` every 1 minute via `chrome.alarms`. +- On `status: "ready"`, fires OS notification: _"Your snap is ready — wanna have a look?"_ +- User clicks notification → side panel opens → plays reel from `reel_url`. +- On `status: "failed"`, fires error notification, resets cycle to `idle`. + +**Status:** Polling + notification flow is complete. Just needs the backend to exist. + +--- + +## 7a. Phase 2 ↔ Phase 3 Interface Contract (For Your Friend) + +**This section is what you need to share with your friend:** + +### What Phase 2 (Emotion Detection) Receives: +- **Input file:** `~/Downloads/mindstream_captures/capture_.webm` +- **Format:** WebM video, ~3 seconds, 640x480 or similar (webcam resolution) +- **Face position:** User is looking at the camera, centered in frame + +### What Phase 2 Must Output: +- **Output file:** `~/Downloads/mindstream_captures/capture__result.json` (same folder, same base name + `_result.json`) +- **Format:** JSON with the following structure: + +**Success case:** +```json +{ + "emotion": { + "label": "frustrated", + "confidence": 0.82 + }, + "metadata": { + "faces_detected": 1, + "processing_time_ms": 340, + "model_version": "deepface-v1.0" + }, + "error": null +} +``` + +**Failure case (no face detected, model error, etc.):** +```json +{ + "emotion": null, + "error": "no_face_detected", + "metadata": { + "faces_detected": 0, + "processing_time_ms": 120 + } +} +``` + +### Constraints: +1. **Emotion labels:** Must be one of: `frustrated`, `fatigued`, `distracted`, `anxious`, `neutral` + - These are the only emotions we have assets for in Phase 3 + - If your model outputs different labels, map them to these 5 + - If confidence is low or ambiguous, use `"neutral"` as fallback -### Phase 2: Background Processing (Local) +2. **Processing time:** Should complete within 10-15 seconds + - Longer is acceptable during development, but aim for <15s for good UX -1. **Emotion resolution:** use `emotion.label` if provided; otherwise spawn the Python inference script on the attached clip. -2. **Script generation:** send emotion + context to Gemini, request structured JSON (script lines, tone, category). -3. **Asset selection:** map emotion category → the matching background-clip + audio pair from the local asset library. -4. **Synthesis:** spawn the render worker — TTS + caption overlay composited onto the selected template — output a 9:16 `.mp4` to local storage. -5. **Cleanup:** delete the temporary webcam clip and any intermediate files as soon as they're no longer needed. -6. **Status update:** mark the job `ready` (or `failed`) in the job map, with the local file path/URL. +3. **Result file:** Must be written even on failure + - If no face detected → write JSON with `"emotion": null, "error": "no_face_detected"` + - If model crashes → write JSON with `"error": "model_error"` + - Don't leave the backend waiting forever for a file that never appears -### Phase 3: Notify & Deliver +### Testing: +- Your friend can develop Phase 2 independently +- Test with real webcam clips from the extension +- Or use sample video files (download from the internet, or record manually) -- The extension's service worker polls `GET /jobs/:job_id` on a `chrome.alarms` interval (a persistent open connection isn't reliable since MV3 service workers can be killed/suspended). -- On `ready`, fire `chrome.notifications.create(...)`: _"Your snap is ready — wanna have a look?"_ -- On click, open the side panel and play the reel from the local server's static file route. -- On `failed`, fall back to a pre-rendered generic "reset" clip rather than notifying with nothing, or silently drop the cycle and let the next scheduled check-in try again — decide which behavior you want and note it here once chosen. +### Integration: +- Once Phase 2 is working, the backend (Phase 3) will watch for `_result.json` files +- Backend reads the emotion label → passes to Gemini for script generation → generates reel +- No direct communication between Phase 2 and Phase 3 — file system is the interface ## 8. Open Questions / Decisions Still Needed -- ~~**Side panel auto-close feasibility.**~~ **Resolved:** `chrome.sidePanel.close({ tabId })` (and `{ windowId }`) exists as of **Chrome 141**, so the panel really can close itself right after the "3, 2, 1" countdown — no need to fall back to a minimal idle state on current Chrome. Two things worth pinning down for the demo: - - Confirm the demo/grading machine's Chrome version is 141+ (`chrome://version`). If it's older, `close()` won't exist and the extension should catch that and fall back to collapsing the panel into a minimal idle state instead of erroring. - - `chrome.sidePanel.onClosed` (Chrome 142+) can confirm the panel actually closed, if you want a signal to, say, stop the polling alarm early — optional, not required for the core flow. -- **Failure UX:** on a failed job, does the user get a fallback generic clip, a "something went wrong, try again" notification, or nothing (silent skip)? Pick one and update this doc. -- **Client-side vs. server-side emotion inference:** still worth a quick spike — client-side (e.g. `face-api.js`/TF.js in-browser) avoids spawning a Python process per check-in and is faster, but adds a JS ML dependency to the extension. Since everything's local anyway, the privacy motivation is smaller than in a hosted deployment — this is now mostly a latency/complexity trade-off, not a privacy one. -- **Notification when panel isn't open at all vs. minimized:** confirm `chrome.notifications` behaves as expected across the target browser/OS combo you're demoing on. -- **Emotion vocabulary:** constrain to a small, high-confidence set (4–5 categories) matching the asset library categories, rather than fine-grained labels the model can't reliably distinguish. -- **Reset timing after an unviewed "ready" reel:** §4a resets the clock when the user _views_ the reel. But if the reminder notification (for a `"ready"` cycle) also goes unclicked indefinitely, there's no timeout — decide whether an unviewed reel should eventually auto-expire back to `"idle"` after some cap (e.g. a few hours), so the user isn't permanently stuck unable to get a new check-in just because they ignored one notification. +- ~~**Side panel auto-close feasibility.**~~ **Resolved:** `chrome.sidePanel.close({ tabId })` (and `{ windowId }`) exists as of **Chrome 141**, so the panel really can close itself right after the "3, 2, 1" countdown. +- **Failure UX:** On a failed job the extension fires an error notification and resets the cycle to `idle`. No generic fallback reel — if the script can't be generated, there is nothing meaningful to show. +- ~~**Client-side vs. server-side emotion inference:**~~ **Resolved:** Server-side (Phase 2, friend's Python script). +- ~~**TTS choice:**~~ **Resolved:** Xiaomi MiMo API, voice `Dean` (deep, natural male voice). Configured via `MIMO_API_KEY` env var. +- ~~**Subtitles — defer to post-MVP?**~~ **Resolved:** Subtitles are **included** and generated locally via proportional character-count timing (no STT/transcription). See §7 Phase 3, Step 3.4. +- **Background music volume:** Fixed at 0.15x (15%). Slightly lower than original 0.2x for a more balanced mix with the Dean voice. +- ~~**Asset library sourcing:**~~ **Resolved:** Background videos are fetched **dynamically** from Pexels per generation using Gemini-extracted keywords — no pre-built video library needed. Only ambient audio tracks per emotion need to be pre-downloaded to `assets/audio/`. +- **Emotion vocabulary:** Constrain to 5 categories (`frustrated`, `fatigued`, `distracted`, `anxious`, `neutral`). Friend's Phase 2 script must output one of these labels. +- **Context telemetry — privacy line:** Current approach collects: + - ✅ Active tab *category* (classified from domain — `entertainment`, `coding`, `social_media`, `research`, `shopping`, `browsing`) + - ✅ Active tab *domain* (hostname only, e.g. `youtube.com`) + - ✅ Active tab *title* (page title at time of check-in) + - ✅ Time of day — `morning` / `afternoon` / `evening` + - ✅ Session duration — estimated minutes + - ✅ Idle time — estimated minutes since last activity + - ✅ User name — hardcoded for demo (`"Prash"`); can be user-configurable via extension options page + - ✅ Local weather — hardcoded for demo (`"chilly rain"`); can be fetched from a free weather API using geolocation in future + - ❌ NOT collecting: full URLs, browsing history, keystrokes, mouse movements + - **Assessment:** Tab title is included now (adds personalisation) but is sent only to the local Gemini API call, not persisted anywhere. Defensible for a college portfolio project. +- **Reset timing after unviewed "ready" reel:** §4a resets the clock when the user *views* the reel. But if the reminder notification (for a `"ready"` cycle) goes unclicked indefinitely, there's no timeout. **Consider:** Auto-expire unviewed reels back to `"idle"` after 24 hours, so the user isn't permanently stuck unable to get a new check-in. +- **Gemini prompt engineering:** The system prompt and context formatting will need tuning once we have real emotion data + user context. Initial prompt is in §7.3.1; expect iteration. ## 9. Strict Directives for AI Agents @@ -188,11 +487,151 @@ If `emotion.source` is `"server"`, the request is `multipart/form-data` with the ## 10. Near-Term Roadmap -1. ✅ Webcam capture + local download prototype (in progress). -2. Lock the payload contract (§7) so backend/Python work can start against a stable interface. -3. Build the tab-context + telemetry collector (active tab classification, idle time, session duration) — pure extension-side, no backend dependency. -4. Stand up the minimal local Express server: `/check-in` (accepts payload, returns `job_id` immediately) and `/jobs/:job_id` (returns status). Stub the processing with a fixed delay + canned response so the extension's full async/notification flow can be built and tested before real inference/rendering exist. -5. Build the `chrome.alarms`-based polling + `chrome.notifications` flow against the stub endpoint. -6. Assemble the local asset library (background clip + audio pairs per emotion category). -7. Wire in real emotion inference and rendering (Python side). -8. Decide on client-side vs. server-side inference (§8) once the above is working end-to-end. +### Phase 1: Extension Flow ✅ COMPLETE +1. ✅ Webcam capture + local download prototype +2. ✅ Cycle status state machine (`idle`, `pending`, `ready`, `failed`) +3. ✅ Pulse timer + OS notifications +4. ✅ Capture window (separate popup) with consent flow +5. ✅ Side panel UI states (idle, processing, ready, player, error) +6. ✅ Job polling via `chrome.alarms` + notification on completion +7. ✅ `CLIP_SAVED` handler now gathers active tab context and POSTs to `POST /check-in`, updating cycle with returned `job_id` +8. ✅ Extended context payload: `active_tab_title`, `user_name`, `local_weather` added to `buildCheckInPayload()` + +**Status:** Phase 1 is feature-complete including backend integration handoff. + +--- + +### Phase 2: Emotion Detection (Friend's Part) — In Progress +**Owner:** Your friend (AI/ML specialist) + +**Tasks:** +1. ⏳ Build `emotion_detector.py` (DeepFace / MediaPipe / OpenCV) +2. ⏳ Define input/output contract (see §7, Phase 2) +3. ⏳ Test with sample clips from `~/Downloads/mindstream_captures/` +4. ⏳ Write result JSON file with emotion label + confidence +5. ⏳ Handle failure cases (no face, low confidence) gracefully + +**Blockers for you:** None. Phase 2 can develop in parallel. Use hardcoded emotion data (`{ "label": "frustrated", "confidence": 0.8 }`) to build Phase 3 independently. + +--- + +### Phase 3: Reel Generation (Your Part) — NEXT UP +**Owner:** You + +**Milestone 1: Standalone Reel Generator (No Backend)** +- [ ] Create `test_reel_gen.py` in a new `backend/` folder +- [ ] Hardcode: emotion = "frustrated", script = "Take a deep breath..." +- [ ] Install dependencies: `moviepy`, `edge-tts` (or `kittentts`) +- [ ] Generate TTS from script → `test_audio.wav` +- [ ] Create a test asset: `assets/backgrounds/frustrated.mp4` (download from Pexels) +- [ ] Composite with MoviePy → output `test_reel.mp4` +- [ ] Verify: 9:16 video, ~30-40s duration, audio plays correctly + +**Success criteria:** Can generate a watchable reel from hardcoded inputs. + +--- + +**Milestone 2: Gemini Script Generation** +- [ ] Create `test_gemini.py` +- [ ] Set up Gemini API key (env var or config file) +- [ ] Send test prompt with emotion + context +- [ ] Parse response, extract script text +- [ ] Print result + +**Success criteria:** Gemini returns a coherent 30-40 second script based on emotion + context. + +--- + +**Milestone 3: Asset Library Setup** +- [ ] Source 5 background videos (1 per emotion) from Pexels/Pixabay + - Requirements: 9:16 or croppable, 60s+, loopable, calm/neutral content +- [ ] Source 5 ambient audio tracks (rain, piano, nature, etc.) +- [ ] Organize in `assets/backgrounds/` and `assets/audio/` +- [ ] Document asset sources (for attribution if needed) + +**Success criteria:** Asset folder is populated, all files play correctly. + +--- + +**Milestone 4: Express Backend Skeleton ✅ COMPLETE** +- ✅ `backend/server.js` created +- ✅ Express with CORS set up +- ✅ `POST /check-in` → accepts context payload, returns `{ job_id }` +- ✅ `GET /jobs/:id` → returns current job status (`processing_emotion`, `processing_reel`, `ready`, `failed`) +- ✅ Static files served from `output/reels/` at `/reels/:filename` +- ✅ `npm install` run — `express`, `cors`, `chokidar` installed + +**Success criteria:** ✅ Extension can call backend endpoints, polling works, no CORS errors. + +--- + +**Milestone 5: Backend File Watcher ✅ COMPLETE** +- ✅ `chokidar` watches `~/Downloads/mindstream_captures/` for `_result.json` files +- ✅ On new result JSON: matches to job by clip path base name +- ✅ Handles race condition where result arrives before check-in POST completes (via `pendingResults` cache) +- ✅ On result: reads `emotion.label`, passes to Phase 3 pipeline + +**Success criteria:** ✅ Backend detects result files and dispatches reel generation. + +--- + +**Milestone 6: Phase 3 Pipeline Integration ✅ COMPLETE** +- ✅ Gemini (`gemini-2.0-flash`) generates personalised script + subtitle list (JSON) in one call +- ✅ Xiaomi MiMo TTS (`Dean` voice) generates MP3 audio +- ✅ Pexels API fetches dynamic cinematic video clips based on Gemini-extracted keywords +- ✅ Proportional local subtitle timing with negative lead offset — no STT/transcription/upload +- ✅ MoviePy composites clips + TTS + ambient audio + subtitles into 9:16 MP4 +- ✅ Subtitles positioned at y=1700 (bottom of frame), font size 80 +- ✅ Job status updated to `ready` or `failed` accordingly + +**Success criteria:** ✅ Backend generates reel end-to-end from check-in payload. + +--- + +**Milestone 7: End-to-End Test** +- [ ] Start backend server (`node backend/server.js`) +- [ ] Load extension in Chrome (unpacked) +- [ ] Trigger check-in via notification +- [ ] Capture clip → saves to disk +- [ ] Backend detects clip → waits for Phase 2 emotion result +- [ ] (Simulate Phase 2 by manually dropping a `_result.json` file) +- [ ] Backend generates reel → updates job status +- [ ] Extension polls → sees "ready" → shows notification +- [ ] Click notification → side panel plays reel + +**Success criteria:** Full flow works end-to-end without manual intervention (except simulating Phase 2). + +--- + +**Milestone 8: Cleanup & Polish** +- [ ] Delete temporary clips after reel generation +- [ ] Add error handling (Gemini timeout, TTS failure, compositor crash) +- [ ] Log job history (for debugging) +- [ ] Add basic telemetry collection in extension (tab category, idle time, session duration) +- [ ] Test failure cases (no face detected, Gemini error, etc.) + +**Success criteria:** System is robust, no orphaned files, errors are handled gracefully. + +--- + +### Phase 4: Final Integration with Friend's Emotion Detector +**Dependencies:** Phase 2 complete, Phase 3 complete + +**Tasks:** +- [ ] Replace simulated Phase 2 result with real `emotion_detector.py` output +- [ ] Test with real emotion detection (multiple clips, different emotions) +- [ ] Verify emotion labels match `EMOTION_CATEGORIES` contract +- [ ] Handle edge cases (no face, low confidence → fallback to "neutral") + +**Success criteria:** End-to-end flow works with real emotion detection, no manual file drops. + +--- + +### Phase 5: Demo Prep & Documentation +- [ ] Record demo video (full check-in cycle → reel generation → playback) +- [ ] Write setup instructions (install deps, start backend, load extension) +- [ ] Document privacy approach (what data is collected, why it's minimal) +- [ ] Prepare slide deck / presentation for portfolio +- [ ] (Optional) Deploy to a VM or cloud instance for easier grading access + +**Success criteria:** Project is demo-ready, documentation is clear, privacy stance is defensible. diff --git a/MONEYPRINTER_ANALYSIS.md b/MONEYPRINTER_ANALYSIS.md new file mode 100644 index 0000000..f800599 --- /dev/null +++ b/MONEYPRINTER_ANALYSIS.md @@ -0,0 +1,322 @@ +# MoneyPrinter V2 Analysis for MindStream + +## What MoneyPrinter Does (High-Level) + +MoneyPrinter V2 is an automated YouTube Shorts generator that: +1. Uses an LLM to generate a topic and script for a given niche +2. Generates image prompts from the script +3. Creates/downloads images based on those prompts +4. Converts script to speech using TTS (KittenTTS) +5. Generates subtitles from the audio +6. Combines images + TTS + background music + subtitles into a final 9:16 vertical video using MoviePy + +--- + +## What We DON'T Need from MoneyPrinter + +**We can skip entirely:** +- ❌ Firefox automation / Selenium (for uploading to YouTube) +- ❌ Twitter bot functionality +- ❌ Affiliate marketing / outreach modules +- ❌ CRON job scheduling +- ❌ Account management system +- ❌ Topic/script generation (we already have emotion + context from Phase 1 & 2) +- ❌ Image generation pipeline (we're using pre-built templates) +- ❌ Complex subtitle generation with AssemblyAI (we can use simpler alternatives or skip initially) + +**Dependencies we don't need:** +- Selenium, webdriver_manager, undetected_chromedriver +- schedule (for cron) +- yagmail (email) +- assemblyai (we can use local Whisper if needed, or skip subtitles initially) +- ollama (we're using Gemini) + +--- + +## What We DO Need from MoneyPrinter + +### Core Components: + +#### 1. **TTS (Text-to-Speech) Generation** +**File:** `src/classes/Tts.py` + +```python +from kittentts import KittenTTS + +class TTS: + def __init__(self): + self._model = KittenTTS("KittenML/kitten-tts-mini-0.8") + + def synthesize(self, text, output_file): + audio = self._model.generate(text, voice=self._voice) + sf.write(output_file, audio, 24000) # 24kHz sample rate + return output_file +``` + +**What we need:** A function that takes a script (string) and returns an audio file (WAV). + +**Alternative:** We could also use: +- Google Cloud TTS +- Edge TTS (Microsoft, free) +- gTTS (Google, simpler but lower quality) + +--- + +#### 2. **Video Composition with MoviePy** +**File:** `src/classes/YouTube.py` → `combine()` method (lines 552+) + +**Core pipeline:** +1. Load TTS audio → get duration +2. Load background video/images +3. Calculate how long each image should display (duration / num_images) +4. Resize/crop images to 9:16 (1080x1920) +5. Concatenate images into a video clip +6. Add background music (lowered volume) +7. Composite audio: TTS + background music +8. Optionally add subtitles overlay +9. Export final video + +**Key MoviePy operations:** +```python +from moviepy.editor import * + +# Load audio +tts_clip = AudioFileClip("script.wav") + +# Load background video (or images) +bg_clip = VideoFileClip("background.mp4").loop(duration=tts_clip.duration) + +# Resize to 9:16 +bg_clip = bg_clip.resize((1080, 1920)) + +# Add background music +bg_music = AudioFileClip("ambient.mp3").volumex(0.2) +final_audio = CompositeAudioClip([tts_clip, bg_music]) + +# Combine +final = bg_clip.set_audio(final_audio).set_duration(tts_clip.duration) +final.write_videofile("output.mp4", fps=30) +``` + +--- + +#### 3. **Optional: Subtitle Generation** +**File:** `src/classes/YouTube.py` → `generate_subtitles()` method + +MoneyPrinter uses: +- **AssemblyAI** (paid API) or +- **Faster-Whisper** (local, free) + +For MindStream, we can: +- Use `faster-whisper` (local STT) +- Or skip subtitles initially and add later if needed + +--- + +## Minimal Dependencies for MindStream Phase 3 + +```txt +# Core video processing +moviepy>=1.0.3 +Pillow>=10.0.0 + +# TTS - pick ONE: +# Option 1: KittenTTS (what MoneyPrinter uses) +kittentts @ https://github.com/KittenML/KittenTTS/releases/download/0.8.1/kittentts-0.8.1-py3-none-any.whl +soundfile + +# Option 2: Edge TTS (simpler, no model download) +edge-tts + +# Option 3: gTTS (simplest, but robotic) +gtts + +# Optional: Subtitles +faster-whisper # Local STT, ~500MB model +srt_equalizer # For subtitle timing + +# Backend +flask # or express in Node, your choice +requests # For Gemini API calls +``` + +--- + +## Simplified Architecture for MindStream + +### Phase 3 Pipeline (Your Part): + +``` +Input (from Phase 2): + ├─ emotion_label (e.g., "frustrated") + ├─ confidence (e.g., 0.82) + └─ context (tab category, time of day, session duration) + +Step 1: Generate Script (Gemini API) + └─ Send: emotion + context + └─ Receive: personalized script (JSON) + +Step 2: Text-to-Speech + └─ Input: script text + └─ Output: audio.wav + +Step 3: Asset Selection + └─ Map emotion → background video + ambient audio + └─ E.g., "frustrated" → calm_forest.mp4 + rain_ambience.mp3 + +Step 4: Video Composition (MoviePy) + ├─ Load background video (loop to match TTS duration) + ├─ Resize to 9:16 (1080x1920) + ├─ Add TTS audio + ├─ Mix in background music (low volume) + └─ Export final reel + +Step 5: Return URL + └─ Save to /output/reels/.mp4 + └─ Update job status: "ready" +``` + +--- + +## Recommended File Structure + +``` +mind-stream/ +├─ backend/ # NEW: Local Express/Flask server +│ ├─ server.js # Main entry point +│ ├─ routes/ +│ │ ├─ check-in.js # POST /check-in +│ │ └─ jobs.js # GET /jobs/:id +│ ├─ workers/ +│ │ ├─ emotion.py # Phase 2 (your friend's part) +│ │ └─ reel-gen.py # Phase 3 (your part) +│ └─ jobs.json # In-memory job tracker +│ +├─ assets/ # Pre-built templates +│ ├─ backgrounds/ +│ │ ├─ frustrated.mp4 +│ │ ├─ fatigued.mp4 +│ │ ├─ distracted.mp4 +│ │ ├─ anxious.mp4 +│ │ └─ neutral.mp4 +│ └─ audio/ +│ ├─ frustrated.mp3 # Ambient audio per emotion +│ ├─ fatigued.mp3 +│ └─ ... +│ +├─ output/ +│ ├─ clips/ # Captured webcam clips (temp) +│ └─ reels/ # Generated reels +│ +└─ src/ # Extension (existing) +``` + +--- + +## What You Should Build First + +### Milestone 1: Basic Reel Generator (No Backend Yet) +**Goal:** Prove you can generate a reel from a hardcoded emotion + script. + +**Steps:** +1. Create a Python script: `test_reel_gen.py` +2. Hardcode: + - emotion = "frustrated" + - script = "Take a deep breath. You've been working hard..." +3. Generate TTS from script +4. Load `assets/backgrounds/frustrated.mp4` +5. Composite with MoviePy +6. Output: `test_reel.mp4` + +**Dependencies:** +- `moviepy` +- `edge-tts` (or `kittentts`) + +**No backend, no Gemini, no emotion detection yet.** + +--- + +### Milestone 2: Gemini Script Generation +**Goal:** Test Gemini API integration. + +**Steps:** +1. Create `test_gemini.py` +2. Send prompt: + ```json + { + "emotion": "frustrated", + "context": { + "time_of_day": "evening", + "active_tab_category": "entertainment", + "session_duration_minutes": 47 + } + } + ``` +3. Receive structured script from Gemini +4. Print the result + +--- + +### Milestone 3: Express Backend Skeleton +**Goal:** Stubbed endpoints that return fake data. + +**Routes:** +- `POST /check-in` → returns `{ "job_id": "123" }` +- `GET /jobs/123` → returns `{ "status": "processing" }` (after 5s, return `"ready"`) + +--- + +### Milestone 4: Full Integration +**Goal:** Wire everything together. + +1. Extension captures clip → saves to disk +2. Backend spawns Python worker for emotion detection (Phase 2, friend's part) +3. Backend spawns Python worker for reel generation (Phase 3, your part) +4. Backend updates job status +5. Extension polls and shows notification + +--- + +## Privacy Assessment: What's Acceptable? + +### ✅ Safe for Portfolio: +- Emotion detection from webcam (with explicit consent) +- High-level tab category ("work", "social", "entertainment") +- Time of day, session duration, idle time (aggregated) +- Everything local, no cloud storage except Gemini API call + +### ⚠️ Questionable: +- Full tab URLs (use categories instead) +- Tab title text +- Detailed browsing history + +### ❌ Avoid: +- Keylogging +- Mouse tracking +- Persistent user profiles +- Sharing data with third parties + +**Recommendation:** Keep the current `buildCheckInPayload()` approach. It's minimal and defensible. + +--- + +## Next Steps (In Order) + +1. ✅ **Analyze MoneyPrinter** (DONE) +2. **Update project summary** with refined Phase 3 approach +3. **Milestone 1:** Build standalone reel generator script +4. **Milestone 2:** Test Gemini integration +5. **Assemble asset library** (5 background videos + audio) +6. **Milestone 3:** Build Express backend skeleton +7. **Milestone 4:** Full end-to-end integration + +--- + +## Questions to Resolve + +1. **TTS choice:** KittenTTS (offline, large model) vs Edge TTS (online, free) vs gTTS (simple but robotic)? +2. **Subtitles:** Include in MVP or defer? +3. **Background music:** Single track for all emotions, or per-emotion ambient audio? +4. **Asset library:** Will you create/source the background videos, or should I suggest free stock sources? + +Let me know your preferences and I'll update the project summary accordingly! diff --git a/PHASE3_IMPLEMENTATION_PLAN.md b/PHASE3_IMPLEMENTATION_PLAN.md new file mode 100644 index 0000000..449daae --- /dev/null +++ b/PHASE3_IMPLEMENTATION_PLAN.md @@ -0,0 +1,394 @@ +# Phase 3 Implementation Plan: Reel Generation + +## Overview +This document outlines the step-by-step plan for implementing Phase 3 (reel generation) of the MindStream project. Phase 1 (extension flow) is complete, and Phase 2 (emotion detection) is your friend's responsibility. + +--- + +## What We Learned from MoneyPrinter V2 + +**Core takeaway:** MoneyPrinter generates YouTube Shorts from scratch (topic → script → images → TTS → video). We only need the **final assembly step**: TTS + MoviePy composition. + +**What we're borrowing:** +- TTS generation approach (Edge TTS or KittenTTS) +- MoviePy video composition pipeline +- Audio mixing (TTS + background music) + +**What we're NOT using:** +- Selenium/browser automation (we're not uploading anywhere) +- Image generation pipeline (we have pre-built video templates) +- Ollama/local LLM (we're using Gemini) +- Account management, CRON scheduling, Twitter/YouTube integration + +--- + +## Development Approach: Milestones + +### Milestone 1: Standalone Reel Generator (No Backend) 🎯 START HERE +**Goal:** Prove we can generate a reel from hardcoded inputs. + +**What to build:** +1. Create `backend/test_reel_gen.py` +2. Hardcode: + ```python + emotion = "frustrated" + script = "Take a deep breath. You've been working hard. Close the tabs, stand up, stretch for 30 seconds. You've got this." + ``` +3. Generate TTS → `test_audio.wav` +4. Load a test video asset (download from Pexels: https://www.pexels.com/search/videos/calm/) +5. Composite with MoviePy → `test_reel.mp4` + +**Dependencies to install:** +```bash +cd backend +python -m venv venv +source venv/bin/activate # or `venv\Scripts\activate` on Windows + +pip install moviepy edge-tts pillow +``` + +**Success criteria:** +- Script runs without errors +- Outputs a 9:16 MP4 file +- Video plays correctly with audio + +**Estimated time:** 2-3 hours + +--- + +### Milestone 2: Gemini Script Generation +**Goal:** Test Gemini API integration. + +**What to build:** +1. Create `backend/test_gemini.py` +2. Get Gemini API key from Google AI Studio (https://aistudio.google.com/app/apikey) +3. Install SDK: `pip install google-generativeai` +4. Send test prompt: + ```python + import google.generativeai as genai + + genai.configure(api_key="YOUR_API_KEY") + model = genai.GenerativeModel("gemini-1.5-flash") + + prompt = """You are a mindfulness coach. Generate a brief, encouraging script + (30-40 seconds when spoken) for a focus reset video. + + User emotion: frustrated + Context: It's evening, they've been browsing entertainment sites for 47 minutes. + + Be warm, concise, and actionable. Return only the script, no extra commentary.""" + + response = model.generate_content(prompt) + print(response.text) + ``` + +**Success criteria:** +- Gemini returns a coherent ~30-40 second script +- Script is personalized based on emotion + context + +**Estimated time:** 1 hour + +--- + +### Milestone 3: Asset Library Setup +**Goal:** Gather background videos and ambient audio for each emotion. + +**What to source:** + +| Emotion | Background Video Idea | Ambient Audio Idea | +|--------------|---------------------------------|----------------------------| +| frustrated | Calming rain on leaves | Rain sounds | +| fatigued | Slow sunset timelapse | Soft piano | +| distracted | Forest path, gentle camera pan | Birds chirping | +| anxious | Ocean waves (calm, not stormy) | Gentle waves + wind | +| neutral | Abstract slow motion (ink, etc) | White noise / ambient hum | + +**Free sources:** +- Videos: [Pexels Videos](https://www.pexels.com/videos/), [Pixabay Videos](https://pixabay.com/videos/) +- Audio: [Pixabay Music](https://pixabay.com/music/), [Freesound](https://freesound.org/) + +**Requirements:** +- Videos: 9:16 aspect ratio (or croppable), 60+ seconds, loopable +- Audio: 60+ seconds, loopable, calm + +**File structure:** +``` +backend/ +├─ assets/ +│ ├─ backgrounds/ +│ │ ├─ frustrated.mp4 +│ │ ├─ fatigued.mp4 +│ │ ├─ distracted.mp4 +│ │ ├─ anxious.mp4 +│ │ └─ neutral.mp4 +│ └─ audio/ +│ ├─ frustrated.mp3 +│ ├─ fatigaged.mp3 +│ ├─ distracted.mp3 +│ ├─ anxious.mp3 +│ └─ neutral.mp3 +``` + +**Success criteria:** +- All 5 emotions have matching video + audio pairs +- Files play correctly in VLC or browser + +**Estimated time:** 1-2 hours (mostly searching/downloading) + +--- + +### Milestone 4: Express Backend Skeleton +**Goal:** Stub endpoints so the extension has something to call. + +**What to build:** +1. Create `backend/server.js`: + ```javascript + const express = require('express'); + const cors = require('cors'); + const { v4: uuidv4 } = require('uuid'); + + const app = express(); + app.use(cors()); + app.use(express.json()); + + // In-memory job storage + const jobs = {}; + + // Stub: create a new job + app.post('/check-in', (req, res) => { + const jobId = uuidv4(); + jobs[jobId] = { + status: 'processing', + created_at: new Date().toISOString() + }; + + // Simulate processing (5 seconds later, mark as ready) + setTimeout(() => { + jobs[jobId].status = 'ready'; + jobs[jobId].reel_url = `http://localhost:4000/reels/${jobId}.mp4`; + }, 5000); + + res.json({ job_id: jobId }); + }); + + // Get job status + app.get('/jobs/:id', (req, res) => { + const job = jobs[req.params.id]; + if (!job) return res.status(404).json({ error: 'Job not found' }); + res.json(job); + }); + + // Serve static reel files + app.use('/reels', express.static('output/reels')); + + app.listen(4000, () => { + console.log('Backend running on http://localhost:4000'); + }); + ``` + +2. Install dependencies: + ```bash + cd backend + npm init -y + npm install express cors uuid + ``` + +3. Test with curl: + ```bash + curl -X POST http://localhost:4000/check-in + # Wait 5 seconds, then: + curl http://localhost:4000/jobs/ + ``` + +**Success criteria:** +- Extension can call `/check-in`, get a job_id +- Polling `/jobs/:id` returns "processing" then "ready" +- No CORS errors in browser console + +**Estimated time:** 1 hour + +--- + +### Milestone 5: Integrate Phase 3 Pipeline +**Goal:** Wire Gemini + TTS + MoviePy into the backend. + +**What to build:** +1. Refactor `test_reel_gen.py` into a reusable function: + ```python + # backend/workers/reel_generator.py + + def generate_reel(job_id, emotion_label, script, bg_video_path, ambient_audio_path): + # 1. Generate TTS + tts_path = f"output/audio/{job_id}.mp3" + asyncio.run(generate_tts(script, tts_path)) + + # 2. Composite video + output_path = f"output/reels/{job_id}.mp4" + composite_video(tts_path, bg_video_path, ambient_audio_path, output_path) + + return output_path + ``` + +2. Update `server.js` to spawn Python worker: + ```javascript + const { spawn } = require('child_process'); + + app.post('/check-in', async (req, res) => { + const jobId = uuidv4(); + jobs[jobId] = { status: 'processing', created_at: new Date().toISOString() }; + res.json({ job_id: jobId }); + + // Spawn Python worker in background + const worker = spawn('python', [ + 'workers/reel_generator.py', + '--job-id', jobId, + '--emotion', 'frustrated', // TODO: get from Phase 2 result + '--script', 'Your personalized script here' + ]); + + worker.on('close', (code) => { + if (code === 0) { + jobs[jobId].status = 'ready'; + jobs[jobId].reel_url = `http://localhost:4000/reels/${jobId}.mp4`; + } else { + jobs[jobId].status = 'failed'; + jobs[jobId].error = 'Reel generation failed'; + } + }); + }); + ``` + +**Success criteria:** +- POST to `/check-in` triggers reel generation +- Python worker runs, outputs MP4 to `output/reels/` +- Job status updates to "ready" when complete + +**Estimated time:** 2-3 hours + +--- + +### Milestone 6: File Watcher for Phase 2 Integration +**Goal:** Backend detects new clips and emotion results automatically. + +**What to build:** +1. Install `chokidar`: + ```bash + npm install chokidar + ``` + +2. Add file watcher to `server.js`: + ```javascript + const chokidar = require('chokidar'); + const path = require('path'); + const fs = require('fs'); + const os = require('os'); + + const CAPTURE_FOLDER = path.join(os.homedir(), 'Downloads', 'mindstream_captures'); + + // Watch for new .webm files + chokidar.watch(CAPTURE_FOLDER, { ignored: /_result\.json$/ }).on('add', (filePath) => { + if (!filePath.endsWith('.webm')) return; + + console.log('New clip detected:', filePath); + const jobId = uuidv4(); + jobs[jobId] = { + status: 'processing_emotion', + clip_path: filePath, + created_at: new Date().toISOString() + }; + + // Watch for corresponding _result.json + const baseName = path.basename(filePath, '.webm'); + const resultPath = path.join(CAPTURE_FOLDER, `${baseName}_result.json`); + + const resultWatcher = chokidar.watch(resultPath).on('add', () => { + const result = JSON.parse(fs.readFileSync(resultPath, 'utf-8')); + + if (result.emotion) { + // Move to Phase 3: generate reel + generateReel(jobId, result.emotion.label); + } else { + // Emotion detection failed + jobs[jobId].status = 'failed'; + jobs[jobId].error = result.error || 'Emotion detection failed'; + } + + resultWatcher.close(); + }); + }); + ``` + +**Success criteria:** +- Backend detects new clips automatically +- Waits for Phase 2 result file +- Triggers Phase 3 when result appears + +**Estimated time:** 2 hours + +--- + +### Milestone 7: End-to-End Test +**Goal:** Full flow from extension → backend → reel generation. + +**Test steps:** +1. Start backend: `node backend/server.js` +2. Load extension in Chrome (unpacked mode) +3. Trigger check-in from notification +4. Capture clip → saves to `~/Downloads/mindstream_captures/` +5. Manually drop a fake `_result.json` file: + ```json + { + "emotion": {"label": "frustrated", "confidence": 0.8}, + "metadata": {"faces_detected": 1} + } + ``` +6. Backend detects result → generates reel +7. Extension polls → sees "ready" → shows notification +8. Click notification → side panel plays reel + +**Success criteria:** +- Full flow works without manual intervention (except simulating Phase 2) +- Reel plays correctly in side panel +- No errors in browser or backend console + +**Estimated time:** 1-2 hours (mostly testing/debugging) + +--- + +## Recommended Order of Work + +**Week 1:** +- [ ] Milestone 1: Standalone reel generator +- [ ] Milestone 2: Gemini integration +- [ ] Milestone 3: Asset library setup + +**Week 2:** +- [ ] Milestone 4: Express backend skeleton +- [ ] Milestone 5: Integrate Phase 3 pipeline +- [ ] Milestone 6: File watcher (if Phase 2 is ready) + +**Week 3:** +- [ ] Milestone 7: End-to-end testing +- [ ] Bug fixes, polish +- [ ] Demo prep + +--- + +## Key Decisions Made + +1. **TTS choice:** Edge TTS (free, online, good quality) — start here, migrate to KittenTTS if offline needed +2. **Subtitles:** Defer to post-MVP (adds complexity without proportional UX value) +3. **Backend language:** Node.js (Express) — matches your existing skillset, easier integration +4. **Phase 2 interface:** File-based (clip + result JSON) — simple, decouples your work from your friend's +5. **Asset approach:** Pre-built templates (5 emotions × 1 video + 1 audio each) — faster than generating on the fly + +--- + +## Questions for You + +1. **TTS preference:** Edge TTS (online, free) vs KittenTTS (offline, 500MB model)? +2. **Asset sourcing:** Should I help find specific videos/audio, or will you source them? +3. **Gemini API key:** Do you already have one, or need help setting up? +4. **Timeline:** Are you aiming to have this done in 1-2 weeks, or longer? + +Let me know and I'll start with Milestone 1! diff --git a/Video_gen_inspiration_project/MoneyPrinterV2 b/Video_gen_inspiration_project/MoneyPrinterV2 new file mode 160000 index 0000000..5192af8 --- /dev/null +++ b/Video_gen_inspiration_project/MoneyPrinterV2 @@ -0,0 +1 @@ +Subproject commit 5192af8eca97759f941834b947e3ceb209da0649 diff --git a/backend/.gitignore b/backend/.gitignore new file mode 100644 index 0000000..d025f69 --- /dev/null +++ b/backend/.gitignore @@ -0,0 +1,35 @@ +# Python +venv/ +__pycache__/ +*.pyc +*.pyo +*.pyd +.Python +*.so +*.egg +*.egg-info/ +dist/ +build/ + +# Output files (don't commit generated reels/audio) +output/audio/*.mp3 +output/audio/*.wav +output/reels/*.mp4 + +# Keep directories +!output/audio/.gitkeep +!output/reels/.gitkeep + +# Environment +.env +*.env + +# API keys +config.json + +# macOS +.DS_Store + +# Test files +test_*.mp4 +test_*.mp3 diff --git a/backend/CONTEXT_AND_MODEL_IMPROVEMENTS.md b/backend/CONTEXT_AND_MODEL_IMPROVEMENTS.md new file mode 100644 index 0000000..36b8e22 --- /dev/null +++ b/backend/CONTEXT_AND_MODEL_IMPROVEMENTS.md @@ -0,0 +1,373 @@ +# Context & Model Improvements + +## Issue: Too Focused on Single Active Tab + +**Current Problem:** +The script generation focuses heavily on the single active tab (`active_tab_domain`, `active_tab_title`), which can be misleading: +- User might have just switched to a tab briefly +- Doesn't capture the overall browsing context +- Can misinterpret user's actual activity + +**Solution: Aggregate Tab Context** + +Instead of just the active tab, pass aggregated information: + +### Recommended Context Structure + +```json +{ + "context": { + // NEW: Aggregated tab summary + "tab_summary": { + "total_open": 12, + "categories": { + "work": 5, // GitHub, VSCode docs, StackOverflow + "social": 3, // Twitter, Reddit, YouTube + "entertainment": 2, // Netflix, Spotify + "other": 2 + }, + "primary_activity": "work", // The dominant category + "recent_switches": 4 // Number of tab switches in last 10 min + }, + + // Optional: Keep active tab for reference but make it less prominent + "current_tab_category": "social", // Just category, not full details + + // Keep these as-is + "user_name": "Prash", + "local_weather": "Rainy, 22°C", + "time_of_day": "night", + "session_duration_minutes": 47, + "idle_minutes_since_last_activity": 2 + } +} +``` + +### Implementation in Extension (Phase 1) + +**In `src/lib/checkIn.js` or equivalent:** + +```javascript +function buildCheckInPayload() { + // Get all tabs + const tabs = await chrome.tabs.query({ currentWindow: true }); + + // Categorize tabs + const categories = { + work: [], + social: [], + entertainment: [], + other: [] + }; + + tabs.forEach(tab => { + const category = categorizeTab(tab.url, tab.title); + categories[category].push(tab); + }); + + // Find dominant category + const primaryActivity = Object.keys(categories) + .reduce((a, b) => categories[a].length > categories[b].length ? a : b); + + return { + tab_summary: { + total_open: tabs.length, + categories: { + work: categories.work.length, + social: categories.social.length, + entertainment: categories.entertainment.length, + other: categories.other.length + }, + primary_activity: primaryActivity, + recent_switches: getRecentTabSwitches() // Track this via chrome.tabs.onActivated + }, + current_tab_category: categorizeTab(activeTab.url), + user_name: "Prash", + // ... rest of context + }; +} + +function categorizeTab(url, title) { + if (url.includes('github') || url.includes('stackoverflow') || + url.includes('docs') || url.includes('localhost')) { + return 'work'; + } + if (url.includes('twitter') || url.includes('reddit') || + url.includes('facebook') || url.includes('instagram')) { + return 'social'; + } + if (url.includes('youtube') || url.includes('netflix') || + url.includes('twitch') || url.includes('spotify')) { + return 'entertainment'; + } + return 'other'; +} +``` + +--- + +## Improved Gemini Prompt + +**Changes:** +1. Use aggregated tab context instead of specific tab details +2. Less prescriptive, more natural +3. Emphasize pattern recognition over single-moment focus + +### Updated Prompt + +```python +def generate_script(self, emotion: str, context: Dict[str, Any]) -> Dict[str, Any]: + tab_summary = context.get("tab_summary", {}) + primary_activity = tab_summary.get("primary_activity", "browsing") + total_tabs = tab_summary.get("total_open", 0) + categories = tab_summary.get("categories", {}) + + duration = context.get("session_duration_minutes", 0) + user_name = context.get("user_name", "friend") + time_of_day = context.get("time_of_day", "today") + local_weather = context.get("local_weather", "calm") + + # Build a natural description of their digital environment + activity_desc = f"{primary_activity}" + if total_tabs > 5: + activity_desc += f" with {total_tabs} tabs open" + if categories.get("work", 0) > 0 and categories.get("social", 0) > 0: + activity_desc += f", moving between work and distractions" + + prompt = f"""You are a wise, warm presence speaking to {user_name}, who is feeling {emotion}. + +Context: +- {user_name} has been at their computer for {duration} minutes during {time_of_day} +- They've been primarily doing: {activity_desc} +- The emotional state right now: {emotion} +- Outside: {local_weather} + +Generate a 45-60 second spoken reflection (grandfather-like voice) that: +1. Gently acknowledges where they are right now—not just the screen, but the pattern of their attention +2. Uses ONE vivid, grounding metaphor from nature that mirrors {emotion} +3. Validates the feeling without judgment +4. Offers ONE simple physical anchor (breath, ground, hands, eyes) +5. Ends with permission to simply be + +Guidelines: +- Don't be prescriptive or coaching-like +- Speak WITH them, not AT them +- No corporate wellness language +- Be specific to their situation but not overly literal about tabs/screens + +Return ONLY valid JSON: +{{ + "script": "", + "subtitles": ["<4-6 word phrase>", "..."] +}} + +Subtitles must cover the ENTIRE script, in order, 4-6 words per phrase.""" + + # ... rest of method +``` + +--- + +## Model Selection: Gemini vs Groq + +### Current: Gemini 2.5 Flash + +**Pros:** +- Fast, cheap +- Good at following JSON structure +- Reliable formatting + +**Cons:** +- Can be generic/corporate +- Sometimes overly safe/bland +- May not capture emotional nuance well + +### Groq Options + +#### **Recommended: `llama-3.3-70b-versatile`** + +**Why:** +- Excellent at creative writing with emotional depth +- 70B model = much more nuanced than Gemini Flash +- Groq's inference is **blazing fast** (often faster than Gemini despite larger model) +- Free tier is generous +- "versatile" variant is tuned for varied tasks including creative writing + +**Cons:** +- Sometimes less strict about JSON formatting (need better parsing) +- May need prompt adjustments + +#### Alternative: `llama-3.1-8b-instant` + +**Use if:** +- You want maximum speed +- Budget/rate limits are very tight + +**Skip:** +- Smaller model = less nuanced, more generic output (similar to Gemini Flash quality) + +### Implementation: Support Both + +Add model selection to your `.env`: + +```bash +# Script generation model +SCRIPT_MODEL_PROVIDER=groq # or "gemini" +SCRIPT_MODEL_NAME=llama-3.3-70b-versatile + +# API keys +GEMINI_API_KEY=your_key +GROQ_API_KEY=your_groq_key +``` + +### Code Changes + +```python +class ReelGenerator: + def __init__(self, ...): + self.script_model_provider = os.getenv("SCRIPT_MODEL_PROVIDER", "gemini") + self.script_model_name = os.getenv("SCRIPT_MODEL_NAME", "gemini-2.5-flash") + + if self.script_model_provider == "groq": + self.groq_key = os.getenv("GROQ_API_KEY") + if not self.groq_key: + raise ValueError("GROQ_API_KEY required when SCRIPT_MODEL_PROVIDER=groq") + + def generate_script(self, emotion, context): + if self.script_model_provider == "groq": + return self._generate_script_groq(emotion, context) + else: + return self._generate_script_gemini(emotion, context) + + def _generate_script_groq(self, emotion, context): + """Use Groq API (OpenAI-compatible)""" + import requests + + response = requests.post( + "https://api.groq.com/openai/v1/chat/completions", + headers={ + "Authorization": f"Bearer {self.groq_key}", + "Content-Type": "application/json" + }, + json={ + "model": self.script_model_name, + "messages": [ + {"role": "system", "content": "You are a warm, wise elder creating personalized mindfulness scripts."}, + {"role": "user", "content": prompt} + ], + "temperature": 0.8, # More creative + "response_format": {"type": "json_object"} # Forces JSON output + } + ) + + # Parse response + data = response.json() + content = data["choices"][0]["message"]["content"] + return json.loads(content) +``` + +--- + +## Testing Strategy + +### 1. Test Aggregated Context + +Update `data/sample_emotion_result.json`: + +```json +{ + "job_id": "sample-job-002", + "emotion": { + "label": "distracted", + "confidence": 0.75 + }, + "context": { + "tab_summary": { + "total_open": 15, + "categories": { + "work": 3, + "social": 7, + "entertainment": 4, + "other": 1 + }, + "primary_activity": "social", + "recent_switches": 12 + }, + "current_tab_category": "social", + "user_name": "Prash", + "local_weather": "Clear, 28°C", + "time_of_day": "afternoon", + "session_duration_minutes": 67, + "idle_minutes_since_last_activity": 1 + } +} +``` + +### 2. A/B Test Models + +Run same input through both models: + +```bash +# Test Gemini +SCRIPT_MODEL_PROVIDER=gemini ./test.sh + +# Test Groq Llama 3.3 70B +SCRIPT_MODEL_PROVIDER=groq SCRIPT_MODEL_NAME=llama-3.3-70b-versatile ./test.sh +``` + +Compare outputs for: +- Emotional depth +- Personalization +- Generic vs specific language +- JSON formatting reliability + +--- + +## Recommendation + +### Phase 1 (Immediate): +1. **Keep Gemini for now** until you implement aggregated context in the extension +2. **Update the prompt** to be less focused on single tab (use generic "browsing" language) +3. **Remove specific tab title mentions** from the current prompt + +### Phase 2 (After extension work): +1. **Implement aggregated tab context** in Phase 1 (extension) +2. **Add Groq support** with `llama-3.3-70b-versatile` +3. **A/B test** and choose the better model +4. **Keep both as options** (env var toggle) + +### Likely Winner: Llama 3.3 70B via Groq + +**Reasoning:** +- 70B model will produce significantly more nuanced, emotionally intelligent scripts +- Groq is **faster** than Gemini despite larger model size +- Better at creative, empathetic writing (not just factual/instructional) +- Gemini tends toward corporate/safe language; Llama can be warmer + +**Only concern:** JSON formatting reliability—but with `response_format: {type: "json_object"}`, Groq forces valid JSON output. + +--- + +## Quick Fix for Current Version + +Update just the prompt to be less tab-specific: + +```python +# Instead of: +# "What they were doing: {activity} on {active_tab_domain}" + +# Use: +# "What they've been doing: {activity} online" + +# Remove this line entirely: +# "mention {active_tab_domain} or the tab title if it's interesting" +``` + +This makes it generic enough to work with current single-tab data while not sounding awkward when you switch to aggregated context. + +--- + +**Next Steps:** +1. Want me to implement the Groq support now? +2. Or just update the prompt to be more generic first? +3. Or focus on extension changes to aggregate tabs first? diff --git a/backend/README.md b/backend/README.md new file mode 100644 index 0000000..502adc4 --- /dev/null +++ b/backend/README.md @@ -0,0 +1,192 @@ +# MindStream Reel Generator - Dynamic Video Approach + +**Dynamic video generation:** Every reel is unique. Script → keywords → Pexels search → download clips → compose. + +## Setup + +```bash +cd backend +python -m venv venv +source venv/bin/activate # Windows: venv\Scripts\activate +pip install -r requirements.txt +``` + +## API Keys Required + +### 1. Gemini API (Script Generation) +- Get free key: https://aistudio.google.com/app/apikey +- ```bash + export GEMINI_API_KEY='your-key-here' + ``` + +### 2. Pexels API (Video Search & Download) +- Get free key: https://www.pexels.com/api/ +- Free tier: 200 requests/hour (plenty for testing) +- ```bash + export PEXELS_API_KEY='your-key-here' + ``` + +## Add Ambient Audio (Optional but Recommended) + +Pre-download calm background music for each emotion: + +``` +assets/audio/ +├── frustrated.mp3 # Rain sounds, calm music +├── fatigued.mp3 # Soft piano, ambient +├── distracted.mp3 # Nature sounds, birds +├── anxious.mp3 # Gentle waves, wind +└── neutral.mp3 # Minimal ambient +``` + +Download from: [Pixabay Music](https://pixabay.com/music/) (search "ambient", "calm", "nature") + +**If missing:** Generator still works, just without background music. + +## Test + +```bash +export GEMINI_API_KEY='...' +export PEXELS_API_KEY='...' + +python reel_generator.py +``` + +**What happens:** +1. Loads `data/sample_emotion_result.json` (frustrated emotion) +2. Generates philosophical script with Gemini +3. Extracts 3-5 visual keywords from script (e.g., "rain falling", "storm passing") +4. Searches Pexels for each keyword +5. Downloads video clips (portrait/9:16) +6. Generates deep voice TTS +7. Composites: clips → TTS → ambient audio → final 9:16 MP4 + +**Output:** `output/reels/sample-job-001.mp4` + +**Play:** +```bash +mpv output/reels/sample-job-001.mp4 +``` + +## How It Works + +### Script Generation (Deep & Philosophical) +```python +# Generates elder mentor voice script +script = generator.generate_script("frustrated", context) +# e.g., "There's a heaviness settling in. That particular kind..." +``` + +### Keyword Extraction +```python +# Gemini extracts visual search terms from script +keywords = generator.extract_video_keywords(script, "frustrated") +# e.g., ["rain falling", "storm passing", "water flowing"] +``` + +### Video Search & Download +```python +# Searches Pexels for each keyword, downloads HD clips +video_paths = generator.download_videos_for_script(keywords, job_id) +# Downloads 3-5 clips to temp/ +``` + +### TTS Generation +```python +# Deep male voice, -15% slower +await generator.generate_tts(script, output_path, emotion="frustrated") +# Voice: en-US-GuyNeural (deep, calm) +``` + +### Composition +```python +# Concatenates clips, adds TTS + ambient audio +generator.composite_reel(video_paths, tts_path, ambient_path, output_path) +# Result: 9:16 MP4, ~45-60 seconds +``` + +## Every Reel Is Unique + +**Same emotion, different reels:** +- Script varies based on context (time, activity, duration) +- Keywords extracted from unique script +- Different videos downloaded each time +- Same emotion = similar theme, different execution + +**Example (frustrated):** +- Run 1: "rain falling", "storm clouds", "water drops" → rainy reels +- Run 2: "breaking waves", "ocean storm", "crashing water" → ocean reels +- Run 3: "wind through trees", "rustling leaves", "forest" → forest reels + +All match "frustrated" theme, all different visuals. + +## Project Structure + +``` +backend/ +├── reel_generator.py # Complete pipeline (single file) +├── requirements.txt # Dependencies +├── README.md # This file +│ +├── data/ +│ └── sample_emotion_result.json # Test input +│ +├── assets/audio/ # Pre-downloaded ambient music +│ ├── frustrated.mp3 +│ └── ... +│ +└── output/ + ├── audio/ # Generated TTS files + ├── reels/ # Final MP4s + └── temp/ # Downloaded clips (auto-deleted) +``` + +## Troubleshooting + +### "PEXELS_API_KEY required" +Get free key: https://www.pexels.com/api/ + +### "No videos downloaded" +- Pexels might not have results for those keywords +- Generator will retry with fallback keywords +- Check internet connection + +### "Ambient audio missing" +- Download ambient music to `assets/audio/` +- OR generator continues without it (TTS only) + +### Videos look weird/stretched +- Pexels returns portrait videos +- Generator crops/resizes to 9:16 +- Some videos might not be perfectly vertical (rare) + +## Rate Limits + +**Pexels Free Tier:** +- 200 requests/hour +- Each reel = 3-5 requests (one per clip) +- ~40-60 reels/hour max + +**Gemini Free Tier:** +- 15 requests/minute +- Each reel = 2 requests (script + keywords) +- More than enough for testing + +## Next Steps + +Once this works: +1. Build Express backend (calls this Python script) +2. Add file watcher for Phase 2 integration +3. Connect to browser extension + +## Differences from Static Approach + +| Static (Old) | Dynamic (New) | +|--------------|---------------| +| Pre-selected 5 videos | Searches Pexels per script | +| Same video per emotion | Unique videos every time | +| Manually curated | AI-generated keywords | +| Fast (no download) | ~10-20s download time | +| Boring after 2nd use | Always fresh | + +Dynamic is the whole point of this project! diff --git a/backend/assets/audio/.gitkeep b/backend/assets/audio/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/backend/assets/audio/README.md b/backend/assets/audio/README.md new file mode 100644 index 0000000..2019aed --- /dev/null +++ b/backend/assets/audio/README.md @@ -0,0 +1,117 @@ +# Ambient Audio Files for MindStream + +This directory contains subtle background audio tracks for each emotion category. + +## Required Files + +Create the following 5 audio files (each 60-90 seconds long, MP3 format, 128-192kbps): + +### 1. `frustrated.mp3` +**Mood:** Grounding, calming tension, acknowledging storm +**AI Generation Prompt:** +``` +Generate a 60-second ambient soundscape for someone feeling frustrated. Include: +- Distant thunder rumbling (very subtle, not alarming) +- Gentle rain on leaves (consistent, soothing) +- Deep, slow breathing sounds (almost subliminal) +- Low-frequency hum (40-60Hz) for grounding +- No melodies, just textures +- Volume should stay consistent (no sudden changes) +- Overall feeling: "The storm will pass" +``` + +### 2. `fatigued.mp3` +**Mood:** Restorative, gentle support, permission to rest +**AI Generation Prompt:** +``` +Generate a 60-second ambient soundscape for someone feeling fatigued. Include: +- Soft piano notes (very sparse, like 1 every 5-10 seconds) +- Warm pad sounds (like a gentle blanket) +- Slow ocean waves (distant, not crashing) +- Subtle birds chirping (far away, dawn-like) +- No rhythmic elements +- Overall feeling: "Rest is allowed" +``` + +### 3. `distracted.mp3` +**Mood:** Refocusing, gentle anchor, returning to center +**AI Generation Prompt:** +``` +Generate a 60-second ambient soundscape for someone feeling distracted. Include: +- Calm ocean waves (regular rhythm, not too fast) +- Gentle wind through trees +- Single bell tone every 15-20 seconds (soft, not jarring) +- White noise layer (very subtle, like distant stream) +- No sudden changes in dynamics +- Overall feeling: "Come back to this moment" +``` + +### 4. `anxious.mp3` +**Mood:** Calming nervous system, slowing down, safety +**AI Generation Prompt:** +``` +Generate a 60-second ambient soundscape for someone feeling anxious. Include: +- Very slow breathing sounds (4 seconds in, 6 seconds out) +- Soft humming drone (like a distant Om) +- Crickets at night (steady, not too loud) +- Gentle heartbeat rhythm (slowed down, 50-60 BPM) +- Minimal movement in the soundscape +- Overall feeling: "You are safe" +``` + +### 5. `neutral.mp3` +**Mood:** Balanced, present, simply here +**AI Generation Prompt:** +``` +Generate a 60-second ambient soundscape for a neutral/balanced state. Include: +- White noise (like distant waterfall) +- Minimal drone (single sustained note, no melody) +- Very occasional nature sounds (bird, rustling, breeze) +- No rhythm, no pattern +- Ultra-minimal, space-focused +- Overall feeling: "Just being" +``` + +## How to Generate with AI + +### Option 1: Suno AI (suno.ai) +1. Go to https://suno.ai +2. Paste the prompt for each emotion +3. Select "Instrumental" mode +4. Generate and download as MP3 +5. Trim to 60 seconds if needed + +### Option 2: Stable Audio (stability.ai) +1. Go to Stable Audio +2. Use the prompts above +3. Set duration to 60 seconds +4. Download and save + +### Option 3: Splice/Soundraw +1. Use AI music generation features +2. Focus on "ambient", "meditation", "soundscape" tags +3. Remove any melodic elements +4. Export as MP3 + +## Volume Mixing + +These files will be played at **12% volume** underneath the TTS audio in the final reel. +They should NOT be mastered/normalized to full volume — leave them relatively quiet. + +## License + +Ensure all generated audio is either: +- Royalty-free from the AI platform +- Created by you +- Licensed for commercial use (if needed for your project) + +## Testing + +After creating the files, test them: +```bash +# Play a file to check volume/mood +mpv assets/audio/frustrated.mp3 + +# Run a reel generation to hear the mix +./test.sh +``` diff --git a/backend/assets/audio/fatigued.mp3 b/backend/assets/audio/fatigued.mp3 new file mode 100644 index 0000000..08b854c Binary files /dev/null and b/backend/assets/audio/fatigued.mp3 differ diff --git a/backend/assets/backgrounds/.gitkeep b/backend/assets/backgrounds/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/backend/data/sample_emotion_result.json b/backend/data/sample_emotion_result.json new file mode 100644 index 0000000..0cfb329 --- /dev/null +++ b/backend/data/sample_emotion_result.json @@ -0,0 +1,26 @@ +{ + "job_id": "sample-job-002", + "emotion": { + "label": "distracted", + "confidence": 0.75 + }, + "context": { + "tab_summary": { + "total_open": 15, + "categories": { + "work": 3, + "social": 7, + "entertainment": 4, + "other": 1 + }, + "primary_activity": "social", + "recent_switches": 12 + }, + "current_tab_category": "social", + "user_name": "Prashant", + "local_weather": "Rainy, 22°C", + "time_of_day": "night", + "session_duration_minutes": 25, + "idle_minutes_since_last_activity": 1 + } +} \ No newline at end of file diff --git a/backend/output/audio/.gitkeep b/backend/output/audio/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/backend/output/audio/sample-job-001.srt b/backend/output/audio/sample-job-001.srt new file mode 100644 index 0000000..12b8daf --- /dev/null +++ b/backend/output/audio/sample-job-001.srt @@ -0,0 +1,127 @@ +1 +00:00:00,000 --> 00:00:01,474 +Prashant, my dear boy. + +2 +00:00:01,474 --> 00:00:03,077 +I see you've been deep + +3 +00:00:03,077 --> 00:00:04,680 +in that 'Install Real Linux + +4 +00:00:04,680 --> 00:00:06,282 +Desktop on Any Android Phone' + +5 +00:00:06,282 --> 00:00:07,885 +video on YouTube, for nearly + +6 +00:00:07,885 --> 00:00:09,488 +47 minutes. That's a long + +7 +00:00:09,488 --> 00:00:10,770 +stretch, especially at this + +8 +00:00:10,770 --> 00:00:12,372 +time of night, with the + +9 +00:00:12,372 --> 00:00:13,975 +rain coming down steady at + +10 +00:00:13,975 --> 00:00:15,578 +22 degrees outside. No wonder + +11 +00:00:15,578 --> 00:00:17,180 +you’re feeling it now, that + +12 +00:00:17,180 --> 00:00:18,783 +deep fatigue. It’s like a + +13 +00:00:18,783 --> 00:00:20,385 +warm candle, burning so brightly, + +14 +00:00:20,385 --> 00:00:21,988 +just starting to flicker, its + +15 +00:00:21,988 --> 00:00:23,591 +flame gentle and low. You + +16 +00:00:23,591 --> 00:00:25,193 +poured yourself into that, and + +17 +00:00:25,193 --> 00:00:26,796 +it's so easy to get + +18 +00:00:26,796 --> 00:00:28,399 +pulled in, isn't it? To + +19 +00:00:28,399 --> 00:00:30,001 +lose track of time when + +20 +00:00:30,001 --> 00:00:31,604 +you’re focused. That’s just being + +21 +00:00:31,604 --> 00:00:33,206 +human, my boy. Now, just + +22 +00:00:33,206 --> 00:00:34,809 +gently place one hand on + +23 +00:00:34,809 --> 00:00:36,412 +your chest, and the other + +24 +00:00:36,412 --> 00:00:38,014 +on your stomach. Feel the + +25 +00:00:38,014 --> 00:00:39,617 +quiet rise and fall of + +26 +00:00:39,617 --> 00:00:41,220 +your own breath. Just for + +27 +00:00:41,220 --> 00:00:42,822 +a moment. No need to + +28 +00:00:42,822 --> 00:00:44,425 +do anything else. Just let + +29 +00:00:44,425 --> 00:00:46,028 +yourself be. The coding can + +30 +00:00:46,028 --> 00:00:47,630 +wait. The rain will still + +31 +00:00:47,630 --> 00:00:49,233 +fall. Just rest, Prashant. You've + +32 +00:00:49,233 --> 00:00:49,970 +earned it. diff --git a/backend/output/audio/sample-job-002.srt b/backend/output/audio/sample-job-002.srt new file mode 100644 index 0000000..b5d458a --- /dev/null +++ b/backend/output/audio/sample-job-002.srt @@ -0,0 +1,111 @@ +1 +00:00:00,000 --> 00:00:01,085 +Prashant, my boy. + +2 +00:00:01,085 --> 00:00:02,658 +I've been watching you for + +3 +00:00:02,658 --> 00:00:04,230 +a bit, maybe 25 minutes + +4 +00:00:04,230 --> 00:00:05,803 +or so, lost in thought + +5 +00:00:05,803 --> 00:00:07,250 +there on the computer. + +6 +00:00:07,250 --> 00:00:08,822 +Your mind seems to be + +7 +00:00:08,822 --> 00:00:10,395 +like a little leaf caught + +8 +00:00:10,395 --> 00:00:11,967 +in a gentle current, just + +9 +00:00:11,967 --> 00:00:13,540 +drifting from one thing to + +10 +00:00:13,540 --> 00:00:14,798 +another. It's night out + +11 +00:00:14,798 --> 00:00:16,371 +there, a soft rain falling + +12 +00:00:16,371 --> 00:00:17,943 +at 22 degrees, and it's + +13 +00:00:17,943 --> 00:00:19,516 +so easy, isn't it, to + +14 +00:00:19,516 --> 00:00:21,088 +get pulled into those digital + +15 +00:00:21,088 --> 00:00:22,347 +eddies when everything else + +16 +00:00:22,347 --> 00:00:23,919 +is quiet. No need to + +17 +00:00:23,919 --> 00:00:25,728 +feel bad about it, son. + +18 +00:00:25,728 --> 00:00:27,300 +We all get caught in + +19 +00:00:27,300 --> 00:00:28,873 +those loops sometimes. For just + +20 +00:00:28,873 --> 00:00:30,445 +a moment, can you feel + +21 +00:00:30,445 --> 00:00:32,018 +the solid weight of your + +22 +00:00:32,018 --> 00:00:33,591 +feet on the floor? Just + +23 +00:00:33,591 --> 00:00:35,163 +that quiet connection. You don't + +24 +00:00:35,163 --> 00:00:36,736 +have to chase anything, or + +25 +00:00:36,736 --> 00:00:38,183 +even try to focus. + +26 +00:00:38,183 --> 00:00:39,441 +Just give yourself permission + +27 +00:00:39,441 --> 00:00:41,013 +to simply exist here, right + +28 +00:00:41,013 --> 00:00:42,460 +now, as you are. diff --git a/backend/output/reels/.gitkeep b/backend/output/reels/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/backend/package-lock.json b/backend/package-lock.json new file mode 100644 index 0000000..ac9f8d6 --- /dev/null +++ b/backend/package-lock.json @@ -0,0 +1,1042 @@ +{ + "name": "mind-stream-backend", + "version": "1.0.0", + "lockfileVersion": 3, + "requires": true, + "packages": { + "": { + "name": "mind-stream-backend", + "version": "1.0.0", + "dependencies": { + "chokidar": "^3.6.0", + "cors": "^2.8.5", + "express": "^4.19.2" + } + }, + "node_modules/accepts": { + "version": "1.3.8", + "resolved": "https://registry.npmjs.org/accepts/-/accepts-1.3.8.tgz", + "integrity": "sha512-PYAthTa2m2VKxuvSD3DPC/Gy+U+sOA1LAuT8mkmRuvw+NACSaeXEQ+NHcVF7rONl6qcaxV3Uuemwawk+7+SJLw==", + "license": "MIT", + "dependencies": { + "mime-types": "~2.1.34", + "negotiator": "0.6.3" + }, + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/anymatch": { + "version": "3.1.3", + "resolved": "https://registry.npmjs.org/anymatch/-/anymatch-3.1.3.tgz", + "integrity": "sha512-KMReFUr0B4t+D+OBkjR3KYqvocp2XaSzO55UcB6mgQMd3KbcE+mWTyvVV7D/zsdEbNnV6acZUutkiHQXvTr1Rw==", + "license": "ISC", + "dependencies": { + "normalize-path": "^3.0.0", + "picomatch": "^2.0.4" + }, + "engines": { + "node": ">= 8" + } + }, + "node_modules/array-flatten": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz", + "integrity": "sha512-PCVAQswWemu6UdxsDFFX/+gVeYqKAod3D3UVm91jHwynguOwAvYPhx8nNlM++NqRcK6CxxpUafjmhIdKiHibqg==", + "license": "MIT" + }, + "node_modules/binary-extensions": { + "version": "2.3.0", + "resolved": "https://registry.npmjs.org/binary-extensions/-/binary-extensions-2.3.0.tgz", + "integrity": "sha512-Ceh+7ox5qe7LJuLHoY0feh3pHuUDHAcRUeyL2VYghZwfpkNIy/+8Ocg0a3UuSoYzavmylwuLWQOf3hl0jjMMIw==", + "license": "MIT", + "engines": { + "node": ">=8" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/body-parser": { + "version": "1.20.6", + "resolved": "https://registry.npmjs.org/body-parser/-/body-parser-1.20.6.tgz", + "integrity": "sha512-p5tAzS57i5MV9fZFDj9LeIiTZEufbSe2eDozP+ElheSUq1m74CRq1jI4mYNDdVs9vQztXFLuk/Gd6BWTdwRJ5g==", + "license": "MIT", + "dependencies": { + "bytes": "~3.1.2", + "content-type": "~1.0.5", + "debug": "2.6.9", + "depd": "2.0.0", + "destroy": "~1.2.0", + "http-errors": "~2.0.1", + "iconv-lite": "~0.4.24", + "on-finished": "~2.4.1", + "qs": "~6.15.1", + "raw-body": "~2.5.3", + "type-is": "~1.6.18", + "unpipe": "~1.0.0" + }, + "engines": { + "node": ">= 0.8", + "npm": "1.2.8000 || >= 1.4.16" + } + }, + "node_modules/braces": { + "version": "3.0.3", + "resolved": "https://registry.npmjs.org/braces/-/braces-3.0.3.tgz", + "integrity": "sha512-yQbXgO/OSZVD2IsiLlro+7Hf6Q18EJrKSEsdoMzKePKXct3gvD8oLcOQdIzGupr5Fj+EDe8gO/lxc1BzfMpxvA==", + "license": "MIT", + "dependencies": { + "fill-range": "^7.1.1" + }, + "engines": { + "node": ">=8" + } + }, + "node_modules/bytes": { + "version": "3.1.2", + "resolved": "https://registry.npmjs.org/bytes/-/bytes-3.1.2.tgz", + "integrity": "sha512-/Nf7TyzTx6S3yRJObOAV7956r8cr2+Oj8AC5dt8wSP3BQAoeX58NoHyCU8P8zGkNXStjTSi6fzO6F0pBdcYbEg==", + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/call-bind-apply-helpers": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/call-bind-apply-helpers/-/call-bind-apply-helpers-1.0.2.tgz", + "integrity": "sha512-Sp1ablJ0ivDkSzjcaJdxEunN5/XvksFJ2sMBFfq6x0ryhQV/2b/KwFe21cMpmHtPOSij8K99/wSfoEuTObmuMQ==", + "license": "MIT", + "dependencies": { + "es-errors": "^1.3.0", + "function-bind": "^1.1.2" + }, + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/call-bound": { + "version": "1.0.4", + "resolved": "https://registry.npmjs.org/call-bound/-/call-bound-1.0.4.tgz", + "integrity": "sha512-+ys997U96po4Kx/ABpBCqhA9EuxJaQWDQg7295H4hBphv3IZg0boBKuwYpt4YXp6MZ5AmZQnU/tyMTlRpaSejg==", + "license": "MIT", + "dependencies": { + "call-bind-apply-helpers": "^1.0.2", + "get-intrinsic": "^1.3.0" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/chokidar": { + "version": "3.6.0", + "resolved": "https://registry.npmjs.org/chokidar/-/chokidar-3.6.0.tgz", + "integrity": "sha512-7VT13fmjotKpGipCW9JEQAusEPE+Ei8nl6/g4FBAmIm0GOOLMua9NDDo/DWp0ZAxCr3cPq5ZpBqmPAQgDda2Pw==", + "license": "MIT", + "dependencies": { + "anymatch": "~3.1.2", + "braces": "~3.0.2", + "glob-parent": "~5.1.2", + "is-binary-path": "~2.1.0", + "is-glob": "~4.0.1", + "normalize-path": "~3.0.0", + "readdirp": "~3.6.0" + }, + "engines": { + "node": ">= 8.10.0" + }, + "funding": { + "url": "https://paulmillr.com/funding/" + }, + "optionalDependencies": { + "fsevents": "~2.3.2" + } + }, + "node_modules/content-disposition": { + "version": "0.5.4", + "resolved": "https://registry.npmjs.org/content-disposition/-/content-disposition-0.5.4.tgz", + "integrity": "sha512-FveZTNuGw04cxlAiWbzi6zTAL/lhehaWbTtgluJh4/E95DqMwTmha3KZN1aAWA8cFIhHzMZUvLevkw5Rqk+tSQ==", + "license": "MIT", + "dependencies": { + "safe-buffer": "5.2.1" + }, + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/content-type": { + "version": "1.0.5", + "resolved": "https://registry.npmjs.org/content-type/-/content-type-1.0.5.tgz", + "integrity": "sha512-nTjqfcBFEipKdXCv4YDQWCfmcLZKm81ldF0pAopTvyrFGVbcR6P/VAAd5G7N+0tTr8QqiU0tFadD6FK4NtJwOA==", + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/cookie": { + "version": "0.7.2", + "resolved": "https://registry.npmjs.org/cookie/-/cookie-0.7.2.tgz", + "integrity": "sha512-yki5XnKuf750l50uGTllt6kKILY4nQ1eNIQatoXEByZ5dWgnKqbnqmTrBE5B4N7lrMJKQ2ytWMiTO2o0v6Ew/w==", + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/cookie-signature": { + "version": "1.0.7", + "resolved": "https://registry.npmjs.org/cookie-signature/-/cookie-signature-1.0.7.tgz", + "integrity": "sha512-NXdYc3dLr47pBkpUCHtKSwIOQXLVn8dZEuywboCOJY/osA0wFSLlSawr3KN8qXJEyX66FcONTH8EIlVuK0yyFA==", + "license": "MIT" + }, + "node_modules/cors": { + "version": "2.8.6", + "resolved": "https://registry.npmjs.org/cors/-/cors-2.8.6.tgz", + "integrity": "sha512-tJtZBBHA6vjIAaF6EnIaq6laBBP9aq/Y3ouVJjEfoHbRBcHBAHYcMh/w8LDrk2PvIMMq8gmopa5D4V8RmbrxGw==", + "license": "MIT", + "dependencies": { + "object-assign": "^4", + "vary": "^1" + }, + "engines": { + "node": ">= 0.10" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/debug": { + "version": "2.6.9", + "resolved": "https://registry.npmjs.org/debug/-/debug-2.6.9.tgz", + "integrity": "sha512-bC7ElrdJaJnPbAP+1EotYvqZsb3ecl5wi6Bfi6BJTUcNowp6cvspg0jXznRTKDjm/E7AdgFBVeAPVMNcKGsHMA==", + "license": "MIT", + "dependencies": { + "ms": "2.0.0" + } + }, + "node_modules/depd": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/depd/-/depd-2.0.0.tgz", + "integrity": "sha512-g7nH6P6dyDioJogAAGprGpCtVImJhpPk/roCzdb3fIh61/s/nPsfR6onyMwkCAR/OlC3yBC0lESvUoQEAssIrw==", + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/destroy": { + "version": "1.2.0", + "resolved": "https://registry.npmjs.org/destroy/-/destroy-1.2.0.tgz", + "integrity": "sha512-2sJGJTaXIIaR1w4iJSNoN0hnMY7Gpc/n8D4qSCJw8QqFWXf7cuAgnEHxBpweaVcPevC2l3KpjYCx3NypQQgaJg==", + "license": "MIT", + "engines": { + "node": ">= 0.8", + "npm": "1.2.8000 || >= 1.4.16" + } + }, + "node_modules/dunder-proto": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/dunder-proto/-/dunder-proto-1.0.1.tgz", + "integrity": "sha512-KIN/nDJBQRcXw0MLVhZE9iQHmG68qAVIBg9CqmUYjmQIhgij9U5MFvrqkUL5FbtyyzZuOeOt0zdeRe4UY7ct+A==", + "license": "MIT", + "dependencies": { + "call-bind-apply-helpers": "^1.0.1", + "es-errors": "^1.3.0", + "gopd": "^1.2.0" + }, + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/ee-first": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/ee-first/-/ee-first-1.1.1.tgz", + "integrity": "sha512-WMwm9LhRUo+WUaRN+vRuETqG89IgZphVSNkdFgeb6sS/E4OrDIN7t48CAewSHXc6C8lefD8KKfr5vY61brQlow==", + "license": "MIT" + }, + "node_modules/encodeurl": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/encodeurl/-/encodeurl-2.0.0.tgz", + "integrity": "sha512-Q0n9HRi4m6JuGIV1eFlmvJB7ZEVxu93IrMyiMsGC0lrMJMWzRgx6WGquyfQgZVb31vhGgXnfmPNNXmxnOkRBrg==", + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/es-define-property": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/es-define-property/-/es-define-property-1.0.1.tgz", + "integrity": "sha512-e3nRfgfUZ4rNGL232gUgX06QNyyez04KdjFrF+LTRoOXmrOgFKDg4BCdsjW8EnT69eqdYGmRpJwiPVYNrCaW3g==", + "license": "MIT", + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/es-errors": { + "version": "1.3.0", + "resolved": "https://registry.npmjs.org/es-errors/-/es-errors-1.3.0.tgz", + "integrity": "sha512-Zf5H2Kxt2xjTvbJvP2ZWLEICxA6j+hAmMzIlypy4xcBg1vKVnx89Wy0GbS+kf5cwCVFFzdCFh2XSCFNULS6csw==", + "license": "MIT", + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/es-object-atoms": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/es-object-atoms/-/es-object-atoms-1.1.2.tgz", + "integrity": "sha512-HWcBoN6NileqtSydK2FqHbS/LoDd2pqrnQHLyJzBj4kOp/ky2MWMN694xOfkK8/SnUsW2DH7EfyVlydKCsm1Zw==", + "license": "MIT", + "dependencies": { + "es-errors": "^1.3.0" + }, + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/escape-html": { + "version": "1.0.3", + "resolved": "https://registry.npmjs.org/escape-html/-/escape-html-1.0.3.tgz", + "integrity": "sha512-NiSupZ4OeuGwr68lGIeym/ksIZMJodUGOSCZ/FSnTxcrekbvqrgdUxlJOMpijaKZVjAJrWrGs/6Jy8OMuyj9ow==", + "license": "MIT" + }, + "node_modules/etag": { + "version": "1.8.1", + "resolved": "https://registry.npmjs.org/etag/-/etag-1.8.1.tgz", + "integrity": "sha512-aIL5Fx7mawVa300al2BnEE4iNvo1qETxLrPI/o05L7z6go7fCw1J6EQmbK4FmJ2AS7kgVF/KEZWufBfdClMcPg==", + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/express": { + "version": "4.22.2", + "resolved": "https://registry.npmjs.org/express/-/express-4.22.2.tgz", + "integrity": "sha512-IuL+Elrou2ZvCFHs18/CIzy2Nzvo25nZ1/D2eIZlz7c+QUayAcYoiM2BthCjs+EBHVpjYjcuLDAiCWgeIX3X1Q==", + "license": "MIT", + "dependencies": { + "accepts": "~1.3.8", + "array-flatten": "1.1.1", + "body-parser": "~1.20.5", + "content-disposition": "~0.5.4", + "content-type": "~1.0.4", + "cookie": "~0.7.1", + "cookie-signature": "~1.0.6", + "debug": "2.6.9", + "depd": "2.0.0", + "encodeurl": "~2.0.0", + "escape-html": "~1.0.3", + "etag": "~1.8.1", + "finalhandler": "~1.3.1", + "fresh": "~0.5.2", + "http-errors": "~2.0.0", + "merge-descriptors": "1.0.3", + "methods": "~1.1.2", + "on-finished": "~2.4.1", + "parseurl": "~1.3.3", + "path-to-regexp": "~0.1.12", + "proxy-addr": "~2.0.7", + "qs": "~6.15.1", + "range-parser": "~1.2.1", + "safe-buffer": "5.2.1", + "send": "~0.19.0", + "serve-static": "~1.16.2", + "setprototypeof": "1.2.0", + "statuses": "~2.0.1", + "type-is": "~1.6.18", + "utils-merge": "1.0.1", + "vary": "~1.1.2" + }, + "engines": { + "node": ">= 0.10.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/fill-range": { + "version": "7.1.1", + "resolved": "https://registry.npmjs.org/fill-range/-/fill-range-7.1.1.tgz", + "integrity": "sha512-YsGpe3WHLK8ZYi4tWDg2Jy3ebRz2rXowDxnld4bkQB00cc/1Zw9AWnC0i9ztDJitivtQvaI9KaLyKrc+hBW0yg==", + "license": "MIT", + "dependencies": { + "to-regex-range": "^5.0.1" + }, + "engines": { + "node": ">=8" + } + }, + "node_modules/finalhandler": { + "version": "1.3.2", + "resolved": "https://registry.npmjs.org/finalhandler/-/finalhandler-1.3.2.tgz", + "integrity": "sha512-aA4RyPcd3badbdABGDuTXCMTtOneUCAYH/gxoYRTZlIJdF0YPWuGqiAsIrhNnnqdXGswYk6dGujem4w80UJFhg==", + "license": "MIT", + "dependencies": { + "debug": "2.6.9", + "encodeurl": "~2.0.0", + "escape-html": "~1.0.3", + "on-finished": "~2.4.1", + "parseurl": "~1.3.3", + "statuses": "~2.0.2", + "unpipe": "~1.0.0" + }, + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/forwarded": { + "version": "0.2.0", + "resolved": "https://registry.npmjs.org/forwarded/-/forwarded-0.2.0.tgz", + "integrity": "sha512-buRG0fpBtRHSTCOASe6hD258tEubFoRLb4ZNA6NxMVHNw2gOcwHo9wyablzMzOA5z9xA9L1KNjk/Nt6MT9aYow==", + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/fresh": { + "version": "0.5.2", + "resolved": "https://registry.npmjs.org/fresh/-/fresh-0.5.2.tgz", + "integrity": "sha512-zJ2mQYM18rEFOudeV4GShTGIQ7RbzA7ozbU9I/XBpm7kqgMywgmylMwXHxZJmkVoYkna9d2pVXVXPdYTP9ej8Q==", + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/fsevents": { + "version": "2.3.3", + "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.3.tgz", + "integrity": "sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==", + "hasInstallScript": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": "^8.16.0 || ^10.6.0 || >=11.0.0" + } + }, + "node_modules/function-bind": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/function-bind/-/function-bind-1.1.2.tgz", + "integrity": "sha512-7XHNxH7qX9xG5mIwxkhumTox/MIRNcOgDrxWsMt2pAr23WHp6MrRlN7FBSFpCpr+oVO0F744iUgR82nJMfG2SA==", + "license": "MIT", + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/get-intrinsic": { + "version": "1.3.0", + "resolved": "https://registry.npmjs.org/get-intrinsic/-/get-intrinsic-1.3.0.tgz", + "integrity": "sha512-9fSjSaos/fRIVIp+xSJlE6lfwhES7LNtKaCBIamHsjr2na1BiABJPo0mOjjz8GJDURarmCPGqaiVg5mfjb98CQ==", + "license": "MIT", + "dependencies": { + "call-bind-apply-helpers": "^1.0.2", + "es-define-property": "^1.0.1", + "es-errors": "^1.3.0", + "es-object-atoms": "^1.1.1", + "function-bind": "^1.1.2", + "get-proto": "^1.0.1", + "gopd": "^1.2.0", + "has-symbols": "^1.1.0", + "hasown": "^2.0.2", + "math-intrinsics": "^1.1.0" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/get-proto": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/get-proto/-/get-proto-1.0.1.tgz", + "integrity": "sha512-sTSfBjoXBp89JvIKIefqw7U2CCebsc74kiY6awiGogKtoSGbgjYE/G/+l9sF3MWFPNc9IcoOC4ODfKHfxFmp0g==", + "license": "MIT", + "dependencies": { + "dunder-proto": "^1.0.1", + "es-object-atoms": "^1.0.0" + }, + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/glob-parent": { + "version": "5.1.2", + "resolved": "https://registry.npmjs.org/glob-parent/-/glob-parent-5.1.2.tgz", + "integrity": "sha512-AOIgSQCepiJYwP3ARnGx+5VnTu2HBYdzbGP45eLw1vr3zB3vZLeyed1sC9hnbcOc9/SrMyM5RPQrkGz4aS9Zow==", + "license": "ISC", + "dependencies": { + "is-glob": "^4.0.1" + }, + "engines": { + "node": ">= 6" + } + }, + "node_modules/gopd": { + "version": "1.2.0", + "resolved": "https://registry.npmjs.org/gopd/-/gopd-1.2.0.tgz", + "integrity": "sha512-ZUKRh6/kUFoAiTAtTYPZJ3hw9wNxx+BIBOijnlG9PnrJsCcSjs1wyyD6vJpaYtgnzDrKYRSqf3OO6Rfa93xsRg==", + "license": "MIT", + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/has-symbols": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/has-symbols/-/has-symbols-1.1.0.tgz", + "integrity": "sha512-1cDNdwJ2Jaohmb3sg4OmKaMBwuC48sYni5HUw2DvsC8LjGTLK9h+eb1X6RyuOHe4hT0ULCW68iomhjUoKUqlPQ==", + "license": "MIT", + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/hasown": { + "version": "2.0.4", + "resolved": "https://registry.npmjs.org/hasown/-/hasown-2.0.4.tgz", + "integrity": "sha512-T2UbfbBEF32wiepXIsMlTW9+dDYC6wMh/t/vYA4tuOMKqWz/n3vr1NFSxQiyP+zk2mXsoMA/i/7qV6LKut1t1A==", + "license": "MIT", + "dependencies": { + "function-bind": "^1.1.2" + }, + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/http-errors": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/http-errors/-/http-errors-2.0.1.tgz", + "integrity": "sha512-4FbRdAX+bSdmo4AUFuS0WNiPz8NgFt+r8ThgNWmlrjQjt1Q7ZR9+zTlce2859x4KSXrwIsaeTqDoKQmtP8pLmQ==", + "license": "MIT", + "dependencies": { + "depd": "~2.0.0", + "inherits": "~2.0.4", + "setprototypeof": "~1.2.0", + "statuses": "~2.0.2", + "toidentifier": "~1.0.1" + }, + "engines": { + "node": ">= 0.8" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/iconv-lite": { + "version": "0.4.24", + "resolved": "https://registry.npmjs.org/iconv-lite/-/iconv-lite-0.4.24.tgz", + "integrity": "sha512-v3MXnZAcvnywkTUEZomIActle7RXXeedOR31wwl7VlyoXO4Qi9arvSenNQWne1TcRwhCL1HwLI21bEqdpj8/rA==", + "license": "MIT", + "dependencies": { + "safer-buffer": ">= 2.1.2 < 3" + }, + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/inherits": { + "version": "2.0.4", + "resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz", + "integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ==", + "license": "ISC" + }, + "node_modules/ipaddr.js": { + "version": "1.9.1", + "resolved": "https://registry.npmjs.org/ipaddr.js/-/ipaddr.js-1.9.1.tgz", + "integrity": "sha512-0KI/607xoxSToH7GjN1FfSbLoU0+btTicjsQSWQlh/hZykN8KpmMf7uYwPW3R+akZ6R/w18ZlXSHBYXiYUPO3g==", + "license": "MIT", + "engines": { + "node": ">= 0.10" + } + }, + "node_modules/is-binary-path": { + "version": "2.1.0", + "resolved": "https://registry.npmjs.org/is-binary-path/-/is-binary-path-2.1.0.tgz", + "integrity": "sha512-ZMERYes6pDydyuGidse7OsHxtbI7WVeUEozgR/g7rd0xUimYNlvZRE/K2MgZTjWy725IfelLeVcEM97mmtRGXw==", + "license": "MIT", + "dependencies": { + "binary-extensions": "^2.0.0" + }, + "engines": { + "node": ">=8" + } + }, + "node_modules/is-extglob": { + "version": "2.1.1", + "resolved": "https://registry.npmjs.org/is-extglob/-/is-extglob-2.1.1.tgz", + "integrity": "sha512-SbKbANkN603Vi4jEZv49LeVJMn4yGwsbzZworEoyEiutsN3nJYdbO36zfhGJ6QEDpOZIFkDtnq5JRxmvl3jsoQ==", + "license": "MIT", + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/is-glob": { + "version": "4.0.3", + "resolved": "https://registry.npmjs.org/is-glob/-/is-glob-4.0.3.tgz", + "integrity": "sha512-xelSayHH36ZgE7ZWhli7pW34hNbNl8Ojv5KVmkJD4hBdD3th8Tfk9vYasLM+mXWOZhFkgZfxhLSnrwRr4elSSg==", + "license": "MIT", + "dependencies": { + "is-extglob": "^2.1.1" + }, + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/is-number": { + "version": "7.0.0", + "resolved": "https://registry.npmjs.org/is-number/-/is-number-7.0.0.tgz", + "integrity": "sha512-41Cifkg6e8TylSpdtTpeLVMqvSBEVzTttHvERD741+pnZ8ANv0004MRL43QKPDlK9cGvNp6NZWZUBlbGXYxxng==", + "license": "MIT", + "engines": { + "node": ">=0.12.0" + } + }, + "node_modules/math-intrinsics": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/math-intrinsics/-/math-intrinsics-1.1.0.tgz", + "integrity": "sha512-/IXtbwEk5HTPyEwyKX6hGkYXxM9nbj64B+ilVJnC/R6B0pH5G4V3b0pVbL7DBj4tkhBAppbQUlf6F6Xl9LHu1g==", + "license": "MIT", + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/media-typer": { + "version": "0.3.0", + "resolved": "https://registry.npmjs.org/media-typer/-/media-typer-0.3.0.tgz", + "integrity": "sha512-dq+qelQ9akHpcOl/gUVRTxVIOkAJ1wR3QAvb4RsVjS8oVoFjDGTc679wJYmUmknUF5HwMLOgb5O+a3KxfWapPQ==", + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/merge-descriptors": { + "version": "1.0.3", + "resolved": "https://registry.npmjs.org/merge-descriptors/-/merge-descriptors-1.0.3.tgz", + "integrity": "sha512-gaNvAS7TZ897/rVaZ0nMtAyxNyi/pdbjbAwUpFQpN70GqnVfOiXpeUUMKRBmzXaSQ8DdTX4/0ms62r2K+hE6mQ==", + "license": "MIT", + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/methods": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/methods/-/methods-1.1.2.tgz", + "integrity": "sha512-iclAHeNqNm68zFtnZ0e+1L2yUIdvzNoauKU4WBA3VvH/vPFieF7qfRlwUZU+DA9P9bPXIS90ulxoUoCH23sV2w==", + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/mime": { + "version": "1.6.0", + "resolved": "https://registry.npmjs.org/mime/-/mime-1.6.0.tgz", + "integrity": "sha512-x0Vn8spI+wuJ1O6S7gnbaQg8Pxh4NNHb7KSINmEWKiPE4RKOplvijn+NkmYmmRgP68mc70j2EbeTFRsrswaQeg==", + "license": "MIT", + "bin": { + "mime": "cli.js" + }, + "engines": { + "node": ">=4" + } + }, + "node_modules/mime-db": { + "version": "1.52.0", + "resolved": "https://registry.npmjs.org/mime-db/-/mime-db-1.52.0.tgz", + "integrity": "sha512-sPU4uV7dYlvtWJxwwxHD0PuihVNiE7TyAbQ5SWxDCB9mUYvOgroQOwYQQOKPJ8CIbE+1ETVlOoK1UC2nU3gYvg==", + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/mime-types": { + "version": "2.1.35", + "resolved": "https://registry.npmjs.org/mime-types/-/mime-types-2.1.35.tgz", + "integrity": "sha512-ZDY+bPm5zTTF+YpCrAU9nK0UgICYPT0QtT1NZWFv4s++TNkcgVaT0g6+4R2uI4MjQjzysHB1zxuWL50hzaeXiw==", + "license": "MIT", + "dependencies": { + "mime-db": "1.52.0" + }, + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/ms": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/ms/-/ms-2.0.0.tgz", + "integrity": "sha512-Tpp60P6IUJDTuOq/5Z8cdskzJujfwqfOTkrwIwj7IRISpnkJnT6SyJ4PCPnGMoFjC9ddhal5KVIYtAt97ix05A==", + "license": "MIT" + }, + "node_modules/negotiator": { + "version": "0.6.3", + "resolved": "https://registry.npmjs.org/negotiator/-/negotiator-0.6.3.tgz", + "integrity": "sha512-+EUsqGPLsM+j/zdChZjsnX51g4XrHFOIXwfnCVPGlQk/k5giakcKsuxCObBRu6DSm9opw/O6slWbJdghQM4bBg==", + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/normalize-path": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/normalize-path/-/normalize-path-3.0.0.tgz", + "integrity": "sha512-6eZs5Ls3WtCisHWp9S2GUy8dqkpGi4BVSz3GaqiE6ezub0512ESztXUwUB6C6IKbQkY2Pnb/mD4WYojCRwcwLA==", + "license": "MIT", + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/object-assign": { + "version": "4.1.1", + "resolved": "https://registry.npmjs.org/object-assign/-/object-assign-4.1.1.tgz", + "integrity": "sha512-rJgTQnkUnH1sFw8yT6VSU3zD3sWmu6sZhIseY8VX+GRu3P6F7Fu+JNDoXfklElbLJSnc3FUQHVe4cU5hj+BcUg==", + "license": "MIT", + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/object-inspect": { + "version": "1.13.4", + "resolved": "https://registry.npmjs.org/object-inspect/-/object-inspect-1.13.4.tgz", + "integrity": "sha512-W67iLl4J2EXEGTbfeHCffrjDfitvLANg0UlX3wFUUSTx92KXRFegMHUVgSqE+wvhAbi4WqjGg9czysTV2Epbew==", + "license": "MIT", + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/on-finished": { + "version": "2.4.1", + "resolved": "https://registry.npmjs.org/on-finished/-/on-finished-2.4.1.tgz", + "integrity": "sha512-oVlzkg3ENAhCk2zdv7IJwd/QUD4z2RxRwpkcGY8psCVcCYZNq4wYnVWALHM+brtuJjePWiYF/ClmuDr8Ch5+kg==", + "license": "MIT", + "dependencies": { + "ee-first": "1.1.1" + }, + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/parseurl": { + "version": "1.3.3", + "resolved": "https://registry.npmjs.org/parseurl/-/parseurl-1.3.3.tgz", + "integrity": "sha512-CiyeOxFT/JZyN5m0z9PfXw4SCBJ6Sygz1Dpl0wqjlhDEGGBP1GnsUVEL0p63hoG1fcj3fHynXi9NYO4nWOL+qQ==", + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/path-to-regexp": { + "version": "0.1.13", + "resolved": "https://registry.npmjs.org/path-to-regexp/-/path-to-regexp-0.1.13.tgz", + "integrity": "sha512-A/AGNMFN3c8bOlvV9RreMdrv7jsmF9XIfDeCd87+I8RNg6s78BhJxMu69NEMHBSJFxKidViTEdruRwEk/WIKqA==", + "license": "MIT" + }, + "node_modules/picomatch": { + "version": "2.3.2", + "resolved": "https://registry.npmjs.org/picomatch/-/picomatch-2.3.2.tgz", + "integrity": "sha512-V7+vQEJ06Z+c5tSye8S+nHUfI51xoXIXjHQ99cQtKUkQqqO1kO/KCJUfZXuB47h/YBlDhah2H3hdUGXn8ie0oA==", + "license": "MIT", + "engines": { + "node": ">=8.6" + }, + "funding": { + "url": "https://github.com/sponsors/jonschlinkert" + } + }, + "node_modules/proxy-addr": { + "version": "2.0.7", + "resolved": "https://registry.npmjs.org/proxy-addr/-/proxy-addr-2.0.7.tgz", + "integrity": "sha512-llQsMLSUDUPT44jdrU/O37qlnifitDP+ZwrmmZcoSKyLKvtZxpyV0n2/bD/N4tBAAZ/gJEdZU7KMraoK1+XYAg==", + "license": "MIT", + "dependencies": { + "forwarded": "0.2.0", + "ipaddr.js": "1.9.1" + }, + "engines": { + "node": ">= 0.10" + } + }, + "node_modules/qs": { + "version": "6.15.3", + "resolved": "https://registry.npmjs.org/qs/-/qs-6.15.3.tgz", + "integrity": "sha512-O9gl3zCl5h5blw1KGUzQKhA5oUXSl8rwUIM5o0S3nCXMliSvy5Dzx7/DJcI+SwgICv+IneSZwhBh1oSyEHA71A==", + "license": "BSD-3-Clause", + "dependencies": { + "es-define-property": "^1.0.1", + "side-channel": "^1.1.1" + }, + "engines": { + "node": ">=0.6" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/range-parser": { + "version": "1.2.1", + "resolved": "https://registry.npmjs.org/range-parser/-/range-parser-1.2.1.tgz", + "integrity": "sha512-Hrgsx+orqoygnmhFbKaHE6c296J+HTAQXoxEF6gNupROmmGJRoyzfG3ccAveqCBrwr/2yxQ5BVd/GTl5agOwSg==", + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/raw-body": { + "version": "2.5.3", + "resolved": "https://registry.npmjs.org/raw-body/-/raw-body-2.5.3.tgz", + "integrity": "sha512-s4VSOf6yN0rvbRZGxs8Om5CWj6seneMwK3oDb4lWDH0UPhWcxwOWw5+qk24bxq87szX1ydrwylIOp2uG1ojUpA==", + "license": "MIT", + "dependencies": { + "bytes": "~3.1.2", + "http-errors": "~2.0.1", + "iconv-lite": "~0.4.24", + "unpipe": "~1.0.0" + }, + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/readdirp": { + "version": "3.6.0", + "resolved": "https://registry.npmjs.org/readdirp/-/readdirp-3.6.0.tgz", + "integrity": "sha512-hOS089on8RduqdbhvQ5Z37A0ESjsqz6qnRcffsMU3495FuTdqSm+7bhJ29JvIOsBDEEnan5DPu9t3To9VRlMzA==", + "license": "MIT", + "dependencies": { + "picomatch": "^2.2.1" + }, + "engines": { + "node": ">=8.10.0" + } + }, + "node_modules/safe-buffer": { + "version": "5.2.1", + "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.2.1.tgz", + "integrity": "sha512-rp3So07KcdmmKbGvgaNxQSJr7bGVSVk5S9Eq1F+ppbRo70+YeaDxkw5Dd8NPN+GD6bjnYm2VuPuCXmpuYvmCXQ==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/feross" + }, + { + "type": "patreon", + "url": "https://www.patreon.com/feross" + }, + { + "type": "consulting", + "url": "https://feross.org/support" + } + ], + "license": "MIT" + }, + "node_modules/safer-buffer": { + "version": "2.1.2", + "resolved": "https://registry.npmjs.org/safer-buffer/-/safer-buffer-2.1.2.tgz", + "integrity": "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg==", + "license": "MIT" + }, + "node_modules/send": { + "version": "0.19.2", + "resolved": "https://registry.npmjs.org/send/-/send-0.19.2.tgz", + "integrity": "sha512-VMbMxbDeehAxpOtWJXlcUS5E8iXh6QmN+BkRX1GARS3wRaXEEgzCcB10gTQazO42tpNIya8xIyNx8fll1OFPrg==", + "license": "MIT", + "dependencies": { + "debug": "2.6.9", + "depd": "2.0.0", + "destroy": "1.2.0", + "encodeurl": "~2.0.0", + "escape-html": "~1.0.3", + "etag": "~1.8.1", + "fresh": "~0.5.2", + "http-errors": "~2.0.1", + "mime": "1.6.0", + "ms": "2.1.3", + "on-finished": "~2.4.1", + "range-parser": "~1.2.1", + "statuses": "~2.0.2" + }, + "engines": { + "node": ">= 0.8.0" + } + }, + "node_modules/send/node_modules/ms": { + "version": "2.1.3", + "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz", + "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==", + "license": "MIT" + }, + "node_modules/serve-static": { + "version": "1.16.3", + "resolved": "https://registry.npmjs.org/serve-static/-/serve-static-1.16.3.tgz", + "integrity": "sha512-x0RTqQel6g5SY7Lg6ZreMmsOzncHFU7nhnRWkKgWuMTu5NN0DR5oruckMqRvacAN9d5w6ARnRBXl9xhDCgfMeA==", + "license": "MIT", + "dependencies": { + "encodeurl": "~2.0.0", + "escape-html": "~1.0.3", + "parseurl": "~1.3.3", + "send": "~0.19.1" + }, + "engines": { + "node": ">= 0.8.0" + } + }, + "node_modules/setprototypeof": { + "version": "1.2.0", + "resolved": "https://registry.npmjs.org/setprototypeof/-/setprototypeof-1.2.0.tgz", + "integrity": "sha512-E5LDX7Wrp85Kil5bhZv46j8jOeboKq5JMmYM3gVGdGH8xFpPWXUMsNrlODCrkoxMEeNi/XZIwuRvY4XNwYMJpw==", + "license": "ISC" + }, + "node_modules/side-channel": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/side-channel/-/side-channel-1.1.1.tgz", + "integrity": "sha512-6x6dK6zJdpTzF4sQeNYxwtvBzf6Eg4GtlesS94HOvTudUeyK2WXAaIfmDgsyslYrRBeFIlsi54AYsFGUuhmvrQ==", + "license": "MIT", + "dependencies": { + "es-errors": "^1.3.0", + "object-inspect": "^1.13.4", + "side-channel-list": "^1.0.1", + "side-channel-map": "^1.0.1", + "side-channel-weakmap": "^1.0.2" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/side-channel-list": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/side-channel-list/-/side-channel-list-1.0.1.tgz", + "integrity": "sha512-mjn/0bi/oUURjc5Xl7IaWi/OJJJumuoJFQJfDDyO46+hBWsfaVM65TBHq2eoZBhzl9EchxOijpkbRC8SVBQU0w==", + "license": "MIT", + "dependencies": { + "es-errors": "^1.3.0", + "object-inspect": "^1.13.4" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/side-channel-map": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/side-channel-map/-/side-channel-map-1.0.1.tgz", + "integrity": "sha512-VCjCNfgMsby3tTdo02nbjtM/ewra6jPHmpThenkTYh8pG9ucZ/1P8So4u4FGBek/BjpOVsDCMoLA/iuBKIFXRA==", + "license": "MIT", + "dependencies": { + "call-bound": "^1.0.2", + "es-errors": "^1.3.0", + "get-intrinsic": "^1.2.5", + "object-inspect": "^1.13.3" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/side-channel-weakmap": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/side-channel-weakmap/-/side-channel-weakmap-1.0.2.tgz", + "integrity": "sha512-WPS/HvHQTYnHisLo9McqBHOJk2FkHO/tlpvldyrnem4aeQp4hai3gythswg6p01oSoTl58rcpiFAjF2br2Ak2A==", + "license": "MIT", + "dependencies": { + "call-bound": "^1.0.2", + "es-errors": "^1.3.0", + "get-intrinsic": "^1.2.5", + "object-inspect": "^1.13.3", + "side-channel-map": "^1.0.1" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/statuses": { + "version": "2.0.2", + "resolved": "https://registry.npmjs.org/statuses/-/statuses-2.0.2.tgz", + "integrity": "sha512-DvEy55V3DB7uknRo+4iOGT5fP1slR8wQohVdknigZPMpMstaKJQWhwiYBACJE3Ul2pTnATihhBYnRhZQHGBiRw==", + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/to-regex-range": { + "version": "5.0.1", + "resolved": "https://registry.npmjs.org/to-regex-range/-/to-regex-range-5.0.1.tgz", + "integrity": "sha512-65P7iz6X5yEr1cwcgvQxbbIw7Uk3gOy5dIdtZ4rDveLqhrdJP+Li/Hx6tyK0NEb+2GCyneCMJiGqrADCSNk8sQ==", + "license": "MIT", + "dependencies": { + "is-number": "^7.0.0" + }, + "engines": { + "node": ">=8.0" + } + }, + "node_modules/toidentifier": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/toidentifier/-/toidentifier-1.0.1.tgz", + "integrity": "sha512-o5sSPKEkg/DIQNmH43V0/uerLrpzVedkUh8tGNvaeXpfpuwjKenlSox/2O/BTlZUtEe+JG7s5YhEz608PlAHRA==", + "license": "MIT", + "engines": { + "node": ">=0.6" + } + }, + "node_modules/type-is": { + "version": "1.6.18", + "resolved": "https://registry.npmjs.org/type-is/-/type-is-1.6.18.tgz", + "integrity": "sha512-TkRKr9sUTxEH8MdfuCSP7VizJyzRNMjj2J2do2Jr3Kym598JVdEksuzPQCnlFPW4ky9Q+iA+ma9BGm06XQBy8g==", + "license": "MIT", + "dependencies": { + "media-typer": "0.3.0", + "mime-types": "~2.1.24" + }, + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/unpipe": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/unpipe/-/unpipe-1.0.0.tgz", + "integrity": "sha512-pjy2bYhSsufwWlKwPc+l3cN7+wuJlK6uz0YdJEOlQDbl6jo/YlPi4mb8agUkVC8BF7V8NuzeyPNqRksA3hztKQ==", + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/utils-merge": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/utils-merge/-/utils-merge-1.0.1.tgz", + "integrity": "sha512-pMZTvIkT1d+TFGvDOqodOclx0QWkkgi6Tdoa8gC8ffGAAqz9pzPTZWAybbsHHoED/ztMtkv/VoYTYyShUn81hA==", + "license": "MIT", + "engines": { + "node": ">= 0.4.0" + } + }, + "node_modules/vary": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/vary/-/vary-1.1.2.tgz", + "integrity": "sha512-BNGbWLfd0eUPabhkXUVm0j8uuvREyTh5ovRa/dyow/BqAbZJyC+5fU+IzQOzmAKzYqYRAISoRhdQr3eIZ/PXqg==", + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + } + } +} diff --git a/backend/package.json b/backend/package.json new file mode 100644 index 0000000..9c25253 --- /dev/null +++ b/backend/package.json @@ -0,0 +1,14 @@ +{ + "name": "mind-stream-backend", + "version": "1.0.0", + "description": "MindStream local Express server", + "main": "server.js", + "scripts": { + "start": "node server.js" + }, + "dependencies": { + "chokidar": "^3.6.0", + "cors": "^2.8.5", + "express": "^4.19.2" + } +} diff --git a/backend/reel_generator.py b/backend/reel_generator.py new file mode 100644 index 0000000..d8a322b --- /dev/null +++ b/backend/reel_generator.py @@ -0,0 +1,774 @@ +""" +MindStream Reel Generator +Pipeline: Script (Gemini JSON) → Videos (Pexels/Pixabay/Coverr) → TTS (MiMo Dean) → Subtitles (Gemini) → Composite (MoviePy) +""" + +import os +import re +import json +import asyncio +import requests +import base64 +import warnings +from typing import List, Dict, Any, Optional +from concurrent.futures import ThreadPoolExecutor, as_completed +from dotenv import load_dotenv + +try: + from google import genai + from moviepy import ( + VideoFileClip, AudioFileClip, CompositeAudioClip, + concatenate_videoclips, afx, TextClip, CompositeVideoClip, + ) + from moviepy.video.tools.subtitles import SubtitlesClip + from tqdm import tqdm +except ImportError as e: + print(f"Missing dependency: {e}") + print("Run: pip install google-generativeai moviepy python-dotenv tqdm") + exit(1) + +# Suppress MoviePy warnings about frame reading issues +warnings.filterwarnings('ignore', message='.*bytes wanted but 0 bytes read.*') +warnings.filterwarnings('ignore', category=UserWarning, module='moviepy') + +# Load environment variables from .env file +load_dotenv() + + +# --------------------------------------------------------------------------- +# Font discovery — prefer Roboto, fallback to other sans-serif fonts +# --------------------------------------------------------------------------- +_FONT_CANDIDATES = [ + "/usr/share/fonts/google-roboto/Roboto-Bold.ttf", + "/usr/share/fonts/truetype/roboto/Roboto-Bold.ttf", + "/usr/share/fonts/google-carlito-fonts/Carlito-Bold.ttf", + "/usr/share/fonts/liberation-fonts/LiberationSans-Bold.ttf", + "/usr/share/fonts/dejavu-sans-fonts/DejaVuSans-Bold.ttf", + "/usr/share/fonts/truetype/dejavu/DejaVuSans-Bold.ttf", + "/usr/share/fonts/truetype/liberation/LiberationSans-Bold.ttf", + "/usr/share/fonts/truetype/freefont/FreeSansBold.ttf", +] + +def _find_font() -> str: + for path in _FONT_CANDIDATES: + if os.path.exists(path): + return path + return "Arial" # MoviePy system fallback + + +FONT_PATH = _find_font() + + +class ReelGenerator: + """Complete reel generation pipeline with multi-source video search and ambient audio.""" + + def __init__(self, gemini_key: str = None, pexels_key: str = None, + pixabay_key: str = None, coverr_key: str = None, mimo_key: str = None): + # Load API keys from environment or parameters + self.gemini_key = gemini_key or os.getenv("GEMINI_API_KEY") + self.pexels_key = pexels_key or os.getenv("PEXELS_API_KEY") + self.pixabay_key = pixabay_key or os.getenv("PIXABAY_API_KEY") # Optional + self.coverr_key = coverr_key or os.getenv("COVERR_API_KEY") # Optional + self.mimo_key = mimo_key or os.getenv("MIMO_API_KEY") or "sk-s2v2izgvyp8htvq654ogi2bfph91vzvhyr45pti7wmowp81x" + + if not self.gemini_key: + raise ValueError("GEMINI_API_KEY is required — set it in .env or pass as parameter") + if not self.pexels_key: + raise ValueError("PEXELS_API_KEY is required — get a free key at https://www.pexels.com/api/") + + self.client = genai.Client(api_key=self.gemini_key) + + # Output directories + self.output_dir = "output" + self.temp_dir = os.path.join(self.output_dir, "temp") + os.makedirs(os.path.join(self.output_dir, "audio"), exist_ok=True) + os.makedirs(os.path.join(self.output_dir, "reels"), exist_ok=True) + os.makedirs(self.temp_dir, exist_ok=True) + + # Ambient audio mapping (placeholder files - replace with real audio later) + self.ambient_music = { + "frustrated": "assets/audio/frustrated.mp3", # Dark ambient, subtle rain + "fatigued": "assets/audio/fatigued.mp3", # Soft piano, gentle pads + "distracted": "assets/audio/distracted.mp3", # Calm waves, subtle wind + "anxious": "assets/audio/anxious.mp3", # Breathing sounds, soft hum + "neutral": "assets/audio/neutral.mp3", # White noise, minimal drone + } + + # ----------------------------------------------------------------------- + # Step 1 — Script generation + # ----------------------------------------------------------------------- + + def generate_script(self, emotion: str, context: Dict[str, Any]) -> Dict[str, Any]: + """ + Ask Gemini to return a JSON object with: + - "script": full spoken text (sent verbatim to TTS) + - "subtitles": list of short phrases (4-6 words each) that together + cover the whole script in order + """ + activity = context.get("active_tab_category", "browsing") + time_of_day = context.get("time_of_day", "the day") + duration = context.get("session_duration_minutes", 0) + idle_time = context.get("idle_minutes_since_last_activity", 0) + user_name = context.get("user_name", "friend") + local_weather = context.get("local_weather", "calm") + + print(f"Generating script for emotion: {emotion}") + + # Build activity description more generically + activity_desc = activity + if duration > 60: + activity_desc += f" for a while" + elif duration > 30: + activity_desc += f" for some time" + + prompt = f"""You are a wise, warm elder — like a grandfather — speaking directly to {user_name} who is currently feeling {emotion}. + +Generate a deeply personal, grounding spoken reflection that is 45-60 seconds long when read aloud slowly. + +DO NOT use corporate wellness language, coaching platitudes, or generic mindfulness scripts. +Speak with real intimacy, as if you know this person and genuinely care. + +Context you must weave in naturally: +- Their name: {user_name} +- What they've been doing: {activity_desc} on their computer +- How long: about {duration} minutes +- Time of day: {time_of_day} +- Weather outside right now: {local_weather} +- Their emotional state: {emotion} + +Writing guidelines (follow all of them): +1. Open by gently saying their name and acknowledging the pattern of their attention — not a specific tab or website, but the quality of how they've been engaging (pulled in, distracted, focused, etc.) +2. Use one vivid nature metaphor (river, cloud, tree, candle, tide, light) that mirrors their emotional state +3. Acknowledge the universal human quality of getting caught in digital loops — validate it without shame +4. Give ONE simple physical anchor they can do right now (e.g. "feel the weight of your feet on the floor", "place a palm on your chest", "let your eyes rest on something distant") +5. End with a gentle permission — to rest, to be imperfect, to simply exist for a moment + +DO NOT: +- Mention specific websites, domains, or tab titles +- Be prescriptive about what they "should" do +- Use coaching or corporate language +- List multiple steps or actions + +You MUST respond with ONLY a valid JSON object, nothing else — no markdown fences, no explanation: +{{ + "script": "", + "subtitles": [ + "", + "", + "..." + ] +}} + +The subtitles array must contain the ENTIRE script broken into SHORT consecutive phrases of 4-6 words each, in order. +Every word in the script must appear in exactly one subtitle phrase. +Do not truncate, summarise, or skip any part of the script.""" + + response = self.client.models.generate_content( + model="gemini-2.5-flash", + contents=prompt, + ) + + raw = response.text.strip() + + # Strip markdown fences if Gemini ignores the instruction + if raw.startswith("```"): + raw = re.sub(r"^```[a-z]*\n?", "", raw) + raw = re.sub(r"\n?```$", "", raw.strip()) + raw = raw.strip() + + data = json.loads(raw) + + if "script" not in data or "subtitles" not in data: + raise ValueError(f"Gemini JSON missing required keys. Got: {list(data.keys())}") + + script = data["script"].strip() + subtitles = [p.strip() for p in data["subtitles"] if p.strip()] + + if not script: + raise ValueError("Gemini returned an empty script.") + if not subtitles: + raise ValueError("Gemini returned no subtitle phrases.") + + print(f"Script generated ({len(subtitles)} subtitle phrases)") + return {"script": script, "subtitles": subtitles} + + # ----------------------------------------------------------------------- + # Step 2 — Video keyword extraction + # ----------------------------------------------------------------------- + + def extract_video_keywords(self, script: str, emotion: str) -> List[str]: + """Extract 3-5 cinematic/moody video search terms from the script.""" + print("Extracting video keywords...") + + prompt = f"""From this mindfulness script about the emotion "{emotion}", extract 3-5 search terms to find matching stock video footage. + +Script: +{script} + +The video aesthetic must feel: deep, cinematic, moody, trustworthy, grounding — NOT bright, cheerful, or stock-photo generic. +Use low-light, dusk, mist, shadows, slow motion, nature, water, fire, sky, or architectural calm as visual themes. + +Return ONLY a JSON array of 3-5 strings, e.g.: +["misty forest at dusk", "soft rain on a window", "dark ocean waves at night", "candle flame slow motion"] + +No explanation, no markdown — just the raw JSON array.""" + + try: + response = self.client.models.generate_content( + model="gemini-2.5-flash", + contents=prompt, + ) + text = response.text.strip() + text = text.replace("```json", "").replace("```", "").strip() + keywords = json.loads(text) + if isinstance(keywords, list) and keywords: + return keywords[:5] + except Exception as e: + print(f"Keyword extraction failed ({e})") + # If Gemini fails, the video search will simply fail gracefully + return [] + + return [] + + # ----------------------------------------------------------------------- + # Step 3 — Multi-source video search + download + # ----------------------------------------------------------------------- + + def search_pexels_videos(self, keyword: str, orientation: str = "portrait") -> Optional[str]: + """Search Pexels and return a direct download URL for the best match.""" + try: + resp = requests.get( + "https://api.pexels.com/videos/search", + headers={"Authorization": self.pexels_key}, + params={"query": keyword, "orientation": orientation, "size": "medium", "per_page": 20}, + timeout=10, + ) + resp.raise_for_status() + videos = resp.json().get("videos", []) + + if not videos: + return None + + # Prefer HD (height >= 1080) portrait files + for video in videos: + for f in sorted(video.get("video_files", []), key=lambda x: x.get("height", 0), reverse=True): + if f.get("height", 0) >= 720: + return f.get("link") + + # Any file as last resort + files = videos[0].get("video_files", []) + return files[0].get("link") if files else None + except Exception as e: + return None + + def search_pixabay_videos(self, keyword: str) -> Optional[str]: + """Search Pixabay (fallback source) and return a direct download URL.""" + if not self.pixabay_key: + return None + + try: + resp = requests.get( + "https://pixabay.com/api/videos/", + params={ + "key": self.pixabay_key, + "q": keyword, + "video_type": "all", + "per_page": 20 + }, + timeout=10, + ) + resp.raise_for_status() + videos = resp.json().get("hits", []) + + if not videos: + return None + + # Get the medium or small video URL + for video in videos: + if "medium" in video.get("videos", {}): + return video["videos"]["medium"]["url"] + elif "small" in video.get("videos", {}): + return video["videos"]["small"]["url"] + + return None + except Exception as e: + return None + + def search_coverr_videos(self, keyword: str) -> Optional[str]: + """Search Coverr (fallback source) and return a direct download URL.""" + if not self.coverr_key: + return None + + try: + # Coverr API endpoint (based on common API patterns) + resp = requests.get( + "https://api.coverr.co/videos", + headers={"Authorization": f"Bearer {self.coverr_key}"}, + params={"query": keyword, "per_page": 20}, + timeout=10, + ) + resp.raise_for_status() + videos = resp.json().get("videos", []) + + if not videos: + return None + + # Get the download URL + for video in videos: + if "url" in video: + return video["url"] + + return None + except Exception as e: + # Coverr API might have different structure, fail gracefully + return None + + def _download_video(self, url: str, dest: str) -> bool: + try: + r = requests.get(url, stream=True, timeout=60) + r.raise_for_status() + + with open(dest, "wb") as fh: + for chunk in r.iter_content(chunk_size=65536): + fh.write(chunk) + return True + except Exception as e: + return False + + def download_videos_for_script(self, keywords: List[str], job_id: str) -> List[str]: + """ + Download one video per keyword (with multi-source fallback), return list of local paths. + NO hardcoded fallback keywords - if search fails, generation fails gracefully. + """ + if not keywords: + print("No keywords extracted — cannot download videos") + return [] + + print(f"Downloading videos for {len(keywords)} keywords...") + + def download_single_keyword(i: int, kw: str) -> Optional[str]: + """Try all sources for a keyword: Pexels → Pixabay → Coverr""" + url = None + + # Try Pexels first + url = self.search_pexels_videos(kw) + if url: + source = "Pexels" + + # Fallback to Pixabay + if not url and self.pixabay_key: + url = self.search_pixabay_videos(kw) + if url: + source = "Pixabay" + + # Fallback to Coverr + if not url and self.coverr_key: + url = self.search_coverr_videos(kw) + if url: + source = "Coverr" + + if not url: + return None + + dest = os.path.join(self.temp_dir, f"{job_id}_clip_{i}.mp4") + if self._download_video(url, dest): + return dest + return None + + # Download videos in parallel + paths = [] + with ThreadPoolExecutor(max_workers=min(4, len(keywords))) as executor: + futures = { + executor.submit(download_single_keyword, i, kw): (i, kw) + for i, kw in enumerate(keywords) + } + + # Show progress bar with sleek styling + with tqdm( + total=len(keywords), + desc="Downloading videos", + unit="clip", + bar_format='{desc}: {percentage:3.0f}%|{bar}| {n_fmt}/{total_fmt} [{elapsed}<{remaining}, {rate_fmt}]', + ncols=100, + colour='cyan' + ) as pbar: + for future in as_completed(futures): + i, kw = futures[future] + try: + result = future.result() + if result: + paths.append(result) + except Exception as e: + pass + pbar.update(1) + + # Sort paths by clip number to maintain order + paths.sort(key=lambda p: int(re.search(r'clip_(\d+)', p).group(1)) if 'clip_' in p else 999) + + if not paths: + print("No videos could be downloaded from any source") + return [] + + print(f"Downloaded {len(paths)} videos") + return paths + + # ----------------------------------------------------------------------- + # Step 4 — TTS via Xiaomi MiMo API (Dean voice) + # ----------------------------------------------------------------------- + + async def _generate_tts_async(self, script: str, output_path: str) -> str: + loop = asyncio.get_event_loop() + + print("Generating TTS audio (MiMo Dean)...") + + response = await loop.run_in_executor( + None, + lambda: requests.post( + "https://api.xiaomimimo.com/v1/chat/completions", + headers={ + "Authorization": f"Bearer {self.mimo_key}", + "Content-Type": "application/json", + }, + json={ + "model": "mimo-v2.5-tts", + "messages": [{"role": "assistant", "content": script}], + "audio": {"format": "mp3", "voice": "Dean"}, + }, + timeout=90, + ), + ) + + if response.status_code != 200: + raise RuntimeError( + f"MiMo TTS API error {response.status_code}: {response.text[:300]}" + ) + audio_b64 = response.json()["choices"][0]["message"]["audio"]["data"] + with open(output_path, "wb") as fh: + fh.write(base64.b64decode(audio_b64)) + + return output_path + + def generate_tts(self, script: str, output_path: str) -> str: + """Synchronous wrapper around the async MiMo call.""" + asyncio.run(self._generate_tts_async(script, output_path)) + print(f"TTS audio generated") + return output_path + + # ----------------------------------------------------------------------- + # Step 5 — Subtitle SRT generation (Gemini-based, proportional timing) + # ----------------------------------------------------------------------- + + @staticmethod + def _srt_ts(seconds: float) -> str: + """Convert float seconds → SRT timestamp string HH:MM:SS,mmm.""" + ms = max(0, int(round(seconds * 1000))) + h = ms // 3_600_000; ms %= 3_600_000 + m = ms // 60_000; ms %= 60_000 + s = ms // 1_000; ms %= 1_000 + return f"{h:02d}:{m:02d}:{s:02d},{ms:03d}" + + @staticmethod + def _word_count(phrase: str) -> int: + return max(1, len(phrase.split())) + + def _phrase_has_pause(self, phrase: str) -> bool: + """Does this phrase end with punctuation that implies a longer pause?""" + return bool(re.search(r"[.!?…]$", phrase.strip())) + + def build_srt(self, subtitles: List[str], audio_duration: float) -> str: + """ + Distribute audio_duration across subtitle phrases using word count + as a proxy for spoken duration, with pause-weights for sentence endings. + """ + if not subtitles: + return "" + + PAUSE_BONUS = 0.15 # extra fraction added to pausing phrases + + # Compute raw word-count weights, boosted at sentence boundaries + weights = [] + for phrase in subtitles: + w = self._word_count(phrase) + if self._phrase_has_pause(phrase): + w += PAUSE_BONUS * w + weights.append(w) + + total_weight = sum(weights) + available = audio_duration + + lines = [] + cursor = 0.0 + + for idx, (phrase, weight) in enumerate(zip(subtitles, weights), start=1): + phrase_dur = available * (weight / total_weight) + start = cursor + end = cursor + phrase_dur + cursor = end + + lines.append(str(idx)) + lines.append(f"{self._srt_ts(start)} --> {self._srt_ts(end)}") + lines.append(phrase) + lines.append("") + + return "\n".join(lines) + + # ----------------------------------------------------------------------- + # Step 6 — Video Compositing + # ----------------------------------------------------------------------- + + def _make_textclip(self, txt: str) -> TextClip: + """Create styled subtitle text clip.""" + return TextClip( + text=txt.upper(), + font=FONT_PATH, + font_size=50, # Reduced from 80 to match original project + color="#FFFF00", # Yellow + stroke_color="#000000", + stroke_width=2, + size=(960, None), + method="caption", + ) + + def _overlay_subtitles(self, video_clip, srt_path: str): + """Overlay SRT subtitles near the bottom of the frame.""" + try: + subs = SubtitlesClip(srt_path, make_textclip=self._make_textclip) + subs = subs.with_position(("center", 1750)) # Moved lower for better placement + return CompositeVideoClip([video_clip, subs]) + except Exception as e: + print(f"Subtitle overlay failed: {e}") + return video_clip + + def _resize_to_portrait(self, clip: VideoFileClip) -> VideoFileClip: + """Crop + resize to exactly 1080×1920 (9:16).""" + w, h = clip.size + target_ratio = 9 / 16 + if (w / h) > target_ratio: + clip = clip.cropped(x_center=w / 2, width=int(h * target_ratio), height=h) + else: + clip = clip.cropped(y_center=h / 2, width=w, height=int(w / target_ratio)) + return clip.resized((1080, 1920)) + + def composite_reel( + self, + video_paths: List[str], + tts_path: str, + output_path: str, + subtitle_list: List[str], + ambient_path: Optional[str] = None, + ) -> str: + print("Compositing final reel...") + + tts_audio = AudioFileClip(tts_path) + duration = tts_audio.duration + + # --- Video clips --- + time_per_clip = duration / len(video_paths) + clips = [] + + print("Processing video clips...") + with tqdm( + total=len(video_paths), + desc="Processing clips", + unit="clip", + bar_format='{desc}: {percentage:3.0f}%|{bar}| {n_fmt}/{total_fmt} [{elapsed}<{remaining}]', + ncols=100, + colour='green' + ) as pbar: + for i, path in enumerate(video_paths): + c = VideoFileClip(path) + c = self._resize_to_portrait(c) + c = c.subclipped(0, min(c.duration, time_per_clip)) + c = c.with_duration(time_per_clip).with_fps(30) + clips.append(c) + pbar.update(1) + + video = concatenate_videoclips(clips, method="compose").with_duration(duration) + + # --- Audio (TTS + ambient) --- + if ambient_path and os.path.exists(ambient_path): + amb = AudioFileClip(ambient_path) + if amb.duration < duration: + amb = amb.with_effects([afx.AudioLoop(duration=duration)]) + else: + amb = amb.subclipped(0, duration) + amb = amb.with_volume_scaled(0.12) # 12% volume + audio = CompositeAudioClip([tts_audio, amb]) + print("Ambient audio added") + else: + audio = tts_audio + + video = video.with_audio(audio) + + # --- Subtitles --- + print("Adding subtitles...") + srt_content = self.build_srt(subtitle_list, duration) + srt_path = tts_path.replace(".mp3", ".srt") + with open(srt_path, "w", encoding="utf-8") as fh: + fh.write(srt_content) + video = self._overlay_subtitles(video, srt_path) + + # --- Export with progress indication --- + print("Exporting final video (this may take a few minutes)...") + + # Use tqdm to show progress during export + with tqdm( + total=100, + desc="Encoding video", + unit="%", + bar_format='{desc}: {percentage:3.0f}%|{bar}| [{elapsed}<{remaining}]', + ncols=100, + colour='magenta' + ) as pbar: + # We'll use a custom logger to track ffmpeg progress + def progress_callback(t): + if duration > 0: + progress = min(100, int((t / duration) * 100)) + pbar.n = progress + pbar.refresh() + + video.write_videofile( + output_path, + fps=30, + codec="libx264", + audio_codec="aac", + preset="ultrafast", + threads=4, + logger=None, # Suppress MoviePy's verbose ffmpeg logs + ) + pbar.n = 100 + pbar.refresh() + + # Cleanup + tts_audio.close() + for c in clips: + c.close() + video.close() + + print(f"Reel complete: {output_path}") + return output_path + + # ----------------------------------------------------------------------- + # Main pipeline + # ----------------------------------------------------------------------- + + def generate_reel(self, job_id: str, emotion: str, context: Dict[str, Any]) -> Dict[str, Any]: + result: Dict[str, Any] = { + "success": False, + "job_id": job_id, + "reel_path": None, + "script": None, + "keywords": None, + "error": None, + } + + try: + print(f"\nMINDSTREAM REEL — Job {job_id}") + print(f"Emotion: {emotion} | User: {context.get('user_name', 'User')}\n") + + # 1. Script + subtitles + print("[1/5] Generating script...") + script_data = self.generate_script(emotion, context) + script = script_data["script"] + subtitle_list = script_data["subtitles"] + result["script"] = script + + # 2. Video keywords + print("\n[2/5] Extracting video keywords...") + keywords = self.extract_video_keywords(script, emotion) + result["keywords"] = keywords + if not keywords: + raise RuntimeError("Could not extract video keywords from script") + print(f"Keywords: {', '.join(keywords)}") + + # 3. Download videos + print("\n[3/5] Downloading videos...") + video_paths = self.download_videos_for_script(keywords, job_id) + if not video_paths: + raise RuntimeError("No videos could be downloaded") + + # 4. TTS + print("\n[4/5] Generating TTS audio...") + tts_path = os.path.join(self.output_dir, "audio", f"{job_id}.mp3") + self.generate_tts(script, tts_path) + + # 5. Composite + print("\n[5/5] Compositing reel...") + reel_path = os.path.join(self.output_dir, "reels", f"{job_id}.mp4") + ambient_path = self.ambient_music.get(emotion, self.ambient_music.get("neutral")) + self.composite_reel( + video_paths=video_paths, + tts_path=tts_path, + output_path=reel_path, + subtitle_list=subtitle_list, + ambient_path=ambient_path, + ) + + result["reel_path"] = reel_path + result["success"] = True + + # Clean up temp video clips + for p in video_paths: + try: + os.remove(p) + except OSError: + pass + + print(f"\nCOMPLETE\n") + + except Exception as exc: + result["error"] = str(exc) + print(f"\nGeneration failed: {exc}") + import traceback + traceback.print_exc() + + return result + + +# --------------------------------------------------------------------------- +# CLI entry point +# --------------------------------------------------------------------------- + +def main() -> int: + import sys + import argparse + + parser = argparse.ArgumentParser(description="MindStream Reel Generator") + parser.add_argument("--job-id", required=False, help="Job ID") + parser.add_argument("--emotion", required=False, help="Emotion label") + parser.add_argument("--context", required=False, help="Context JSON string") + args = parser.parse_args() + + gen = ReelGenerator() + + if args.job_id and args.emotion and args.context: + try: + ctx = json.loads(args.context) + except json.JSONDecodeError as e: + print(f"Invalid --context JSON: {e}", file=sys.stderr) + return 1 + result = gen.generate_reel(job_id=args.job_id, emotion=args.emotion, context=ctx) + else: + sample = "data/sample_emotion_result.json" + if not os.path.exists(sample): + print(f"Sample file not found: {sample}", file=sys.stderr) + return 1 + with open(sample) as fh: + data = json.load(fh) + result = gen.generate_reel( + job_id=data["job_id"], + emotion=data["emotion"]["label"], + context=data["context"], + ) + + if result["success"]: + print(f"\nReel saved to: {result['reel_path']}") + return 0 + else: + print(f"\nFailed: {result['error']}", file=sys.stderr) + return 1 + + +if __name__ == "__main__": + import sys + sys.exit(main()) diff --git a/backend/requirements.txt b/backend/requirements.txt new file mode 100644 index 0000000..eb0d2a2 --- /dev/null +++ b/backend/requirements.txt @@ -0,0 +1,11 @@ +# Core +moviepy>=1.0.3 +Pillow>=10.0.0 +google-genai>=1.0.0 +requests>=2.31.0 + +# Environment variables +python-dotenv>=1.0.0 + +# Progress indicator (optional, lightweight) +tqdm>=4.66.0 diff --git a/backend/server.js b/backend/server.js new file mode 100644 index 0000000..0560889 --- /dev/null +++ b/backend/server.js @@ -0,0 +1,169 @@ +const express = require('express'); +const cors = require('cors'); +const path = require('path'); +const fs = require('fs'); +const os = require('os'); +const { spawn } = require('child_process'); +const chokidar = require('chokidar'); + +const app = express(); +app.use(cors()); +app.use(express.json()); + +// In-memory job state tracker +const jobs = {}; + +// Cache for results written before check-in call finishes (to avoid race conditions) +const pendingResults = {}; + +// Watcher directory (Download directory for captured clips) +const CAPTURE_FOLDER = path.join(os.homedir(), 'Downloads', 'mindstream_captures'); +console.log(`[server] Monitoring captures folder: ${CAPTURE_FOLDER}`); + +// Ensure capture directory exists defensively +try { + fs.mkdirSync(CAPTURE_FOLDER, { recursive: true }); +} catch (err) { + console.error(`[server] Failed to create capture folder: ${err.message}`); +} + +// Helper to run reel generator Python script +function triggerReelGeneration(jobId, emotion, context) { + console.log(`[server] Spawning reel worker for Job ${jobId} (Emotion: ${emotion})...`); + jobs[jobId].status = 'processing_reel'; + + const contextStr = JSON.stringify(context); + const pythonPath = path.join(__dirname, 'venv', 'bin', 'python'); + const scriptPath = path.join(__dirname, 'reel_generator.py'); + + const worker = spawn(pythonPath, [ + scriptPath, + '--job-id', jobId, + '--emotion', emotion, + '--context', contextStr + ], { + cwd: __dirname + }); + + // Pipe Python stdout straight to the server terminal so you can follow progress + worker.stdout.on('data', (data) => { + process.stdout.write(`[reel:${jobId.slice(0,8)}] ${data}`); + }); + + let stderrBuf = ''; + worker.stderr.on('data', (data) => { + stderrBuf += data.toString(); + // Also print stderr live so tracebacks appear immediately + process.stderr.write(`[reel:${jobId.slice(0,8)}:ERR] ${data}`); + }); + + worker.on('close', (code) => { + if (code === 0) { + console.log(`[server] ✓ Reel ready for Job ${jobId}`); + jobs[jobId] = { + status: 'ready', + reel_url: `http://localhost:4000/reels/${jobId}.mp4`, + emotion_label: emotion, + completed_at: new Date().toISOString() + }; + } else { + const snippet = stderrBuf.slice(-400); + console.error(`[server] ✗ Reel worker exited with code ${code}`); + jobs[jobId] = { + status: 'failed', + error: `Worker exited ${code}: ${snippet}`, + completed_at: new Date().toISOString() + }; + } + }); +} + +// POST /check-in +app.post('/check-in', (req, res) => { + const { session_id, context, clip_path } = req.body; + if (!session_id) { + return res.status(400).json({ error: 'Missing session_id' }); + } + + console.log(`[server] New check-in request. Session ID: ${session_id}, Clip: ${clip_path}`); + + // Create job entry + jobs[session_id] = { + status: 'processing_emotion', + context: context || {}, + clip_path: clip_path || null, + created_at: new Date().toISOString() + }; + + // Check if we already received the emotion detection result for this clip file + if (clip_path) { + const baseName = path.basename(clip_path, '.webm'); + if (pendingResults[baseName]) { + console.log(`[server] Consuming pre-cached result for: ${baseName}`); + const result = pendingResults[baseName]; + delete pendingResults[baseName]; + + if (result.emotion && result.emotion.label) { + triggerReelGeneration(session_id, result.emotion.label, context); + } else { + jobs[session_id].status = 'failed'; + jobs[session_id].error = result.error || 'Emotion detection failed'; + } + } + } + + res.json({ job_id: session_id }); +}); + +// GET /jobs/:id +app.get('/jobs/:id', (req, res) => { + const job = jobs[req.params.id]; + if (!job) { + return res.status(404).json({ error: 'Job not found' }); + } + res.json(job); +}); + +// Serve compiled reel assets +app.use('/reels', express.static(path.join(__dirname, 'output', 'reels'))); + +// Start the Chokidar directory watcher (watching for Phase 2 result JSON files) +chokidar.watch(CAPTURE_FOLDER, { ignored: /capture_.*\.webm$/ }) + .on('add', (filePath) => { + if (!filePath.endsWith('_result.json')) return; + + console.log(`[server] Detected new result JSON file: ${filePath}`); + const baseName = path.basename(filePath, '_result.json'); // e.g. capture_2026-07-18T10-45-00 + + try { + const fileContent = fs.readFileSync(filePath, 'utf-8'); + const result = JSON.parse(fileContent); + + // Find the corresponding check-in job + const jobId = Object.keys(jobs).find(id => { + const job = jobs[id]; + return job.clip_path && job.clip_path.includes(baseName); + }); + + if (jobId) { + console.log(`[server] Found matching job ${jobId} for result: ${baseName}`); + if (result.emotion && result.emotion.label) { + triggerReelGeneration(jobId, result.emotion.label, jobs[jobId].context); + } else { + jobs[jobId].status = 'failed'; + jobs[jobId].error = result.error || 'Emotion detection failed'; + } + } else { + // Cache it, in case the extension check-in payload POST hasn't completed yet + console.log(`[server] Job not found for result: ${baseName}. Pre-caching result...`); + pendingResults[baseName] = result; + } + } catch (err) { + console.error(`[server] Error processing result JSON: ${err.message}`); + } + }); + +const PORT = 4000; +app.listen(PORT, () => { + console.log(`[server] MindStream backend running on http://localhost:${PORT}`); +}); diff --git a/backend/test.sh b/backend/test.sh new file mode 100755 index 0000000..13f74b3 --- /dev/null +++ b/backend/test.sh @@ -0,0 +1,71 @@ +#!/bin/bash +# MindStream Phase 3 — Reel Generator Test +# +# Usage: +# ./test.sh # Run with default sample (data/sample_emotion_result.json) +# ./test.sh --emotion anxious --context '{"active_tab_category":"social_media",...}' +# +# How it works: +# Phase 1 (Extension) captures browser context → POST /check-in +# Phase 2 (Friend's script) detects emotion → writes _result.json +# server.js combines them → spawns this script with --job-id --emotion --context +# +# This test simulates that combined output. + +set -e + +echo "MindStream Phase 3 — Reel Generator Test" +echo "" + +# --- Python check --- +if ! command -v python3 &> /dev/null; then + echo "ERROR: Python 3 not found" + exit 1 +fi + +# --- Venv setup --- +if [ ! -d "venv" ]; then + echo "Creating virtual environment..." + python3 -m venv venv +fi + +source venv/bin/activate + +echo "Installing dependencies..." +pip install -q -r requirements.txt 2>/dev/null + +# # --- API keys (set via env or use defaults) --- +# if [ -z "$GEMINI_API_KEY" ]; then +# echo "Using default GEMINI_API_KEY" +# export GEMINI_API_KEY='AIzaSyBNyFh2UyLRRtBCK2P0pqr629bKPj9zWE4' +# fi + +# if [ -z "$PEXELS_API_KEY" ]; then +# echo "Using default PEXELS_API_KEY" +# export PEXELS_API_KEY='6Vta0QamMfVjjdTA8vg2AhdKrFGDBIs7SdFLYT2cnDWvSQqki3xeuU2v' +# fi + +# echo "" + +# --- Run --- +if [ $# -gt 0 ]; then + # Custom args passed — forward to reel_generator.py + echo "Running with custom args: $@" + python reel_generator.py "$@" +else + # No args — run with default sample data + echo "Running with default sample: data/sample_emotion_result.json" + # echo "(Edit this file to change emotion/context)" + echo "" + python reel_generator.py +fi + +echo "" + +# --- Check result --- +if [ $? -eq 0 ]; then + echo "SUCCESS — reel saved to output/reels/" +else + echo "FAILED — check errors above" + exit 1 +fi diff --git a/src/background/index.js b/src/background/index.js index 3ea260e..f98d9ca 100644 --- a/src/background/index.js +++ b/src/background/index.js @@ -9,6 +9,7 @@ import { API_ROUTES, CAPTURE_WINDOW, } from "../lib/constants.js"; +import { buildCheckInPayload } from "../lib/checkIn.js"; chrome.sidePanel .setPanelBehavior({ openPanelOnActionClick: true }) @@ -257,6 +258,44 @@ chrome.notifications.onClicked.addListener(async (notificationId) => { } }); +/** Queries the active tab to extract category, domain, title, and mock personalization variables. */ +async function getActiveTabInfo() { + try { + const [tab] = await chrome.tabs.query({ active: true, lastFocusedWindow: true }); + if (!tab || !tab.url) return null; + + const url = new URL(tab.url); + const domain = url.hostname; + + // Simple classifier for college portfolio demo + let category = "browsing"; + if (domain.includes("youtube.com") || domain.includes("netflix.com") || domain.includes("twitch.tv") || domain.includes("tiktok.com")) { + category = "entertainment"; + } else if (domain.includes("github.com") || domain.includes("stackoverflow.com") || domain.includes("developer") || domain.includes("localhost")) { + category = "coding"; + } else if (domain.includes("linkedin.com") || domain.includes("twitter.com") || domain.includes("facebook.com") || domain.includes("instagram.com") || domain.includes("reddit.com")) { + category = "social_media"; + } else if (domain.includes("google.com") || domain.includes("wikipedia.org") || domain.includes("medium.com")) { + category = "research"; + } else if (domain.includes("amazon.com") || domain.includes("ebay.com") || domain.includes("shopify")) { + category = "shopping"; + } + + return { + category, + domain, + title: tab.title ?? "unknown", + userName: "Prash", + weather: "chilly rain", + sessionDurationMinutes: Math.floor(Math.random() * 45) + 15, + idleMinutes: Math.floor(Math.random() * 5) + }; + } catch (e) { + console.error("Failed to get active tab info:", e); + return null; + } +} + // --- Messages from the panel and the capture window ----------------------- chrome.runtime.onMessage.addListener((message, _sender, sendResponse) => { if (message?.type === MESSAGE_TYPES.START_CHECKIN) { @@ -295,10 +334,40 @@ chrome.runtime.onMessage.addListener((message, _sender, sendResponse) => { } if (message?.type === MESSAGE_TYPES.CLIP_SAVED) { - // The capture window saved the clip to disk — record the path so the - // sidebar can show "processing" and the onRemoved listener knows not - // to reset the cycle when the capture window closes. - setCycle({ clip_path: message.clipPath }).then(() => sendResponse({ ok: true })); + // 1. Get active tab info + getActiveTabInfo().then(async (tabInfo) => { + // 2. Build check-in payload + const payload = buildCheckInPayload({ tabInfo }); + payload.clip_path = message.clipPath; + + try { + console.log("[background] Submitting check-in to server...", payload); + // 3. POST payload to the server + const response = await fetch(API_ROUTES.CHECK_IN, { + method: "POST", + headers: { + "Content-Type": "application/json" + }, + body: JSON.stringify(payload) + }); + + if (!response.ok) throw new Error(`Check-in request failed with status ${response.status}`); + const data = await response.json(); // expected: { job_id } + + console.log("[background] Check-in submitted, job ID:", data.job_id); + + // 4. Update the cycle status with job_id and clip_path + await setCycle({ clip_path: message.clipPath, job_id: data.job_id }); + } catch (err) { + console.error("[background] Failed to check-in with Express server:", err); + await setCycle({ + cycle_status: CYCLE_STATUS.FAILED, + error_message: "Could not connect to the local reel generation server. Make sure it is running on port 4000." + }); + } + }); + + sendResponse({ ok: true }); return true; } }); diff --git a/src/lib/checkIn.js b/src/lib/checkIn.js index 49349bc..adaee36 100644 --- a/src/lib/checkIn.js +++ b/src/lib/checkIn.js @@ -13,6 +13,9 @@ export function buildCheckInPayload({ tabInfo } = {}) { context: { active_tab_category: tabInfo?.category ?? "unknown", active_tab_domain: tabInfo?.domain ?? "unknown", + active_tab_title: tabInfo?.title ?? "unknown", + user_name: tabInfo?.userName ?? "friend", + local_weather: tabInfo?.weather ?? "calm", time_of_day: new Date().getHours() < 12 ? "morning" : new Date().getHours() < 18 ? "afternoon" : "evening", session_duration_minutes: tabInfo?.sessionDurationMinutes ?? 0, idle_minutes_since_last_activity: tabInfo?.idleMinutes ?? 0,