diff --git a/examples/README.md b/examples/README.md index efcfaf6..a405c36 100644 --- a/examples/README.md +++ b/examples/README.md @@ -2,16 +2,17 @@ Self-contained, runnable Next.js apps built on `@reactor-team/js-sdk` and the typed per-model packages. These are the same templates that `npx create-reactor-app` scaffolds — each folder here can be cloned, installed, and run independently. -| Example | Model | Highlights | -| --------------------------------------- | ----------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| [`happy-oyster/`](./happy-oyster) | [`@reactor-models/happy-oyster`](https://www.npmjs.com/package/@reactor-models/happy-oyster) | Interactive world model. Build a world from a prompt (or attach a permanent one), then travel it live: **Adventure** worlds you drive with WASD, **Director** worlds you steer with text `instruct` + pause/rewind. Mode-fixed sessions, authoritative `world_state` snapshot. | -| [`helios/`](./helios) | [`@reactor-models/helios`](https://www.npmjs.com/package/@reactor-models/helios) | Continuous prompt-driven video generation. Curated text and image scenes, mid-stream prompt hot-swap via `setPrompt`, atomic `setConditioning({ prompt, image })` for image-to-video, snap-clip recording, design tokens from `@reactor-team/ui`. | -| [`lingbot/`](./lingbot) | [`@reactor-models/lingbot`](https://www.npmjs.com/package/@reactor-models/lingbot) | Interactive world model. Pick a starting image, drive the scene with WASD, layer in curated dynamic events (rain, fog, …) as live prompt swaps, snap-clip recording. | -| [`lingbot-world-2/`](./lingbot-world-2) | [`@reactor-models/lingbot-world-2`](https://www.npmjs.com/package/@reactor-models/lingbot-world-2) | Interactive world model you drive like a game. Two-axis WASD driving, per-latent `set_camera_pose` motion (mouse-look, roll, orbit, jump arcs, crouch dips), hold-key world events, a layered prompt workbench with live editor + inspector, attn-window / KV-cache knobs. | -| [`longlive-v2/`](./longlive-v2) | [`@reactor-models/longlive-v2`](https://www.npmjs.com/package/@reactor-models/longlive-v2) | Multi-shot **director's storyboard**. Compose shots (`set_shot`) and cuts (`scene_cut`) on a chunk timeline, schedule beats with `schedule_shot` / `schedule_scene_cut`, then direct live. Surfaces the per-scene 48-chunk budget and how cuts extend length. Snap-clip recording. | -| [`ltx2/`](./ltx2) | [`@reactor-models/ltx2`](https://www.npmjs.com/package/@reactor-models/ltx2) | Streaming **talking-head avatar**. Upload a face and a script; the model generates voice and lip-synced video together and streams both. The take in flight is frozen, but the session stays editable: mid-run edits are accepted and queue for the next take, surfaced from `state_update.queued_changes`. Server-authoritative `valid_commands`, crop-before-upload, measured TTFF. | -| [`sana-streaming/`](./sana-streaming) | [`@reactor-models/sana-streaming`](https://www.npmjs.com/package/@reactor-models/sana-streaming) | Streaming **V2V editor**. Live webcam transform via manual `camera` publish, file-clip editing with side-by-side compare, mid-stream re-prompting, seed control, snap-clip recording. | -| [`x2/`](./x2) | [`@reactor-team/js-sdk`](https://www.npmjs.com/package/@reactor-team/js-sdk) (typed client vendored in-app) | Streaming **V2V editor** on XMAX X2. Webcam, file-clip, or still-image sources on one `source` track, side-by-side compare, mid-stream re-prompting, reference-image conditioning via `uploadFile`, drag-to-steer pointer on the output, keep-backlog toggle, snap-clip recording. | +| Example | Model | Highlights | +| --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| [`happy-oyster/`](./happy-oyster) | [`@reactor-models/happy-oyster`](https://www.npmjs.com/package/@reactor-models/happy-oyster) | Interactive world model. Build a world from a prompt (or attach a permanent one), then travel it live: **Adventure** worlds you drive with WASD, **Director** worlds you steer with text `instruct` + pause/rewind. Mode-fixed sessions, authoritative `world_state` snapshot. | +| [`helios/`](./helios) | [`@reactor-models/helios`](https://www.npmjs.com/package/@reactor-models/helios) | Continuous prompt-driven video generation. Curated text and image scenes, mid-stream prompt hot-swap via `setPrompt`, atomic `setConditioning({ prompt, image })` for image-to-video, snap-clip recording, design tokens from `@reactor-team/ui`. | +| [`lingbot/`](./lingbot) | [`@reactor-models/lingbot`](https://www.npmjs.com/package/@reactor-models/lingbot) | Interactive world model. Pick a starting image, drive the scene with WASD, layer in curated dynamic events (rain, fog, …) as live prompt swaps, snap-clip recording. | +| [`lingbot-world-2/`](./lingbot-world-2) | [`@reactor-models/lingbot-world-2`](https://www.npmjs.com/package/@reactor-models/lingbot-world-2) | Interactive world model you drive like a game. Two-axis WASD driving, per-latent `set_camera_pose` motion (mouse-look, roll, orbit, jump arcs, crouch dips), hold-key world events, a layered prompt workbench with live editor + inspector, attn-window / KV-cache knobs. | +| [`longlive-v2/`](./longlive-v2) | [`@reactor-models/longlive-v2`](https://www.npmjs.com/package/@reactor-models/longlive-v2) | Multi-shot **director's storyboard**. Compose shots (`set_shot`) and cuts (`scene_cut`) on a chunk timeline, schedule beats with `schedule_shot` / `schedule_scene_cut`, then direct live. Surfaces the per-scene 48-chunk budget and how cuts extend length. Snap-clip recording. | +| [`ltx2/`](./ltx2) | [`@reactor-models/ltx2`](https://www.npmjs.com/package/@reactor-models/ltx2) | Streaming **talking-head avatar**. Upload a face and a script; the model generates voice and lip-synced video together and streams both. The take in flight is frozen, but the session stays editable: mid-run edits are accepted and queue for the next take, surfaced from `state_update.queued_changes`. Server-authoritative `valid_commands`, crop-before-upload, measured TTFF. | +| [`sana-streaming/`](./sana-streaming) | [`@reactor-models/sana-streaming`](https://www.npmjs.com/package/@reactor-models/sana-streaming) | Streaming **V2V editor**. Live webcam transform via manual `camera` publish, file-clip editing with side-by-side compare, mid-stream re-prompting, seed control, snap-clip recording. | +| [`visko-orbis-distilled/`](./visko-orbis-distilled) | [`@reactor-models/visko-orbis-distilled`](https://www.npmjs.com/package/@reactor-models/visko-orbis-distilled) (vendored pre-publish) | Continuous **steerable video generation** (Visko Orbis, Distilled tier). Text-to-video and image-anchored starts, **per-chunk live prompt morphing**, resolution picker from `available_resolutions`, seed, audio on/off + muted `main_audio` outlet, snap-clip recording. | +| [`x2/`](./x2) | [`@reactor-team/js-sdk`](https://www.npmjs.com/package/@reactor-team/js-sdk) (typed client vendored in-app) | Streaming **V2V editor** on XMAX X2. Webcam, file-clip, or still-image sources on one `source` track, side-by-side compare, mid-stream re-prompting, reference-image conditioning via `uploadFile`, drag-to-steer pointer on the output, keep-backlog toggle, snap-clip recording. | ## Running an example diff --git a/examples/visko-orbis-distilled/.env.example b/examples/visko-orbis-distilled/.env.example new file mode 100644 index 0000000..c8eeda2 --- /dev/null +++ b/examples/visko-orbis-distilled/.env.example @@ -0,0 +1 @@ +REACTOR_API_KEY=rk_your_api_key_here diff --git a/examples/visko-orbis-distilled/.gitignore b/examples/visko-orbis-distilled/.gitignore new file mode 100644 index 0000000..298fe3e --- /dev/null +++ b/examples/visko-orbis-distilled/.gitignore @@ -0,0 +1,15 @@ +node_modules +.next +.env +.env.local +*.log +.DS_Store +*.tsbuildinfo + +# The vendored typed SDK ships only its built output (dist/ + package.json + +# README) — exactly what `npm i @reactor-models/visko-orbis-distilled` would +# lay down once the package is published. The repo-root .gitignore ignores +# `dist/` everywhere (it exists for the library build), which would otherwise +# strip the vendored package's only payload and make a fresh clone unrunnable. +# Re-include it here so the PR is self-contained. See vendor/README.md. +!vendor/**/dist/ diff --git a/examples/visko-orbis-distilled/README.md b/examples/visko-orbis-distilled/README.md new file mode 100644 index 0000000..d622a30 --- /dev/null +++ b/examples/visko-orbis-distilled/README.md @@ -0,0 +1,103 @@ +# Visko Orbis Distilled + +A Next.js + TypeScript reference frontend for **Visko Orbis · Distilled** — Reactor's real-time, steerable video generation model (the Distilled tier of the Visko Orbis family). + +Connect, send a prompt, and watch the model produce a continuous video stream you can **steer mid-flight**. Start from a curated text prompt, an example image, or your own image — then hot-swap prompts to morph the scene live, no restart, no cut. Built on the typed `@reactor-models/visko-orbis-distilled` SDK. + +``` +┌──────────────────────┬─────────────────────────────────────┐ +│ Status ▸ ready │ │ +│ │ │ +│ Try a prompt │ │ +│ ┌────────┬────────┐ │ live video output │ +│ │ Coast │ Desert │ │ (ViskoOrbisDistilledMainVideoView)│ +│ └────────┴────────┘ │ │ +│ Or start from image │ │ +│ ┌────────┬────────┐ │ │ +│ │ Citadel│ Neon │ │ + muted-by-default main_audio│ +│ └────────┴────────┘ │ │ +│ Session ▾ resolution│ │ +│ Audio ▸ on │ │ +│ [Snap clip] │ │ +└──────────────────────┴─────────────────────────────────────┘ +``` + +## Quick start + +You'll need a Reactor API key — grab one at [reactor.inc/account/api-keys](https://www.reactor.inc/account/api-keys). It starts with `rk_`. + +```bash +cp .env.example .env +# add your key: REACTOR_API_KEY=rk_... + +pnpm install +pnpm dev +``` + +Open [http://localhost:3000](http://localhost:3000), click **Connect**, and pick a starting point. + +> **Typed package — read this if `@reactor-models/visko-orbis-distilled` 404s on npm.** +> This example ships with the typed SDK **vendored** at [`vendor/visko-orbis-distilled/`](vendor/visko-orbis-distilled) and wired in via a `pnpm` override (`link:./vendor/visko-orbis-distilled`). That's a pre-publication shim: the package is generated from the model's live PROD schema but isn't on the public registry yet. `package.json` declares `"@reactor-models/visko-orbis-distilled": "^2.0.0"`, so once the real package is published you can drop the `pnpm.overrides` block and `pnpm up` — nothing else changes. See [`vendor/README.md`](vendor/README.md). + +## What you can do with it + +- **Start a scene from text alone (T2V).** Curated prompt presets in the sidebar, plus free-text. No image required — the model invents the opening frame. +- **Start from an image (I2V).** Curated image+prompt cards, or upload your own. The image anchors the first chunk; every later chunk inherits it through the model's history. +- **Steer the scene live — the hero feature.** Prompts are **per-chunk**: send a new prompt mid-run and the picture morphs into it at the next chunk boundary (~1.8 s). One-click "evolutions" keep the same world and shift its mood (golden hour → storm, calm sea → night fog), or type any new scene. +- **Pick a delivery resolution.** Rendered from the deployment's `available_resolutions` (e.g. 1080p / 2k / 4k) — never hard-coded. Applies at the next `start`; survives `reset`. +- **Audio on/off + muted playback.** `main_audio` is a real 48 kHz mono track, chunk-aligned with the video. Playback starts muted; the "generate sound" toggle saves compute and applies at the next `start`. +- **Reproducible runs.** `set_seed` — same seed + same prompts reproduces the same video. Applies at the next `start`. +- **Pause / Resume / Reset.** Real-time transport. `generation_complete` returns to waiting (no auto-restart) — "Start again" re-runs with the same conditions, "Reset" clears prompt + image. +- **Snap a clip.** Capture the last 10 s of the live stream and download an MP4 — base-SDK recording, no extra services. + +## Reality notes (things that will look "broken" but aren't) + +These are measured behaviours of the deployed model, surfaced so you don't misread them: + +- **Connect takes MINUTES.** There's one live session per deployment, and startup pays for SR-model compile + three warmup chunks. The StatusBadge labels `waiting` honestly — it's not stuck. If it hangs in `waiting`, another tab (or a zombie session) is holding the pod. +- **The first chunk emits zero frames.** The super-resolution model primes on chunk 1 (`frames_emitted: 0`); first picture lands ~2 chunks / ~3.7 s in. The video panel holds a "Priming the stream…" overlay over this window instead of a black box. +- **The resolution picker changes the DELIVERED raster, not the dream.** The model generates at 832×480 and SparkVSR upscales to the picked tier. Picking 4k doesn't change what the model invents — only the output size. +- **Resolution, seed, and audio settings apply at the NEXT `start`** and survive `reset`; the prompt and image do not. +- **Non-16:9 images squash.** The reference image is resized to 832×480 with no crop. Use a 16:9 frame (the curated images are). +- **No audio prompt box on purpose.** Feeding a scene description into `set_audio_prompt` makes the audio measurably worse than leaving it unset (unset = sound generated from the picture alone). The example deliberately doesn't expose it. + +## Architecture at a glance + +The sidebar UI has two phases driven by the model's `state` snapshot (the single source of truth — never re-derive it from other messages): + +| Phase | When | What's visible | +| --------- | ----------------------------- | ------------------------------------------------------------------------------------- | +| **Setup** | before generation has started | prompt presets, example images, custom upload, free-text textarea, session options | +| **Live** | while generating or paused | active prompt, chunk counter, Pause / Resume / Reset, live steering, run-finished CTA | + +Each component subscribes to the snapshot itself and self-hides when it's not in its phase. No central orchestrator. + +## Code tour + +The interesting bits, in roughly the order you'd read them: + +| File | What's in it | +| ------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| [`app/page.tsx`](app/page.tsx) | Server Component. Checks `REACTOR_API_KEY` is set, otherwise renders [`SetupRequired.tsx`](app/SetupRequired.tsx). | +| [`app/api/reactor/token/route.ts`](app/api/reactor/token/route.ts) | GET route that mints a session-scoped Reactor JWT (pinned to `reactor/visko-orbis-distilled` via `authorization_details`) and sets `Cache-Control: private, max-age=`. The browser handles caching transparently. | +| [`app/ViskoOrbisDistilledApp.tsx`](app/ViskoOrbisDistilledApp.tsx) | First `"use client"` boundary. Wires ``, lays out the sidebar + video pane, mounts the hidden ``. | +| [`app/lib/prompts.ts`](app/lib/prompts.ts) | The scene library. Every prompt the app suggests — starting prompts and mid-stream evolutions — lives here. Same source feeds the setup presets, the example image cards, and the live steering picker. | +| [`app/components/PromptComposer.tsx`](app/components/PromptComposer.tsx) | Setup phase. Preset prompts + free-text input → `setPrompt` + `start`. Works text-only (T2V). | +| [`app/components/ImageStarter.tsx`](app/components/ImageStarter.tsx) | Setup phase. Curated image scenes run `uploadFile` → `setImage` → **wait for `image_accepted`** → `setPrompt` → `start` (no atomic `setConditioning` on this model, so the chain is explicit). Custom uploads just call `setImage`. Surfaces the 16:9 squash note. | +| [`app/components/EvolveScene.tsx`](app/components/EvolveScene.tsx) | **Live phase — the hero.** Matches the active prompt against the scene library and renders evolutions as one-click morphs, plus a free-form morph box. `setPrompt` only — no restart. | +| [`app/components/SessionOptions.tsx`](app/components/SessionOptions.tsx) | Resolution picker (from `available_resolutions`, never hard-coded) + seed. Both apply at the next `start`. | +| [`app/components/AudioPanel.tsx`](app/components/AudioPanel.tsx) | "Generate sound" toggle (`set_audio_enabled`) + why playback starts muted and there's no audio-prompt box. | +| [`app/components/AudioOutlet.tsx`](app/components/AudioOutlet.tsx) | Renders `main_audio` as a hidden, muted-by-default `