Skip to content

Live camera integration: G1 real-camera baseline + G2 recording/annotation/real-data tooling - #4

Open
ThatKJ wants to merge 25 commits into
mainfrom
feat/live-camera-integration
Open

Live camera integration: G1 real-camera baseline + G2 recording/annotation/real-data tooling#4
ThatKJ wants to merge 25 commits into
mainfrom
feat/live-camera-integration

Conversation

@ThatKJ

@ThatKJ ThatKJ commented Sep 12, 2026

Copy link
Copy Markdown
Owner

Summary

Brings the real-camera-in-the-loop work (previously stranded on feat/phone-camera-in-loop, never merged) onto main, fixes the frame/telemetry identity gap it had, and adds the G2 recording + annotation + real-data-dataset tooling on top.

G1 — physical live baseline

  • Merges feat/phone-camera-in-loop's FrameSource/OpenCV camera/LiveTrackingSession/VirtualPanTiltActuator pipeline (honestly labeled, reuses the frozen detector/hybrid-perception/tracker stack).
  • fsoc::LiveFramePublisher: fixes a real frame↔telemetry identity/atomicity gap — writes frame_<N>.jpg/telemetry_<N>.json via write-to-temp-then-atomic-rename, flips manifest.json last, so a client can never observe a mismatched pair.
  • fsoc_live --uncalibrated: pixel-only tracking with no fabricated field of view — angular telemetry is always null, control/actuation force-disabled. Lets a real preview run before a calibration file exists.
  • /mission/live: CALIBRATION/SESSION ID/TIMING (measured processed + displayed FPS) rows, a live detected-beacon reticle overlay, 6 new route-mocked e2e tests for previously-untested risks (reconnect, stale, missing frame, recording status).

G2 — recording, annotation, real-data tooling (software only — no real reviewed recordings ship in this PR)

  • fsoc::RealSessionRecorder: a second, unpruned sink (separate from the live preview buffer) — raw frames, append-only telemetry, event markers, atomically-updated manifest with full software/calibration/model identity.
  • Start/Stop/Mark-Event recording controls on /mission/live, driven by a polled command.txt.
  • /mission/annotate: review a recording, scrub frames, label presence/occlusion, click-set the beacon center, review (never auto-accept) a detector suggestion.
  • tools/ai/real_dataset.py + real_dataset_split.py: additive real-data loader (same tensor contract as the synthetic BeaconDataset) with group-based (never frame-based) train/val/test splitting and an explicit ambiguous/unreviewed exclusion policy.

Fixes found testing live against a real camera

  • to_json() was pretty-printed (multi-line), silently corrupting telemetry.jsonl's one-object-per-line contract.
  • The live reticle / annotation marker overlay ignored the image's centering offset within its container, pulling every marker toward the panel corner instead of the detected point.
  • /mission/annotate assumed frame indices start at 0 and are contiguous; a recording's frame index is the camera's own continuous counter, so it usually starts well above 0.

Full gate tracking, camera setup checklist, and next steps: docs/LIVE_REALDATA_TASK_STATE.md. Audit of what was real vs. simulated before this work: docs/LIVE_DATA_AUDIT.md.

Test plan

  • cmake --build --preset debug clean; ctest --preset debug 24/24 PASS
  • npm run typecheck / npm run lint / npm run build clean
  • npx playwright test 61/61 PASS
  • .venv-ai/bin/python3 tools/ai/real_dataset_tests.py 13/13 PASS (labeled fixtures, not real data)
  • Verified live against a real webcam: camera capture, classical detection, live reticle tracking, Start/Mark-Event/Stop recording producing a real recording on disk, /mission/annotate reviewing real captured frames

🤖 Generated with Claude Code

https://claude.ai/code/session_01YTok4Srfs3CSL6Mp5TiYXD

ThatKJ and others added 24 commits September 7, 2026 16:42
@ThatKJ @claude
feat(sih): add SIH judge-evidence capture script; fix estimator panel…
d2cd149 
… gating

State Estimator (P0-v2) panel in TelemetryStream was gated on a UI-only
context flag (trackerEnabled) that no control ever sets, instead of on
whether tracker data is actually present in the response. Gate on
current.tracker presence instead, matching the sibling perception panel's
own pattern. Needed to make the estimator panel render during the new
evidence-pack capture script's Hybrid+tracker runs.

Add frontend/scripts/capture-sih-assets.mjs: drives the real Mission
Control frontend (production build, ENGINE mode) with Playwright to
capture deterministic screenshots/metrics/diagrams for the SIH judge
evidence pack. No telemetry is fabricated; Hybrid/tracker runs are
requested via route interception on the app's own already-shipped
/api/simulation/:scenario query params (no UI toggle exists yet).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014h1TXJD8THZH4NbMqzrx97
@ThatKJ @claude
docs(sih): add SIH 2026 judge-facing visual evidence pack
94521a5 
17 real assets under presentation_assets/, generated by
frontend/scripts/capture-sih-assets.mjs against the real fsoc_demo C++
engine (ENGINE mode) plus real terminal output and graphics built from
committed measurement docs (docs/MVP_METRICS.md, docs/MVP_ABLATION.md).
No telemetry, chart value, or terminal line is fabricated or hand-edited.

- screenshots/: hero tracking, misalignment->convergence pair, Hybrid
  fusion + state-estimator proof, target-loss/coasting/reacquisition
  sequence, full telemetry dashboard, Step-9 perception frame, real
  CTest/CLI terminal captures
- metrics/: measured-results and ablation graphics from the frozen
  Stage-4 protocol
- diagrams/: implemented-only architecture, SEE->ESTIMATE->PREDICT->
  CORRECT story graphic, before/after and failure-recovery collages
  (collages composited from real screenshots, no pixels altered)
- manifest.json: full per-asset record (purpose, scenario, command,
  data source, ENGINE/REPLAY, commit SHA, capture date, sim-vs-hardware,
  recommended slide, caveats), including two items that could not be
  truthfully produced (07_clutter_rejection: no live UI route for that
  scenario; 14_frontend_validation: deprioritized) and one deliberately
  skipped (demo video, optional per task scope)
- README.md: navigation + per-asset table
- PPT_SHORTLIST.md: 8 assets ranked MUST USE / STRONG OPTIONAL /
  TECHNICAL BACKUP with what each proves and does not prove

All assets are simulation-only; no physical hardware exists in this
repository. See docs/SIH_MVP_FREEZE.md for the hardware boundary.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014h1TXJD8THZH4NbMqzrx97
@ThatKJ @claude
feat(sih): hybrid hero shot, true-peak misalignment frame, open/close…
29c753b 
…d-loop stat

Three targeted evidence-pack fixes on real, verified data:

- 01_mission_control_tracking (hero): recapture in Hybrid+tracker mode
  after convergence instead of Classical mid-run, so the hero frame shows
  Hybrid/AI perception (MODE: HYBRID) and the state estimator (LOCK
  STATE: TRACKING, CONFIDENCE 1.00) both visibly active, not just a bare
  tracking UI. Added findLateRatioWithLockState() to search backward from
  the end of a run for a converged, still-locked frame.

- 02_initial_misalignment: the previous ratio-based seek landed on frame
  5/200 (1.498deg), well past the true initial peak. Use the transport's
  own "Restart" button (reset() -> seek(0), exact) instead of a
  pixel-offset slider click, which cannot reach frame 0 exactly. Now
  reads 4.1275deg, matching the frozen docs/MVP_METRICS.md /
  docs/17_DEMO_FREEZE.md static-scenario figure (4.13deg).

- New metrics/20_open_vs_closed_loop.png: compact Open Loop 6.4549deg
  RMS -> Closed Loop 0.5461deg RMS (x11.8) graphic, labeled SIMULATION /
  DEVELOPMENT-MACHINE RESULTS, sourced from docs/MVP_METRICS.md,
  docs/17_DEMO_FREEZE.md, and the real step10_validation_smoke terminal
  output already captured in 12_validation_terminal.png.

- 18_before_after: collageHtml() now takes an optional headline stat;
  this collage now leads with the real "4.13deg -> 0.00deg" figure
  instead of two unlabeled screenshots.

manifest.json, README.md, and PPT_SHORTLIST.md updated to match: the
hero and before/after entries now describe the Hybrid/converged capture
and the true frame-0 value, the new RMS graphic is documented with its
exact data source, and the shortlist is re-ranked around the two
strengthened assets.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014h1TXJD8THZH4NbMqzrx97
@ThatKJ @claude
feat(camera): add mobile phone camera-in-the-loop C++ pipeline
f098d91 
Extends FSOC from a synthetic camera + simulated actuator to a real
mobile-phone camera + real perception + real state estimation + real
controller + an honestly-virtual actuator, without modifying the frozen
simulation baseline (SimulationRunner, PanTiltCamera, BeaconDetector,
AiBeaconDetector, resolve_perception, TargetTracker, PIDController are
all reused unchanged).

New, additive C++ pieces:
- FrameSource (fsoc/frame_source.hpp): pure I/O boundary, no coupling to
  perception or world truth. OpenCVCameraFrameSource wraps cv::VideoCapture
  for both a native camera index and a network URL (one class, since
  VideoCapture already exposes both through the same surface).
- LiveCameraCalibrationConfig: the simplest defensible camera geometry --
  a declared/measured pinhole FOV in a dependency-free key=value file, plus
  an angular-substitution estimation method from one known-size object.
  Never assumed equal to the simulation's CameraConfig.
- preprocess_live_frame(): grayscale + resize to the exact 640x480
  AiBeaconDetector's heatmap decode requires (kInputStride is fixed --
  a real frame at any other resolution would silently produce wrong
  centroids without this).
- VirtualPanTiltActuator: integrates the real PID output into a bookkeeping
  pan/tilt angle. Drives nothing physical, and does not feed back into how
  the next real frame's pixel error is interpreted. Deliberately NOT
  PanTiltCamera reuse, to keep the simulation/virtual-actuator boundary
  sharp. Always labeled ACTUATOR_TYPE = VIRTUAL in telemetry.
- LiveTrackingSession: the real-camera counterpart to SimulationRunner,
  mirroring its control-path order exactly. LiveFrameResult carries no
  target_truth/observation/target_visible field -- there is no ground
  truth for a real frame, and none is fabricated (see
  test_no_target_no_ground_truth_leak). Its sensing-reference PanTiltCamera
  is built at the PREPROCESSED frame size, not the phone's raw capture
  resolution -- fx_px scales with width_px for a fixed FOV, so getting this
  backwards would silently scale every angular error.

New CLI tools: fsoc_camera_probe (device discovery), fsoc_camera_view
(minimal live viewer, snapshot-based since no display is assumed),
fsoc_camera_calibrate (manual or known-object FOV estimation), fsoc_live
(the full pipeline: --mode classical|ai|hybrid, --tracker,
--manual-assist for human-readable PAN/TILT correction cues,
--no-control for observe-only, clean handling of camera-open failure,
consecutive-read-failure disconnect detection, and missing-model
diagnostics).

18 automated tests added (fsoc_virtual_actuator_tests,
fsoc_live_camera_calibration_tests, fsoc_frame_source_tests,
fsoc_live_preprocessing_tests x7, fsoc_live_tracking_session_tests x8),
all using fabricated frames/configs -- no real camera hardware is opened
from an automated test run (see docs/PHONE_CAMERA_METRICS.md for why).
All pre-existing 17 C++ tests remain green; full suite now 22/22.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@ThatKJ @claude
feat(ui): expose real-camera mode in Mission Control
92b2a2d 
Adds a real-camera view without rewriting the frontend architecture or
touching the existing 9-screen Stitch nav (lib/nav.ts stays fixed at 9 --
see STITCH_IMPLEMENTATION_MAP.md): /mission/live is reachable by direct
link, nested under Mission Control, reusing the existing Panel/
KeyValueRow primitives.

/api/simulation/:scenario runs a finite scenario to completion and
returns every frame as one array -- that pattern does not fit fsoc_live,
a long-running process with no natural end. Instead:

  fsoc_live -> overwrites generated/live/telemetry.json + frame.jpg
            -> GET /api/live-camera + /api/live-camera/frame (polled every
               500ms by the new page)

This is an honest polling snapshot read of a separate process YOU start
yourself, not a push/streaming connection -- documented as such in both
routes and the page. When no session is running, the page shows "No live
session" and never fabricates telemetry (verified manually: 503 from both
routes, clean UI state). The page labels CAMERA SOURCE and ACTUATOR
explicitly and flags stale snapshots (age > 3s).

Verified with a scratch (gitignored, deleted after) fabricated telemetry
snapshot: full page renders correctly, staleness detection works.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@ThatKJ @claude
docs(phone): add real-camera architecture, test plan, and golden demo
3921900 
- docs/PHONE_CAMERA_METRICS.md: architecture (why almost nothing in the
  existing perception stack needed to change), the JSON telemetry schema,
  Mission Control transport rationale, calibration methodology, the
  virtual-actuator honesty statement, a hardware-ready PanTiltActuator
  interface sketch (rate-vs-position command mismatch a future serial
  adapter must solve), known limitations, and an explicit claim boundary
  (acceptable vs. not-acceptable phrasing).
- docs/PHONE_CAMERA_TEST_PLAN.md: automated (CTest, all passing) vs.
  manual (hardware-required, PASS/FAIL checklist) test split.
- docs/PHONE_CAMERA_GOLDEN_DEMO.md: a 15-step, zero-additional-hardware
  demo runbook (Mac/phone camera + tools/beacon_display.html as the
  target).
- tools/beacon_display.html: a real, physical bright-circle test target
  (static/horizontal/vertical/sinusoidal motion) for real-camera testing
  -- explicitly not an FSOC transmitter.
- README.md: new "Mobile Phone Camera-in-the-Loop" section; the Hardware
  boundary section is corrected to be precise now that an optional
  real-camera-sensing prototype exists (actuator side remains virtual;
  the frozen SIH MVP simulation is unaffected).
- docs/09_FUTURE_ARCHITECTURE.md, docs/README.md: cross-references to the
  above.
- .gitignore (previous commit): configs/ (personal camera calibration,
  hardware-specific, never a general default).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@ThatKJ @claude
feat(launcher): add one-command project launcher (./run_fsoc.sh)
b744cdb 
Pure orchestration -- zero tracking/perception/controller logic. Every
command it runs is one already documented in README.md,
docs/PHONE_CAMERA_GOLDEN_DEMO.md, docs/PHONE_CAMERA_TEST_PLAN.md, or
.github/workflows/ci.yml; nothing was guessed (verified fsoc_demo/
fsoc_live/fsoc_camera_probe/fsoc_camera_calibrate's exact CLI flags
against their source, the frontend port against frontend/package.json,
and the Playwright webServer/reuseExistingServer behavior against
playwright.config.ts before writing anything).

run_fsoc.sh (repo root) execs scripts/run_fsoc.sh, the real
implementation, matching the existing scripts/run_baseline_demo.sh
convention. Interactive 8-item menu, or non-interactive subcommands
(simulation|phone|probe|ui|test|golden|build), plus --rebuild and
--no-browser.

Key correctness points found and fixed during testing (not guessed):
- The frontend port is never hardcoded -- read live from
  frontend/package.json's own `-p` flag.
- C++ build delegates entirely to `cmake --preset debug` +
  `cmake --build --preset debug`; Ninja's own incremental tracking
  decides what's stale, never reimplemented here.
- Process-group-scoped shutdown: `set -m` + `kill -- -PID` (not a bare
  PID kill) is required to actually stop a background frontend's whole
  descendant tree (npm -> next dev -> next-server) -- verified
  empirically that a bare `kill $PID` left next-server running and the
  port occupied; the process-group kill leaves nothing behind and frees
  the port. Never a broad killall/pkill -f.
- Port reuse is content-verified (curl + grep for an FSOC-identifying
  string), not just "something answered" -- verified it refuses to
  touch a port occupied by an unrelated process (tested against a dummy
  python http.server) and correctly reuses/never-duplicates its own
  already-running instance.
- mode_validate uses plain variables, not `local -A`: this machine's
  default `/usr/bin/env bash` is 3.2 (no associative-array support) --
  verified directly, not assumed.
- A command-substitution subshell's internal `exit` (e.g.
  ensure_calibration() declining to write a calibration file) correctly
  propagates through `set -e` at the call site -- verified empirically
  rather than assumed.
- Fixed a real ordering bug: mode_golden's manual checklist was printed
  after the frontend's blocking wait, which never returns before Ctrl+C
  triggers the INT trap's `exit` -- the checklist would never have been
  seen. Split mode_phone_start()/block_until_stopped() so anything
  meant to be visible prints before the block, and block_until_stopped
  now tells the truth (no blocking, no misleading "Press Ctrl+C") when
  Mission Control was reused rather than started by this instance.

fsoc.env.example: non-secret local overrides (FSOC_CAMERA_INDEX,
FSOC_CAMERA_CALIBRATION, FSOC_AI_MODEL). No FSOC_FRONTEND_PORT knob --
documented as not real, since the port is auto-detected from
package.json instead of guessable/overridable here.

Manually tested end-to-end (no camera hardware touched, per this
project's established boundary): build, simulation demo (full fsoc_demo
terminal run + Mission Control + clean shutdown), Mission Control Only
(start/reuse/port-conflict-refusal/process-group shutdown), Run Full
Validation (real 22/22 ctest + step10 + frontend typecheck/lint/build +
20/20 Playwright, correct PASS summary, correct exit code), interactive
menu navigation, and the phone-camera calibration-missing decline path.
shellcheck: zero warnings on both files.

README.md Quick Start now leads with ./run_fsoc.sh; the full manual
step-by-step is preserved underneath in a <details> block.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@ThatKJ @claude
chore: add MIT LICENSE
258eec5 
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YTok4Srfs3CSL6Mp5TiYXD
@ThatKJ @claude
feat(web): honest local/public empty state on /mission/live
0c990e2 
Replace the generic "No live session" panel with a designed Real-Camera
Mode card explaining why the public deployment can never show live
telemetry (fsoc_live is a local native process, not something Vercel can
reach) and giving the exact local command + setup guide link. Behavior is
unchanged locally: it's the same honest "no session" state, just no
longer indistinguishable from an error.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YTok4Srfs3CSL6Mp5TiYXD
@ThatKJ @claude
docs: fill README gaps and add Vercel deployment guide
e1a34df 
- README: correct stale 17/17 -> 22/22 ctest count (3 spots), add a
  Working Modes table (simulation / replay / engine / phone camera /
  future hardware), a Deployment section pointing at the new guide, and
  Contributing/Team/License sections.
- Add docs/DEPLOYMENT.md: public Vercel architecture, project settings,
  why the phone-camera prototype can't run on Vercel, local workflow,
  troubleshooting, and Claude Code + Vercel MCP setup.
- Index the new doc in docs/README.md.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YTok4Srfs3CSL6Mp5TiYXD
@ThatKJ @claude
feat(web): add SEO metadata, OG image, favicon, robots, sitemap
464acd1 
- layout.tsx: metadataBase, refreshed title/description (matches the
  brief's problem-statement framing rather than the old "frozen
  v1_baseline" internal wording), Open Graph + Twitter card metadata,
  robots directive, keywords/authors.
- opengraph-image.tsx: a code-generated (next/og), on-brand social
  preview using the real Stitch palette -- no external image asset, no
  fake satellite/dashboard imagery.
- icon.svg: a small crosshair/reticle favicon in the same palette
  (matches the existing Crosshair nav icon for Mission Control).
- robots.ts / sitemap.ts: standard Next.js App Router conventions,
  pointing at the project's real Vercel domain (fsoc-iota.vercel.app).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YTok4Srfs3CSL6Mp5TiYXD
@ThatKJ @claude
docs: add CONTRIBUTING and SECURITY
1d4fd00 
Short, project-specific rather than boilerplate: CONTRIBUTING points at
the frozen architecture boundaries (CLAUDE.md) and the existing PR
template instead of restating them; SECURITY states the actual (low)
risk surface honestly rather than a generic disclosure policy.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YTok4Srfs3CSL6Mp5TiYXD
@ThatKJ @claude
test(e2e): make the smoke suite deploy-safe against a remote URL
d0ec2a7 
FSOC_BASE_URL already overrode baseURL, but playwright.config.ts always
tried to boot a local `npm run start` regardless -- pointless (and
occasionally fatal, if no local production build exists) when the target
is actually a deployed Preview/production URL. Skip the local webServer
whenever FSOC_BASE_URL is set. The one test that needs the local C++
engine already skips itself when unreachable, so the existing suite was
otherwise already deploy-safe. Documented in docs/DEPLOYMENT.md.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YTok4Srfs3CSL6Mp5TiYXD
@ThatKJ @claude
chore(repo): remove stale workflow placeholder and tighten local arti…
20183da 
…fact ignores

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YTok4Srfs3CSL6Mp5TiYXD
@ThatKJ @claude
test(e2e): use "load" instead of "networkidle" for route-load checks
503a1e5 
Found during Preview QA: the /world route (WebGL/three.js) legitimately
keeps background network activity going past networkidle's 500ms-quiet
window on a cold remote connection, timing out page.goto() at 30s even
though the page renders correctly (confirmed visually: the 3D scene
paints within ~5-8s on a cold Vercel Preview). networkidle is a known-
fragile wait condition for canvas/client-heavy pages; the existing
assertions (header text, nav count, main visible) are a more honest
liveness check than network quiescence. All 20 tests pass locally and
this specifically fixes the remote-deployment flake.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YTok4Srfs3CSL6Mp5TiYXD
@ThatKJ @claude
fix(web): stop homepage stat grid from overlapping the CTA buttons
f41111b 
Found during Preview visual QA: the System Status / Uplink Frequency /
Simulation Rate / Sensor stat grid was `absolute bottom-12 left-12`
inside the hero panel, independent of the normal-flow title/description/
button stack above it. At ordinary windowed browser heights (~700-770px,
not just small viewports) the button stack's natural height collided
with the stats pinned to the container bottom, rendering "RUN DEMO
SCENARIO" directly on top of "SYSTEM STATUS / NOMINAL". Switched the
stat grid to a normal-flow flex item pushed to the bottom with
`mt-auto` (falls back to flowing directly after the buttons, never
overlapping, if the panel is too short) instead of absolute positioning.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YTok4Srfs3CSL6Mp5TiYXD
@ThatKJ @claude
fix(web): make mission control responsive across mobile viewports
b042b66 
Real evidence, gathered via a new deterministic Playwright viewport suite
(tests/e2e/responsive.spec.ts, screenshots to generated/qa/mobile/, gitignored)
rather than the browser-extension resize tool, which reported success without
actually changing the rendered viewport.

Root cause, present on every page: TopBar is a `position:fixed` header with
no responsive handling -- at 390px its content (wordmark, SIH tag, scenario
picker, SIM clock, feed-status chip, avatar) extends to x=629px, invisible
past the viewport edge (fixed elements don't count toward
document.documentElement.scrollWidth, so this was invisible to a naive
overflow check -- caught only by visual screenshot review). Fixed by hiding
secondary chrome below `lg` (SIH tag, SIM clock text, feed-status text -> a
plain dot), abbreviating the wordmark to "FSOC", and giving the scenario
picker real flex space (its label was measured collapsing to a literal 0px
box -- a flex shrink-to-fit + truncate interaction, not just tight).

Other real, screenshot-confirmed issues fixed:
- Screen (AppShell): was `h-[calc(100vh-48px)] overflow-hidden` unconditionally,
  clipping any page whose content needed more height once stacked on mobile.
  Now scrolls vertically below `lg`, fixed/no-scroll at `lg:` and up (unchanged
  desktop behavior).
- Homepage: hero/viewport was a hard `w-1/3`/`w-2/3` split regardless of
  viewport, producing unreadably narrow hyphenated text at phone widths.
  Stacks below `lg`; secondary FPS/EXPOSURE/GAIN chips (which collided with
  the feed label at narrow widths) collapse on mobile.
- /mission: TelemetryStream and the Event Log panel were fixed `w-[320px]`,
  squeezing the primary tracking viewport to ~6px at 390px width. Both stack
  full-width below `lg`.
- /world: the telemetry aside was an absolute `w-[320px]` panel spanning
  nearly the full mobile viewport, hiding the 3D scene entirely. Now a
  compact `w-[168px]` top-right card on mobile; the view-switcher gets a
  matching max-width so its longest label wraps instead of colliding with it.
- /telemetry: the 4-stat metrics bar used unconstrained flex-1 columns whose
  labels wrapped inside a fixed h-[64px] row, clipping values. Given fixed
  widths + nowrap + contained horizontal scroll (explicitly sanctioned for
  telemetry bars) instead of forcing an unreadable fit.

Verified: 51/51 local Playwright (20 smoke + 31 responsive, 7 viewports x
3 pages + all 10 routes @ 390x844), 22/22 CTest, typecheck/lint/build clean.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YTok4Srfs3CSL6Mp5TiYXD
@ThatKJ @claude
chore(web): update Next.js to patched release
3a61337 
next 14.2.18 -> 14.2.35 (+ eslint-config-next in lockstep, per Next's own
convention). Resolves CVE-2025-55184 (High, DoS: a crafted HTTP request to
any App Router endpoint can hang the server via an infinite loop during
deserialization) for the 14.x line, per the official advisory
(nextjs.org/blog/security-update-2025-12-11). This app uses the App Router,
so it was in-scope. CVE-2025-55183 (source-code exposure) only affects
15.0+, not 14.x.

Patch-level bump within the same minor line, not a major migration -- no
React/TypeScript/ESLint version change. 5 packages touched in package-lock.json.

Other npm audit findings (postcss, playwright, uuid-via-@react-three/drei)
are pre-existing, unrelated to this CVE, and every suggested fix requires a
breaking/major-version change -- explicitly out of scope for this security
maintenance pass.

Verified: typecheck/lint/build clean, 51/51 local Playwright
(20 smoke + 31 responsive), 22/22 CTest.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YTok4Srfs3CSL6Mp5TiYXD
@ThatKJ @claude
test(e2e): make the responsive suite deterministic, not flaky-tolerant
683f708 
Found while re-verifying the mobile fix on Preview after the Next.js
upgrade: /scenarios occasionally failed the horizontal-overflow check
(17-25px, growing across runs) when the full 31-test suite ran with
Playwright's default 3 parallel workers. Investigated properly rather
than loosening the assertion blind:

- 12+ standalone reproductions (single page, 3 concurrent pages, matching
  the test's exact goto/wait conditions) never once reproduced it --
  always scrollWidth == innerWidth.
- Only ever reproduced inside the real multi-worker suite, and the
  measured overflow grew across successive runs -- a resource-contention
  signature (3 concurrent SwiftShader/software-rendered Chrome workers
  competing for CPU during layout of a text-dense, always-taller-than-
  viewport page), not a deterministic CSS bug.
- Confirmed via app/globals.css: `body { overflow-x: hidden }` is already
  a global rule, so normal-flow overflow (unlike the real TopBar bug this
  suite caught, which was `position:fixed` and NOT subject to that clip)
  was never visible or reachable by a real user regardless of the exact
  pixel count -- consistent with every screenshot in generated/qa/mobile/
  being clean throughout.

Fix: run this file serially (test.describe.configure({mode:"serial"})) --
it's a QA/diagnostic suite, not a speed-critical CI gate, so correctness
wins over parallel speed. 2 consecutive full remote runs (31/31 each) and
one local run (51/51 combined with smoke.spec.ts) confirm this eliminates
the flake rather than papering over it with a loose tolerance.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YTok4Srfs3CSL6Mp5TiYXD
@ThatKJ @claude
Merge feat/phone-camera-in-loop into live-camera integration branch
ba253a7 
Brings in the real-camera-in-the-loop prototype (FrameSource abstraction,
OpenCVCameraFrameSource, LiveTrackingSession reusing the frozen
detector/hybrid-perception/tracker pipeline, honest VirtualPanTiltActuator,
fsoc_live CLI, camera probe/view/calibration tools, and the /mission/live
Mission Control page) as the G1 starting point for the live-camera task.

Resolved README.md conflicts by combining main's rewritten landing-page
prose with the branch's Mobile Phone Camera-in-the-Loop / Hardware
boundary / Working modes sections, and correcting now-stale claims
(LICENSE file now exists; phone-camera work is merged, not a separate
branch).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YTok4Srfs3CSL6Mp5TiYXD
@ThatKJ @claude
G1: fix frame/telemetry identity+atomicity in the live-camera transport
d5992bc 
Adds fsoc::LiveFramePublisher (include/fsoc/live_frame_publisher.hpp,
src/live_frame_publisher.cpp): writes frame_<N>.jpg / telemetry_<N>.json
via write-to-temp-then-atomic-rename, then flips manifest.json (same
pattern) LAST, so any reader observing a frame index in the manifest is
guaranteed both files for that index already exist, complete. Replaces
fsoc_live's previous in-place overwrite of a single telemetry.json/
frame.jpg pair, which two independently-polled HTTP requests could read
mismatched or mid-write (docs/LIVE_DATA_AUDIT.md section 2). Bounded
retention (default 3 pairs) keeps generated/live/ from growing without
limit during a long session. 3 new unit tests.

Rewires apps/fsoc_live.cpp to use the publisher, adds a per-run sessionId
(a reconnect always gets a new one) and schemaVersion/sessionId telemetry
fields.

Rewires the frontend live-camera routes and /mission/live to fetch the
image by the exact frameIndex the telemetry response names (object-URL
fetch, not a cache-busted <img src>), 404/400 on a bad index rather than
falling back to any other frame, and drop the displayed frame when
sessionId changes so a reconnect doesn't inherit the previous session's
last image.

Verified: ctest 23/23 PASS, frontend typecheck/lint/build clean, Playwright
51/51 PASS, and an end-to-end smoke test against a hand-built fixture
confirming the manifest -> exact-frame-index wiring works without opening
a camera. Physical camera verification needs the operator (interactive
OS permission prompt) -- see docs/LIVE_REALDATA_TASK_STATE.md "Next action".

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YTok4Srfs3CSL6Mp5TiYXD
@ThatKJ @claude
G1 pixel-only mode + G2 recording, annotation, and real-dataset loader
c5bde72 
G1 additions (physical live baseline, still software-only pending an operator
run with a real camera):
- fsoc_live --uncalibrated: pixel-only tracking with no real FOV. Angular
  telemetry fields are always null in this mode (never derived from the
  placeholder FOV used internally for the otherwise-unused control math), and
  control/actuation is force-disabled. --manual-assist and the terminal
  error= line fall back to real pixel offsets. Lets a real camera preview run
  before calibration is ever a blocker.
- /mission/live: new CALIBRATION and SESSION ID rows, and a TIMING panel
  (measured PROCESSED FPS from the C++ side's own dtS, measured DISPLAYED FPS
  client-side). Poll interval 500ms -> 200ms.
- frontend/tests/e2e/live-camera.spec.ts: 6 route-mocked tests covering the
  previously-untested risks -- reconnect drops the prior session's frame, a
  pruned/missing frame doesn't crash or mismatch, honest no-session/stale
  states, recording panel reflects telemetry not optimistic state.

G2 (recording + annotation + real-dataset plumbing; no real reviewed
recordings exist yet -- that half of G2 and all of G3 stay blocked on the
operator):
- fsoc::RealSessionRecorder (include/fsoc/real_session_recorder.hpp,
  src/real_session_recorder.cpp): a second sink, separate from
  LiveFramePublisher's pruned preview buffer -- every accepted frame is kept,
  never pruned. Writes manifest.json (session/software/calibration/model
  identity, git commit captured at CMake configure time, counters,
  atomically rewritten every frame), raw frames/frame_<N>.jpg, append-only
  telemetry.jsonl, and events.jsonl for human markers. Synchronous per-frame
  writes -- a disk error is counted, never stalls tracking. 4 new unit tests.
- fsoc_live: Start/Stop Recording + event markers via a polled
  generated/live/command.txt (key=value, temp-then-rename), driven by new
  Start/Stop/Mark-Event controls on /mission/live wired through
  POST /api/live-camera/record. UI status always reflects polled telemetry,
  never optimistic local state.
- /mission/annotate: a practical (not a platform) tool to open a recording,
  scrub frames, label presence (present/partial/full-occlusion/absent/
  ambiguous), click-set the beacon center in raw pixel space, review a
  detector suggestion (never auto-saved as ground truth), and save/reload
  reviewed labels (labels.json, keyed by frame index). 4 new e2e tests.
- tools/ai/real_dataset.py + real_dataset_split.py: additive real-data
  dataset reader (same __getitem__ contract as the synthetic BeaconDataset)
  and group-based (by recording, never by frame) train/val/test split with
  an explicit too-few-groups error instead of a silently empty split.
  Validates required files, in-bounds/contradictory label centers, and
  applies an explicit ambiguous/unreviewed exclusion policy. 13 plumbing
  checks against clearly-labeled fixtures (not real training data).

Also extracted fsoc::write_text_atomic (include/fsoc/atomic_file_io.hpp) out
of LiveFramePublisher so RealSessionRecorder shares the same
temp-then-rename guarantee instead of duplicating it.

Verified: ctest 24/24 PASS, frontend typecheck/lint/build clean, Playwright
61/61 PASS, tools/ai/real_dataset_tests.py 13/13 PASS (.venv-ai).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YTok4Srfs3CSL6Mp5TiYXD
@ThatKJ @claude
fix: fsoc_live telemetry JSON was pretty-printed, breaking telemetry.…
97171a3 
…jsonl

to_json() built multi-line, indented JSON. LiveFramePublisher's standalone
per-frame files tolerated that fine, but RealSessionRecorder appends the same
string as one line into telemetry.jsonl -- discovered live while testing G2
recording end-to-end against a real webcam: a 2042-frame recording produced a
91890-line telemetry.jsonl (each frame's pretty-printed object spanning ~20
physical lines), which silently breaks any line-oriented JSONL reader
(including frontend/app/api/real-sessions/[recordingId]/route.ts's parser).

Made to_json() emit compact single-line JSON instead -- valid either way and
now safe for both consumers. Verified against a fresh real-camera run:
telemetry_<N>.json files are exactly 1 line and parse correctly.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YTok4Srfs3CSL6Mp5TiYXD
@ThatKJ @claude
fix: overlay reticle/label-marker position ignored image-container of…
82d7682 
…fset

Both /mission/live's detected-beacon reticle and /mission/annotate's
marker overlay computed CSS position as a fraction of the <img>'s own
rendered size (getBoundingClientRect()), but never accounted for the
img's own offset within its flex-centered container -- when the raw
frame's aspect ratio didn't exactly fill the panel (object-contain
letterboxing), every marker was pulled toward the panel's corner
instead of landing on the actual detected point. Caught live testing
against a real camera: the reticle sat near the top-left of the feed
instead of on the beacon.

Fixed by switching to offsetLeft/offsetTop/offsetWidth/offsetHeight
(relative to the img's offsetParent, the `relative` container) instead
of getBoundingClientRect(), which correctly includes that centering
offset. Verified live against the real camera afterward -- reticle
lands on the actual bright source.

Also fixes /mission/annotate's frame navigation, which assumed frame
indices start at 0 and run contiguously -- a recording's frameIndex is
the camera's own continuous counter across the whole fsoc_live session,
so a recording started partway through begins at a large, non-zero
index (e.g. 1000), and frame 0 usually doesn't exist in that recording
at all. Navigation now moves by position in the telemetry array
(cursor) instead of by raw frame-number arithmetic.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YTok4Srfs3CSL6Mp5TiYXD
@vercel

vercel Bot commented Sep 12, 2026 •
edited
Loading

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
fsoc Ready Ready Preview Sep 12, 2026 9:42am UTC

@ThatKJ @claude
test: fix annotate.spec.ts assertions after frame-title format change
849f145 
CI caught this: 82d7682 changed the frame counter title from
"Frame <frameIndex> / <totalFrames - 1>" to
"Frame <cursor + 1> / <totalFrames> (frameIndex <frameIndex>)" (fixing
navigation for a recording that doesn't start at frame 0), but the
existing tests still asserted on the old "Frame 0 / 2" / "Frame 1 / 2"
text. Updated to "Frame 1 / 3" / "Frame 2 / 3" to match. Verified
locally: 4/4 annotate.spec.ts pass, full suite 61/61 pass.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YTok4Srfs3CSL6Mp5TiYXD

This branch was successfully deployed

1 active deployment
Preview — 849f1454 Deployed Sep 12, 2026 by vercel[bot]
View deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@ThatKJ