Skip to content

Rust core, protocol 1.1, Android camera work, Rust Linux desktop receiver - #3

Merged
spizganed merged 20 commits into
mainfrom
rust-rewrite-cloud
Sep 26, 2026
Merged

Rust core, protocol 1.1, Android camera work, Rust Linux desktop receiver#3
spizganed merged 20 commits into
mainfrom
rust-rewrite-cloud

Conversation

@spizganed

@spizganed spizganed commented Sep 26, 2026

Copy link
Copy Markdown
Owner

Same work as #2, moved to the rust-rewrite-cloud branch. It gets tested on a PC and a real phone before merging.

Cloud-session work on Tasks 1–8 and the final docs step. Each task has its own commits.

What changed

Task 1: salvaged app/lib/ui/screens/receiver_screen.dart from origin/test (there is no rust-test branch on the remote). It didn't compile because the _Preview constructor required a field that doesn't exist; fixed that. flutter analyze and flutter test pass.

Task 2: core ported to Rust (/core, crate lenny_core: cdylib + staticlib + rlib)

  • The C ABI in include/lenny/lenny.h is unchanged. core/tools/abi_check.sh regenerates the header with cbindgen and compares it with lenny.h: struct layouts, constant values and function types must match. The original C++ end-to-end tests pass unmodified against the Rust library (core/tests/c_abi, run under ASan/UBSan in CI).
  • Wire, session, transport, pairing, timing and c_api are ported. Tests are ported, including all vectors, and lenny_probe is ported too.
  • Android: Gradle runs cargo-ndk, and the JNI shim links the Rust .so. The .so sets a SONAME. Without it, the shim recorded the build machine's path and would not have loaded on a phone.
  • lenny_vcam: IVirtualCamera trait with two backends, V4l2LoopbackCamera and NullVirtualCamera, with automatic fallback to null.
  • CI, ADR-0006 (supersedes ADR-0003). Intentional deviations are listed in ADR-0006; none are visible on the wire or through the ABI.

Tasks 3–5 (Android, Camera2 kept on purpose)

  • Per-lens capability discovery, with no hardcoded sizes or frame rates. The stream only uses a mode the current lens lists.
  • Zoom (CONTROL_ZOOM_RATIO) and pan (SCALER_CROP_REGION) happen on the camera, not on finished frames. The software zoom/pan in the Flutter desktop preview is removed.
  • Protocol 1.1 adds per-lens modes and zoom in CAPS, the CONTROL pan command, and pan in CONTROL_STATE, all gated on the negotiated minor version. C ABI 1.3 appends fields to existing structs.
  • Auto | Manual replaces Auto + Lock focus. Every connect and reconnect now starts in Auto; the docs are updated.

Tasks 6–7: Rust desktop receiver (/desktop, egui, Linux)

  • Borderless window with a drawn title bar: real minimize, maximize/restore, close, drag, double-click and edge resize.
  • Layout adapts to the window size.
  • openh264 decoding.
  • The preview box takes the video's real aspect ratio.
  • Tap-to-focus, drag-to-pan, wheel zoom, QR, discovery, known phones, stats, and virtual camera status.
  • fake_phone sender and a headless loopback test.

Task 8: no emulator in the sandbox (no KVM). Coverage instead comes from JVM, widget and loopback tests; the new app.yml workflow runs them.

Test at home before merging

  • Real phone → Linux desktop over Wi-Fi: pairing (QR and Allow prompt), streaming, reconnect
  • sudo modprobe v4l2loopback exclusive_caps=1 card_label=Lenny, then Chrome, OBS and Zoom see "Lenny" live, with the placeholder when no phone is connected
  • Camera on a real phone: per-lens modes, lens switch, zoom and pan cropping, Manual tap-to-focus hold, back to Auto
  • 4K keyframes decode (openh264 rejects frames over 1 MB)
  • Windows: not in scope for this PR (ADR-0007)

🤖 Generated with Claude Code

https://claude.ai/code/session_01Kb92TxNwgV1nExrrfTzERt

Generated by Claude Code

claude added 20 commits September 26, 2026 17:14
@claude
docs: CLAUDE.md desktop build order (Linux first, Windows later)
0e976cd 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kb92TxNwgV1nExrrfTzERt
@claude
fix(ui): salvage receiver_screen.dart from the test branch
2371990 
Brought over only app/lib/ui/screens/receiver_screen.dart from origin/test
(the rust-test work). Fixed the _Preview constructor, which required a
nonexistent 'ystate' field and broke analysis.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kb92TxNwgV1nExrrfTzERt
@claude
core: port the C++ core to Rust behind the identical C ABI
e9003ad 
Rust crate lenny_core (rlib + cdylib + staticlib) with wire, session, transport,
pairing, timing and c_api modules, ported one to one from core/src/*.cpp. The C
ABI in include/lenny/lenny.h is unchanged: tools/abi_check.sh regenerates a
header with cbindgen and checks struct layouts, constant values and function
types against the hand-written one, and the original C++ end-to-end tests
(tests/test_session.cpp) pass unmodified when linked against the Rust library.

Tests ported to tests/wire.rs (incl. protocol/vectors) and tests/session.rs,
plus a struct-layout test. lenny_probe ported to src/bin/lenny_probe.rs.
The C++ sources stay until the build integration switches over.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kb92TxNwgV1nExrrfTzERt
@claude
core: switch builds to the Rust core, add lenny_vcam, CI and ADR-0006
9b4db91 
- Android: android_camera's Gradle build runs cargo-ndk (arm64, armv7, x86_64,
  x86) and the JNI shim links the Rust liblenny_core.so. build.rs sets its
  SONAME; without it the shim recorded the build machine's path as DT_NEEDED
  and would not load on a device.
- core/CMakeLists.txt now wraps cargo and exposes the same lenny_core target to
  CMake consumers (windows_receiver; not built on Windows yet).
- C++ core sources removed. The C++ session test stays in core/tests/c_abi as a
  C-caller conformance test against the Rust library.
- vcam/: IVirtualCamera trait with V4l2LoopbackCamera and NullVirtualCamera,
  automatic fallback to null, frame compose/letterbox/placeholder helpers.
- CI: fmt, clippy, tests, cbindgen ABI check, C ABI test under ASan/UBSan,
  cargo-ndk build.
- Docs: ADR-0006 (supersedes ADR-0003), README, architecture, testing, CLAUDE.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kb92TxNwgV1nExrrfTzERt
@claude
protocol 1.1: per-lens modes and zoom range, pan, ABI 1.3
aa413f2 
Wire (additive, gated on the negotiated minor so 1.0 peers see no change):
- CAPS lens entries carry their own modes (tag 4) and zoom range (tag 5), so a
  receiver only offers what the selected lens can stream.
- CONTROL pan (21) + controls bit 8: moves the zoomed crop across the lens's
  field of view; CONTROL_STATE pan_x/pan_y (10, 11).
- focus_at now holds until Auto; controls reset to Auto on every reconnect.
New vectors control_pan.hex and caps_lens_1_1.hex; tests for gating, including a
fake 1.0 phone over raw TCP.

C ABI 1.3 (appended, like 1.1/1.2): LENNY_CTL_PAN, LENNY_CAP_PAN,
lenny_control_state.pan_x/pan_y, lenny_sender_config.lens_caps +
lenny_lens_caps. Rust receivers get the full CAPS via Session::peer_caps().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kb92TxNwgV1nExrrfTzERt
@claude
android: per-lens camera discovery, capture-level pan, Auto/Manual modes
d3b4725 
Task 3: every lens reports its own modes, built from its StreamConfigurationMap,
AE fps ranges and the AVC encoder (no hardcoded size whitelist or 24/30/60 list),
and its zoom range (CONTROL_ZOOM_RATIO_RANGE, or max digital zoom before API 30).
Sent per lens in CAPS (protocol 1.1). The stream only uses a mode the current
lens lists; a lens switch moves to that lens's nearest mode and reports it in
STREAM_START instead of silently falling back.

Task 4: pan (CONTROL pan) moves SCALER_CROP_REGION over the lens's full field of
view on top of CONTROL_ZOOM_RATIO, which still drives sensor switching; the
ratio only drops as far as panning toward an edge needs. Pre-API 30 it's crop
only. The salvaged Flutter desktop preview's software zoom/pan
(InteractiveViewer + BoxFit.cover) is removed.

Task 5: Auto | Manual segmented control replaces Auto + Lock focus on the phone
and the Flutter desktop. Manual: tap-to-focus (holds), EV slider, exposure lock.
Auto sends reset_auto (3A only). Every connect/reconnect starts in Auto.

JNI: lens caps and pan in control state. Dart bindings regenerated (ffigen).
JVM unit tests for the crop/pan math and mode matching.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kb92TxNwgV1nExrrfTzERt
@claude
protocol 1.1: lens base zoom in CAPS zoom_range, for mapping pan drags
31bbee4 
A 2x lens of a logical camera at zoom 1.0 is already 2x on the camera's field
of view, which is what pan moves over. zoom_range gets a third field (base) and
lenny_lens_caps.zoom_base, so a receiver can turn a drag into the right pan
change. Vector, docs, JNI and Kotlin updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kb92TxNwgV1nExrrfTzERt
@claude
desktop: Rust receiver app (egui), Linux first
6ab4715 
lenny_desktop, eframe/egui over winit, linking lenny_core and lenny_vcam directly:
- Borderless window with an app-drawn title bar per docs/design.md: minimize,
  maximize/restore and close send real viewport commands; the bar drags the
  window, double-click maximizes, window edges resize (BeginResize).
- Layout sized from the window every frame (fractions + clamps): preview hero
  plus a control column when wide, one scrolling column when narrow.
- Engine: core receiver session; openh264 decoder thread; virtual camera
  thread writing a steady 30 fps (live frame, last good frame for 500 ms, then
  a placeholder with the reason); UDP discovery answers; single-use QR tokens
  renewed every 80 s and after each pairing; known phones persisted.
- Preview sized to the decoded frame's real aspect ratio after rotation
  (Task 6), never stretched. Manual mode: tap to focus. Zoomed: drag pans the
  phone's sensor crop, wheel zooms (Task 4, protocol 1.1).
- Cards: connection (QR, address chips, known phones, approval dialog),
  camera (lens, zoom, torch), Auto/Manual, video modes of the selected lens,
  stream stats and virtual camera status (active vs unavailable here).
- fake_phone: synthetic camera -> openh264 -> core sender, for trying the app
  without a phone; tests/loopback.rs runs the whole pipeline headless.

openh264 (built from source by the crate) over FFmpeg: no system libraries,
same build on Windows, BSD licensed, and it decodes the Baseline/Constrained
High streams Android's encoder produces. Decoded I420 feeds the I420 virtual
camera with no conversion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kb92TxNwgV1nExrrfTzERt
@claude
test: Auto/Manual widget tests, app CI workflow
498aa8e 
No emulator in this sandbox (no /dev/kvm; emulator -accel-check: "KVM requires
a CPU that supports vmx or svm"), so UI/logic coverage comes from Flutter widget
tests (Auto/Manual), JVM unit tests (camera math) and the fake-phone loopback
test in desktop/tests. app.yml runs analyze, widget tests, the APK build
(cargo-ndk via Gradle) and the JVM tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kb92TxNwgV1nExrrfTzERt
@claude
docs: ADR-0007 (Linux desktop first), where we are, desktop architect…
ecae0e2 
…ure and tests

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kb92TxNwgV1nExrrfTzERt
@claude
ci: fix clippy 1.98 collapsible_match, use the runner's ANDROID_NDK_ROOT
eb192ae 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kb92TxNwgV1nExrrfTzERt
@claude
tools: PowerShell script for a Linux test VM on Windows
4fd7223 
VirtualBox (winget) + Ubuntu 24.04 cloud disk + cloud-init seed ISO built with
IMAPI2: Xfce, v4l2loopback (/dev/video10 "Lenny"), OBS, Discord, Chromium, and
Lenny Desktop built from the branch. Bridged networking so the phone reaches
the VM directly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kb92TxNwgV1nExrrfTzERt
@claude
docs: handoff to the Windows PC session: verify PR #3, then the Direc…
2492fea 
…tShow + MF backends

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kb92TxNwgV1nExrrfTzERt
@claude
framebuf: shared-memory frame ring format for the Windows virtual cam…
6322e04 
…eras

Byte layout from architecture.md 7.3 (header, 3 seqlock slots, NV12,
heartbeat, state), a writer (desktop app) and a bounds-checked reader (the
camera DLLs). No dependencies, no OS code; tested here, including a
writer/reader race that must never return a torn frame and corrupt headers
that must not be read past the mapping.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kb92TxNwgV1nExrrfTzERt
@claude
vcam: Windows backend writing the shared frame ring, registering the …
b794b66 
…MF camera

WindowsCamera implements IVirtualCamera: creates Global\LennyFrames_v1 (DACL
from architecture.md 7.3; falls back to Local\ without SeCreateGlobalPrivilege,
DirectShow-only), writes I420 frames as NV12, signals the frame event, and on
Win11 registers the MF media source via MFCreateVirtualCamera (looked up at
runtime). Status text reports what's registered. open_best uses it on Windows.
Compile-checked for x86_64 and i686 Windows only; not run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kb92TxNwgV1nExrrfTzERt
@claude
vcam/dshow: DirectShow virtual camera filter in Rust (x64 + x86)
d6d4975 
Push source filter reading the lenny_framebuf shared-memory ring:
IBaseFilter + one capture pin (IPin, IAMStreamConfig, IKsPropertySet),
1280x720 NV12/YUY2 at 30 fps, allocator negotiation, streaming thread
from Pause, graph-clock timestamps while running. Last good frame for
500 ms, then the placeholder. Every COM entry point and the streaming
thread are wrapped in catch_unwind; a caught panic switches the filter
to placeholder-only. No lenny_core, no network code.

regsvr32 registration: COM class (ThreadingModel=Both) plus a
VideoInputDeviceCategory entry named "Lenny" (Win10) or
"Lenny (Classic)" (Win11). exports.def keeps x86 export names plain.

Compile-checked and clippy-clean for x86_64/i686-pc-windows-msvc only;
not linked or run yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kb92TxNwgV1nExrrfTzERt
@claude
vcam/com: add the Media Foundation virtual camera source; one DLL for…
ea87747 
… both cameras

Renames vcam/dshow to vcam/com (lenny_vcam_com.dll): one COM DLL with
the DirectShow filter and the MF media source that MFCreateVirtualCamera
points Frame Server at (CLSID {5F0F9024-...}). The MF side follows
Microsoft's SimpleMediaSource shape: IMFActivate (attributes forwarded to
a real store) -> IMFMediaSourceEx + IMFGetService + IKsControl, one
NV12 1280x720 capture stream (IMFMediaStream2 + IKsControl), samples
delivered from a worker thread paced to 30 fps, with the request token.

frames.rs holds the picture choice both cameras share (live, last good
for 500 ms, placeholder). regsvr32 registers both classes.

Compile-checked and clippy-clean for x86_64/i686-pc-windows-msvc only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kb92TxNwgV1nExrrfTzERt
@claude
CI: windows-latest job; in-process tests for both virtual cameras
e9101f5 
vcam/com/tests/com.rs drives the DirectShow filter in a real filter
graph (Render + Run) and the MF source through IMFActivate the way
Frame Server does (Start, RequestSample x3, paced). The windows job
runs clippy and the workspace tests on MSVC x64, the camera tests again
as x86, and a regsvr32 register/unregister round trip for both builds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kb92TxNwgV1nExrrfTzERt
@claude
docs: Windows virtual cameras (vcam/com, framebuf), CI windows job, h…
2806c7b 
…andoff update

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kb92TxNwgV1nExrrfTzERt
@claude
Fix struct-layout test on Windows: compare against the fixture with L…
67200fe 
…F line endings

A Windows checkout converts abi_layout_64.txt to CRLF. The layouts
themselves matched. The windows job now runs tests with --no-fail-fast.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kb92TxNwgV1nExrrfTzERt
@spizganed
spizganed merged commit 67200fe into main Sep 26, 2026
8 checks passed
@spizganed
spizganed deleted the rust-rewrite-cloud branch September 26, 2026 22:14
spizganed added a commit that referenced this pull request Sep 26, 2026
@spizganed @claude
docs: CLAUDE.md handoff, PR #3 merged
ea385c2 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@spizganed @claude