Erlang AI Vision — Laptop Edge

A Qwen-powered edge agent that turns continuous camera and audio streams into a small number of verified, actionable events—then keeps working when the cloud does not.

This repository is the laptop edge tier of Erlang AI Vision, built by the Littlebois Team for the Qwen Cloud Global Hackathon. It sits between an ESP32-S3 camera and the cloud application, runs perception and first-pass reasoning locally, controls the camera, and sends only valuable events upstream.

Judge snapshot

Question	Answer
Track	Track 5 — EdgeAgent
What perceives?	YOLO video detection and optional YAMNet audio detection at the edge
What reasons?	Local Qwen VLM via Ollama for visual triage; cloud Qwen for higher-level verification and agent decisions
What acts?	ESP32 pan, tilt, and snapshot commands through guarded camera tools
What stays local?	Continuous video, routine detections, rejected candidates, and locally resolved low-value events
What reaches the cloud?	Escalated event metadata, a keyframe, and an event clip uploaded through a signed URL
What happens offline?	Local Qwen becomes the authority, events enter a disk-backed queue, and replay resumes when the backend recovers
Can it run without hardware?	Yes—the bundled simulator and sample video exercise the same bridge and pipeline

For a reproducible walkthrough, follow the five-minute demo without hardware below. It uses the bundled simulator and sample video, so judges can exercise the real bridge and pipeline without a camera.

System architecture

flowchart LR
    Camera[ESP32-S3 camera<br/>video + audio + pan/tilt] -->|Wi-Fi WebSocket or USB serial| Edge
    subgraph Edge[LaptopEdge — this repository]
        Detect[YOLO + optional YAMNet]
        Filter[Agent rule filter<br/>class · ROI · dwell · schedule · cooldown]
        LocalQwen[Local Qwen VLM<br/>Ollama]
        Control[Camera control<br/>off · auto-track · agent]
        Queue[Recorder + offline queue]
        Detect --> Filter --> LocalQwen --> Queue
        LocalQwen --> Control
    end
    Edge -->|valuable events only| Backend[FastAPI backend<br/>Alibaba Cloud]
    Backend --> CloudQwen[Qwen Cloud<br/>verification + agent decisions]
    Backend --> OSS[Alibaba OSS<br/>event clips]
    Backend --> App[Flutter console]
    CloudQwen -->|guarded action| Control
    Control -->|pan · tilt · snapshot| Camera

Loading

The Mermaid diagram above is the version-controlled source of truth and renders directly on GitHub.

Why this is an EdgeAgent

• Edge perception: each frame is processed locally; continuous sensor data is not sent to an AI API.
• Qwen reasoning: local Qwen rejects false positives; cloud Qwen verifies escalations and can choose camera actions.
• Physical action: the agent pans, tilts, or requests a snapshot through the real ESP32 transport.
• Bandwidth awareness: class, ROI, dwell, schedule, and cooldown gates run before Qwen or cloud escalation.
• Privacy: continuous footage and locally rejected events stay on the laptop.
• Graceful degradation: a circuit breaker, durable queue, local Qwen, and deterministic geometry keep the system useful offline.

As-built edge pipeline

Stage	Implementation	Behavior
Detect	pipeline/detectors/	YOLO video detection; optional YAMNet audio classification on Wi-Fi/simulator
Filter	pipeline/event_filter.py	Per-agent class, confidence, ROI, dwell, schedule, and cooldown gates
Local triage	pipeline/triage.py	Qwen VLM through Ollama returns structured trigger, confidence, and severity
Active verification	pipeline/verifier.py	Bounded pan and re-snapshot loop for uncertain events
Camera control	pipeline/tracker.py, pipeline/agent_controller.py	Deterministic tracking or Qwen-driven action selected at runtime
Record	pipeline/recorder.py	Save a short local clip and keyframe before routing
Route	pipeline/router.py, pipeline/queue.py	Drop, keep local, escalate, or queue for idempotent replay
Upload	pipeline/uploader.py	Best-effort clip upload through backend-issued OSS signed URLs
Measure	pipeline/metrics.py	Funnel, queue, latency, tokens, clips, bytes, upload, and pan metrics

Detection stays on the fast path; slower Qwen and routing work runs asynchronously so model or backend delays do not stop the live stream.

Three camera-control modes

Mode	Behavior
off	No autonomous movement; manual commands still work
auto_track	Deterministic subject tracking
agent	Qwen-assisted follow, patrol, or scan behavior from the assigned agent

Agent-mode fallback:

1. Cloud Qwen while the backend is reachable.
2. Local Qwen via Ollama when the cloud circuit breaker is open.
3. Deterministic geometry if both model paths are unavailable.

Movement is clamped, rate-limited, and gated by the current control mode.

Five-minute demo without hardware

Install

pwsh scripts/setup.ps1

Add -WithAudio for TensorFlow/YAMNet. Use -Check to verify without installing.

Run the operator console

cd src
python transport/edge_console.py

Select Simulator Video, choose demo_videos/family_living_room_footage.mp4, leave the edge token blank, click Start, then Open MJPEG.

Preview: http://localhost:8766/video.mjpg Health: http://localhost:8766/health

At INFO level, logs show candidates, Qwen verdicts, routing decisions, and funnel metrics.

CLI alternative

# Terminal A
cd src
python transport/edge_bridge.py --agents-file agents.example.json --log-level INFO

# Terminal B
cd src
python transport/simulate_device.py --video demo_videos/family_living_room_footage.mp4 --loop-video --tone

With no backend, escalations queue locally and the system enters degraded mode—an intentional offline demonstration.

Connect hardware and cloud

cd src

# ESP32 over USB serial
python transport/edge_bridge.py --serial-port COM5

# ESP32 over Wi-Fi plus cloud forwarding
python transport/edge_bridge.py --edge-token <raw-device-token> --api-base-url <backend-url>

Wi-Fi supports multiple cameras per receiver. USB is a direct single-camera path. Audio currently requires Wi-Fi/simulator because the serial protocol has no audio channel.

Important configuration

Setting	Purpose
--agents-file PATH	Load local agents without a backend
--serial-port COMx	Use USB serial instead of Wi-Fi
--edge-token TOKEN	Authenticate with the backend
--api-base-url URL	Set the backend URL
--no-pipeline	Run transport/preview without detector dependencies
--no-audio	Disable optional audio
--track	Start in auto-track mode
--track-tilt	Permit vertical auto-tracking
--log-file PATH	Write rotating JSONL events and metrics
SENTINELEDGE_TRIAGE_MODEL	Local Ollama vision model; default qwen3.5:0.8b
SENTINELEDGE_TRIAGE_UPGRADE_MODEL	Larger degraded-mode model, if installed
SENTINELEDGE_CLIP_UPLOAD=0	Disable cloud clips but retain local recording

See python transport/edge_bridge.py --help and the detailed architecture.

Metrics and evidence

The funnel tracks frames, candidates, local resolutions, cloud calls, queued/degraded events, latency percentiles, tokens, clips, bytes, upload failures, and pans. Its headline is filter_ratio: how few Stage-1 candidates require a cloud call.

python scripts/bench_triage.py --video src/demo_videos/family_living_room_footage.mp4 --runs 5

Capture results from the target judge hardware. This repository does not claim universal performance without a recorded benchmark environment.

Testing

From src/:

python -m pytest tests -q
python -m pytest tests/pipeline -q
python -m pytest tests/transport -q

Tests requiring Ollama or TensorFlow may skip when unavailable. To make missing real-model dependencies fail:

$env:REQUIRE_OLLAMA=1
$env:REQUIRE_TF=1
python -m pytest tests -q

Model-backed tests exercise YOLO, Qwen/Ollama, and YAMNet when installed. Injection and fakes keep control, failure, and timing tests deterministic.

Repository map

src/
├── pipeline/                # detection, Qwen triage, control, routing
│   ├── detectors/           # YOLO and optional YAMNet
│   ├── agent_controller.py  # cloud/local Qwen camera decisions
│   ├── orchestrator.py      # fast/slow pipeline coordination
│   ├── queue.py             # durable offline events
│   └── metrics.py           # edge/cloud funnel
├── transport/               # ESP32 hubs, bridge, console, simulator
├── tests/                   # pipeline and transport verification
├── demo_videos/             # no-hardware footage
└── agents.example.json      # local agent definitions

Privacy, resilience, and safeguards

• Continuous video stays at the edge unless live viewing is enabled.
• Local filtering and Qwen resolve routine events before escalation.
• Event IDs make queue replay idempotent.
• The disk-backed queue survives restarts and is bounded.
• Clip upload uses short-lived signed URLs; failed uploads remain local.
• Detector failures are isolated from transport and preview.
• Heavy dependencies load lazily for transport-only operation.
• Camera actions respect mode, clamps, minimum intervals, and bounded verification pans.

Known limitations

• Audio detection is not available over USB serial.
• Degraded events are recorded, queued, logged, and marked unverified; there is no dedicated local notification channel yet.
• Local VLM performance depends on CPU/GPU capability and keyframe resolution.
• The operator console targets a local Windows/Python environment; packaged installers are not provided.

Related repositories

• Erlang AI Vision — Fullstack: FastAPI, Qwen Cloud verification, Flutter, and Alibaba Cloud.
• Erlang AI Vision — IoT Camera: ESP32-S3 firmware, provisioning, transport, and pan/tilt.

All related repositories are public and linked using their current names.

License

Released under the MIT License. Copyright © 2026 Littlebois Team.