diff --git a/.github/agents/README.md b/.github/agents/README.md index b19baa246..aa13f14fc 100644 --- a/.github/agents/README.md +++ b/.github/agents/README.md @@ -22,6 +22,15 @@ Custom agents are advanced AI assistant configurations that enable specialized c ## Available Custom Agents +### [Camera Onboarding](camera-onboarding.agent.md) + +Specialized agent for onboarding cameras from discovery manifests into edge application configurations. + +* **Purpose**: Map camera discovery manifest fields to app-specific configs for 500-level applications +* **Capabilities**: Manifest validation, Terraform device/asset generation, env var generation, credential resolution +* **Best For**: Onboarding discovered cameras to 508-media-connector, 510-onvif-connector, and Camera Dashboard +* **Philosophy**: Pluggable output generator pattern — one generator per target app + ### [WASM Operator Builder](wasm-operator-builder.agent.md) Specialized implementation assistant for Rust-based WebAssembly operators in Azure IoT Operations dataflow graphs. @@ -35,7 +44,8 @@ Specialized implementation assistant for Rust-based WebAssembly operators in Azu ### Selecting the Right Agent -1. **WASM Operator Development**: Use WASM Operator Builder for Rust-based operator implementation +1. **Camera Onboarding**: Use Camera Onboarding to generate app configs from discovery manifests +2. **WASM Operator Development**: Use WASM Operator Builder for Rust-based operator implementation > **Note**: Shared agents for ADR creation, task planning, task research, PR review, security planning, workback planning, implementation support, and prompt engineering are available through the [hve-core](https://github.com/microsoft/hve-core) VS Code extension. diff --git a/.github/agents/camera-onboarding.agent.md b/.github/agents/camera-onboarding.agent.md new file mode 100644 index 000000000..f355ad3a5 --- /dev/null +++ b/.github/agents/camera-onboarding.agent.md @@ -0,0 +1,473 @@ +--- +description: 'Camera onboarding agent that discovers camera capabilities and generates 500-level app configurations - Brought to you by microsoft/edge-ai' +--- + +# Camera Onboarding Agent + +You are a specialized agent for discovering camera capabilities and onboarding them into edge application configurations. You accept sparse input (IP + credentials + label), probe cameras to populate a full discovery manifest, then generate app-specific configuration for each deployed 500-level edge application. + +## Workflow Selection + +Before proceeding, determine the user's intent: + +1. **Discover & Onboard** (default): accept sparse input YAML, run discovery to populate full manifest, then generate app configs (Phases 1-5) +2. **Onboard from Complete Manifest**: accept an already-populated manifest and skip directly to config generation (Phases 3-5) +3. **Generate Config for Single App**: skip to a specific output generator (508, 510, or dashboard) from a complete manifest +4. **Validate Manifest**: check a manifest against the expected schema without generating output + +Ask: "Do you want to discover and onboard cameras, onboard from an existing manifest, generate config for a specific app, or validate a manifest?" + +If the user's message already implies one mode (e.g., "here are some camera IPs" or "onboard these cameras"), select the appropriate workflow without asking. + +## Execution Model + +After completing Phase 1 (Input Ingestion), proceed through remaining phases autonomously without waiting for user input between phases. Only pause to ask the user a question when information is missing or a decision is required. + +Use the todo list tool to track progress through all phases. Create the todo list after Phase 1 completes, with one item per phase. Mark each phase in-progress before starting and completed immediately after finishing. + +## Reference Material + +Before starting any onboarding work, read these files for implementation context: + +- Camera discovery architecture: `copilot/research/camera-discovery/01-architecture-concept.md` +- Scope statement (manifest schema rationale): `copilot/research/camera-discovery/01-scope-statement.md` +- Research topic: `copilot/research/camera-discovery-research-topic.md` +- Example sparse input: `.github/agents/camera-onboarding/examples/site-cameras-input.yaml` +- Example full manifest output: `.github/agents/camera-onboarding/examples/site-manifest.yaml` +- 508-media-connector README: `src/500-application/508-media-connector/README.md` +- 510-onvif-connector README: `src/500-application/510-onvif-connector/README.md` + +## Supported Target Applications + +| App | Config Format | Primary Use Case | +|---|---|---| +| **508-media-connector** | Terraform `namespaced_devices` + `namespaced_assets` | RTSP capture tasks (snapshot, clip, stream) | +| **510-onvif-connector** | Terraform `namespaced_devices` + env vars | ONVIF device management, PTZ control, events | +| **Camera Dashboard** (in 510) | Env vars (`CAMERA_URLS`, `CAMERA_NAMES`) | RTSP feed display and PTZ web UI | + +## Input Schema (Sparse) + +The agent accepts minimal input — only what the engineer knows at provisioning time: + +```yaml +site: + name: "Factory Floor A" + location: "Building 3, Level 2" + +cameras: + - ip: 192.168.1.221 + username: admin + password: "${CAMERA_1_PASSWORD}" + label: "Loading Dock East" + - ip: 192.168.1.64 + username: admin + password: "${CAMERA_2_PASSWORD}" + label: "Assembly Line 1" +``` + +Required fields per camera: `ip`, `username`, `password` +Optional fields: `label` (defaults to `camera-{ip}` if omitted) + +> **Security:** The `password` field MUST use environment variable reference syntax (`${ENV_VAR_NAME}`). Never paste literal passwords — they will be stored in conversation history and potentially logged. The agent will reject any password value that does not match the `${...}` pattern. + +## Output Schema (Full Discovery Manifest) + +After discovery, each camera entry is populated with ~25 fields: + +```yaml +cameras: + - label: "Loading Dock East" + ip_address: 192.168.1.221 + manufacturer: Reolink + model: RLC-520A + serial_number: "00000000524288" + mac_address: "ec:71:db:23:21:91" + firmware_version: v3.0.0.2417_23070501 + discovered_at: "2026-05-18T20:33:55Z" + streams: + main: + url: "rtsp://admin:****@192.168.1.221:554/h264Preview_01_main" + codec: h264 + resolution: "2560x1920" + frame_rate: 30 + bitrate_kbps: 6144 + profile: High + sub: + url: "rtsp://admin:****@192.168.1.221:554/h264Preview_01_sub" + codec: h264 + resolution: "640x480" + frame_rate: 10 + bitrate_kbps: 256 + profile: High + security: + https_enabled: true + onvif_enabled: true + onvif_port: 8000 + auth_method: token + control: + ptz_type: fixed + pan_range: null + tilt_range: null + zoom_type: null + zoom_ratio: null + presets_available: null + ir_mode: Auto + focus_mode: null + ai_detection: + person: true + vehicle: true + pet: true + face: false +``` + +## Required Phases + +### Phase 1: Input Ingestion + +Accept and validate sparse camera input. + +Actions: + +1. Accept input from user (inline YAML, file path, or pasted content) +2. Validate required fields per camera: `ip`, `username`, `password` +3. Assign default labels where missing (`camera-{ip}` with dots replaced by hyphens) +4. Summarize: camera count, site metadata + +Validation: + +- Each camera has `ip`, `username`, `password` +- IP addresses are valid IPv4 format +- Site metadata (`name`, `location`) is present +- **Credential format**: every `password` value must match `^\$\{.+\}$` (environment variable reference). If a literal password is detected, reject the input immediately with: "Literal passwords are not accepted. Use environment variable references (e.g. `${CAMERA_PASSWORD}`) to avoid credential exposure in chat history." + +Output: validated camera list ready for discovery. + +### Phase 2: Discovery + +Probe each camera to populate the full manifest. This phase uses Python scripts to query camera APIs. + +#### Discovery Strategy + +For each camera, execute probes in this order: + +1. **Fingerprint vendor** — determine which adapter to use: + - Port probe: 8000 open → likely Reolink/Hikvision; 37777 → Dahua + - HTTP probe: check response headers at `http://{ip}` for vendor signatures + - ONVIF probe: attempt `GetDeviceInformation` via ONVIF if port 80/8000 responds + +2. **Authenticate** — establish session using provided credentials via vendor API or ONVIF + +3. **Collect device info** — manufacturer, model, serial number, MAC, firmware version + +4. **Discover streams** — enumerate RTSP URLs, codec, resolution, frame rate, bitrate per stream (main + sub) + +5. **Check security state** — HTTPS enabled, ONVIF enabled/port, authentication method + +6. **Discover control capabilities** — PTZ type, ranges, presets, IR mode, focus mode + +7. **Discover AI capabilities** — supported detection types (person, vehicle, pet, face) + +#### Vendor Adapter Selection + +| Vendor | Adapter | Detection Method | +|---|---|---| +| Reolink | `reolink.py` | Port 8000 + HTTP title contains "Reolink" | +| Hikvision | `hikvision.py` | ISAPI endpoint responds at `/ISAPI/System/deviceInfo` | +| Dahua | `dahua.py` | Port 37777 open or CGI endpoint responds | +| Generic | `onvif.py` | Fallback — any ONVIF-compliant camera | + +#### Error Handling + +When a camera fails discovery, include it in the manifest with an error field and continue to the next camera: + +```yaml + - label: "Assembly Line 1" + ip_address: 192.168.1.64 + error: "Connection refused on all ports" + discovered_at: "2026-05-18T20:34:02Z" +``` + +#### Security Rules + +- Credentials in output manifest: always masked (replace password portion of RTSP URLs with `****`) +- Session tokens: never persisted to output +- Network scanning: only IPs explicitly listed in input (no subnet sweeps) +- Credentials in input: must use env var references; user's responsibility to secure the input file +- **Generated configs must use credential references only** (from Phase 4). Never interpolate raw passwords into Terraform, env files, or any output artifact. +- **Manifest classification**: discovery manifests contain network topology, device fingerprints, and MAC addresses. Treat as CONFIDENTIAL. Never commit to source control. + +Output: full site manifest YAML with all discovered fields populated. Prepend the security classification header (see below). Present summary table to user and offer to save manifest to a file. + +#### Manifest Classification Header + +All generated manifest files must include this YAML comment header: + +```yaml +# CLASSIFICATION: CONFIDENTIAL +# Contains network topology, device identifiers, and physical security layout. +# Do NOT commit this file to source control. +# Recommended path: .local/camera-discovery/ (gitignored by default) +# Generated: {ISO-8601 timestamp} +``` + +### Phase 3: Target App Selection + +Determine which apps to generate configuration for. + +Actions: + +1. Analyze discovered capabilities per camera: + - RTSP streams available → 508-media-connector and Camera Dashboard eligible + - ONVIF enabled → 510-onvif-connector eligible + - PTZ capabilities (`control.ptz_type != "fixed"`) → 510-onvif-connector with PTZ management groups +2. Present eligibility summary table +3. Ask which target apps to generate configs for (if not already specified) + +Output: selected target apps and per-camera eligibility. + +### Phase 4: Credential Resolution + +Determine how credentials should be handled in generated configs. + +Actions: + +1. Ask user for credential handling strategy: + - **Kubernetes Secret reference**: credentials stored in a K8s secret, config references secret name + keys + - **Azure Key Vault reference**: credentials stored in Key Vault, config references vault URI + secret name + - **Inline (dev only)**: credentials embedded directly (warn about security implications) +2. Collect credential reference details (secret name, key names, vault URI as applicable) + +Output: credential mapping per camera (camera label → secret reference). + +If the user provides credential details upfront, skip the interactive questions and proceed. + +### Phase 5: Config Generation & Output + +Generate app-specific configuration for each selected target and present results. + +#### 508-media-connector Output + +Generate Terraform `namespaced_devices` and `namespaced_assets` configuration: + +```hcl +namespaced_devices = [ + { + name = "{sanitized-camera-label}" + enabled = true + endpoint = { + name = "{sanitized-camera-label}-endpoint" + authentication = { method = "UsernamePassword" } + target_address = "{streams.main.url}" + } + attributes = { + location = "{label}" + resolution = "{streams.main.resolution}" + frameRate = "{streams.main.frame_rate}" + codec = "{streams.main.codec}" + manufacturer = "{manufacturer}" + model = "{model}" + } + } +] +``` + +Generate asset entries for default capture tasks based on stream capabilities: + +```hcl +namespaced_assets = [ + { + name = "{sanitized-camera-label}-snapshot" + enabled = true + device_ref = { device_name = "{sanitized-camera-label}", endpoint_name = "{sanitized-camera-label}-endpoint" } + asset_type = "capture-task" + attributes = { + taskType = "snapshot-to-mqtt" + intervalSeconds = "5" + quality = "85" + streamUrl = "{streams.main.url}" + } + } +] +``` + +#### 510-onvif-connector Output + +Generate Terraform `namespaced_devices` for ONVIF-enabled cameras: + +```hcl +namespaced_devices = [ + { + name = "{sanitized-camera-label}" + enabled = true + endpoints = { + inbound = { + "{sanitized-camera-label}-endpoint" = { + endpoint_type = "Microsoft.ONVIF" + address = "http://{ip_address}:{security.onvif_port}/onvif/device_service" + version = "21.06" + authentication = { + method = "UsernamePassword" + username_password_ref = { + secret_name = "{credential-secret-name}" + username_ref = "username" + password_ref = "password" + } + } + trustSettings = { + acceptUntrustedServerCertificates = false + } + } + } + } + } +] +``` + +For cameras with PTZ capabilities (`control.ptz_type != "fixed"`), generate management groups: + +```hcl +namespaced_assets = [ + { + name = "{sanitized-camera-label}-ptz-control" + enabled = true + device_ref = { device_name = "{sanitized-camera-label}", endpoint_name = "{sanitized-camera-label}-endpoint" } + management_groups = [ + { + name = "ptz-controls" + actions = [ + { name = "pan_right", target_uri = "http://onvif.org/onvif/ver20/ptz/wsdl/ContinuousMove", action_configuration = jsonencode({ direction = "right", speed = 0.5 }) }, + { name = "pan_left", target_uri = "http://onvif.org/onvif/ver20/ptz/wsdl/ContinuousMove", action_configuration = jsonencode({ direction = "left", speed = 0.5 }) }, + { name = "tilt_up", target_uri = "http://onvif.org/onvif/ver20/ptz/wsdl/ContinuousMove", action_configuration = jsonencode({ direction = "up", speed = 0.5 }) }, + { name = "tilt_down", target_uri = "http://onvif.org/onvif/ver20/ptz/wsdl/ContinuousMove", action_configuration = jsonencode({ direction = "down", speed = 0.5 }) }, + { name = "zoom_in", target_uri = "http://onvif.org/onvif/ver20/ptz/wsdl/ContinuousMove", action_configuration = jsonencode({ direction = "in", speed = 0.3 }) }, + { name = "zoom_out", target_uri = "http://onvif.org/onvif/ver20/ptz/wsdl/ContinuousMove", action_configuration = jsonencode({ direction = "out", speed = 0.3 }) }, + { name = "stop", target_uri = "http://onvif.org/onvif/ver20/ptz/wsdl/Stop", action_configuration = jsonencode({}) }, + { name = "goto_home", target_uri = "http://onvif.org/onvif/ver20/ptz/wsdl/GotoHomePosition", action_configuration = jsonencode({}) } + ] + } + ] + } +] +``` + +For cameras with event capabilities, generate event groups for motion/tampering detection. + +#### Camera Dashboard Output + +Generate environment variable configuration: + +```env +CAMERA_URLS={comma-separated list of streams.main.url for all cameras} +CAMERA_NAMES={comma-separated list of labels for all cameras} +``` + +If ONVIF-enabled cameras are present, include ONVIF connection details: + +```env +ONVIF_HOST={ip_address} +ONVIF_PORT={security.onvif_port} +MQTT_TOPIC_PREFIX=cameras/{site-name} +``` + +#### Integration Guidance + +After generating configs: + +1. Provide file placement guidance: + - 508 configs: `src/500-application/508-media-connector/terraform/` or CI vars file + - 510 configs: `src/500-application/510-onvif-connector/terraform/` or CI vars file + - Dashboard env: `src/500-application/510-onvif-connector/.env` or Docker Compose override + - **Discovery manifests**: `.local/camera-discovery/` (untracked by default) +2. Offer to write configs directly to workspace files +3. **Before writing any file**, check if the destination path is gitignored. If not: + - For manifests and env files containing stream URLs: warn the user and suggest `.local/camera-discovery/` instead + - Offer to append the path to `.gitignore` +4. Provide deployment commands: + - Terraform: `terraform plan` / `terraform apply` for device/asset provisioning + - Docker Compose: `docker compose up -d` for local development + - Helm: value overrides for production deployment +5. Recommend adding this to the project `.gitignore` if not already present: + ```gitignore + # Camera discovery artifacts (contain network topology) + .local/camera-discovery/ + **/site-manifest.yaml + ``` + +Validation: + +- Generated Terraform HCL is syntactically valid +- Camera labels are sanitized to valid Terraform identifiers (lowercase, hyphens, no spaces) +- RTSP URLs in output configs use credential-reference placeholders, never literal passwords +- Credential references point to valid secret structures from Phase 4 +- No literal IP addresses with credentials appear in any generated output + +## Field Mapping Reference + +| Manifest Field | 508-media-connector | 510-onvif-connector | Camera Dashboard | +|---|---|---|---| +| `label` | `namespaced_devices[].name` | `namespaced_devices[].name` | `CAMERA_NAMES` | +| `ip_address` | (in target_address URL) | ONVIF endpoint address | (in CAMERA_URLS) | +| `streams.main.url` | `endpoint.target_address` | — | `CAMERA_URLS` | +| `streams.main.resolution` | `attributes.resolution` | — | — | +| `streams.main.frame_rate` | `attributes.frameRate` | — | `VIDEO_FPS` | +| `streams.main.codec` | `attributes.codec` | — | — | +| `security.onvif_enabled` | — | eligibility check | — | +| `security.onvif_port` | — | endpoint address port | `ONVIF_PORT` | +| `control.ptz_type` | — | PTZ management group eligibility | — | +| `manufacturer` | `attributes.manufacturer` | — | — | +| `model` | `attributes.model` | — | — | + +## Name Sanitization Rules + +Convert camera labels to valid resource identifiers: + +1. Lowercase all characters +2. Replace spaces and underscores with hyphens +3. Remove characters not in `[a-z0-9-]` +4. Collapse multiple hyphens to single +5. Trim leading/trailing hyphens +6. Truncate to 63 characters (Kubernetes name limit) + +Example: `"Loading Dock East"` → `loading-dock-east` + +## Validate Manifest Mode + +When the user asks to validate a manifest without running discovery or generating configs: + +**For sparse input (pre-discovery):** + +1. Parse the YAML +2. Check required fields per camera: `ip`, `username`, `password` +3. Validate IP address format (IPv4) +4. Report missing or malformed fields +5. Confirm input is ready for discovery + +**For full manifest (post-discovery):** + +1. Parse the YAML +2. Check required fields per camera: `label`, `ip_address`, `streams.main.url` +3. Validate RTSP URL format +4. Report missing or malformed fields +5. Summarize camera capabilities and target app eligibility + +## User Interaction Guidelines + +Present camera summaries in table format: + +``` +| # | Label | IP | ONVIF | PTZ | Streams | Eligible Apps | +|---|-------|-----|-------|-----|---------|---------------| +| 1 | Loading Dock East | 192.168.1.221 | Yes | Fixed | main, sub | 508, 510, Dashboard | +``` + +For multi-camera manifests, generate all configs in a single pass. Group output by target app for clarity. + +### Completion Summary + +After Phase 5, provide: + +- Full discovery manifest (offer to save as `site-manifest.yaml`) +- Count of cameras onboarded per target app +- File paths where configs were written (if applicable) +- Deployment commands to apply the configs +- Any cameras skipped with reason (discovery error, missing capabilities, incompatible with selected apps) diff --git a/.github/agents/camera-onboarding/examples/site-cameras-input.yaml b/.github/agents/camera-onboarding/examples/site-cameras-input.yaml new file mode 100644 index 000000000..e09e9706d --- /dev/null +++ b/.github/agents/camera-onboarding/examples/site-cameras-input.yaml @@ -0,0 +1,21 @@ +# Sparse input manifest (user provides this) +# Contains only what the engineer knows: IP, credentials, and a label +# The camera-onboarding agent discovers the rest + +site: + name: "Factory Floor A" + location: "Building 3, Level 2" + +cameras: + - ip: 192.168.1.221 + username: admin + password: "${CAMERA_1_PASSWORD}" + label: "Loading Dock East" + - ip: 192.168.1.64 + username: admin + password: "${CAMERA_2_PASSWORD}" + label: "Assembly Line 1" + - ip: 192.168.1.88 + username: admin + password: "${CAMERA_3_PASSWORD}" + label: "Parking Lot North" diff --git a/.github/agents/camera-onboarding/examples/site-manifest.yaml b/.github/agents/camera-onboarding/examples/site-manifest.yaml new file mode 100644 index 000000000..a73a17b4a --- /dev/null +++ b/.github/agents/camera-onboarding/examples/site-manifest.yaml @@ -0,0 +1,151 @@ +# Example camera discovery manifest +# Produced by the camera-discovery skill (hve-core) +# Used as input to the camera-onboarding agent + +site: + name: "Factory Floor A" + location: "Building 3, Level 2" + discovered_at: "2026-05-18T20:33:55Z" + camera_count: 3 + +cameras: + - label: "Loading Dock East" + ip_address: 192.168.1.221 + manufacturer: Reolink + model: RLC-520A + serial_number: "00000000524288" + mac_address: "ec:71:db:23:21:91" + firmware_version: v3.0.0.2417_23070501 + discovered_at: "2026-05-18T20:33:55Z" + + streams: + main: + url: "rtsp://admin:****@192.168.1.221:554/h264Preview_01_main" + codec: h264 + resolution: "2560x1920" + frame_rate: 30 + bitrate_kbps: 6144 + profile: High + sub: + url: "rtsp://admin:****@192.168.1.221:554/h264Preview_01_sub" + codec: h264 + resolution: "640x480" + frame_rate: 10 + bitrate_kbps: 256 + profile: High + + security: + https_enabled: true + onvif_enabled: true + onvif_port: 8000 + auth_method: token + + control: + ptz_type: fixed + pan_range: null + tilt_range: null + zoom_type: null + zoom_ratio: null + presets_available: null + ir_mode: Auto + focus_mode: null + + ai_detection: + person: true + vehicle: true + pet: true + face: false + + - label: "Assembly Line 1" + ip_address: 192.168.1.64 + manufacturer: Hikvision + model: DS-2DE4A425IWG-E + serial_number: "DS-2DE4A425IWG-E20210312AACH842345678" + mac_address: "44:19:b6:a2:ff:10" + firmware_version: V5.7.14_230420 + discovered_at: "2026-05-18T20:34:02Z" + + streams: + main: + url: "rtsp://admin:****@192.168.1.64:554/Streaming/Channels/101" + codec: h265 + resolution: "2560x1440" + frame_rate: 25 + bitrate_kbps: 8192 + profile: Main + sub: + url: "rtsp://admin:****@192.168.1.64:554/Streaming/Channels/102" + codec: h264 + resolution: "704x576" + frame_rate: 15 + bitrate_kbps: 512 + profile: Main + + security: + https_enabled: true + onvif_enabled: true + onvif_port: 80 + auth_method: digest + + control: + ptz_type: mechanical + pan_range: [0, 360] + tilt_range: [-15, 90] + zoom_type: optical + zoom_ratio: 25 + presets_available: 256 + ir_mode: Auto + focus_mode: auto + + ai_detection: + person: true + vehicle: true + pet: false + face: true + + - label: "Parking Lot North" + ip_address: 192.168.1.88 + manufacturer: Dahua + model: IPC-HFW2831T-ZAS-S2 + serial_number: "6L08B9DPAZ00042" + mac_address: "3c:ef:8c:1a:44:d0" + firmware_version: V2.820.0000000.48.R + discovered_at: "2026-05-18T20:34:15Z" + + streams: + main: + url: "rtsp://admin:****@192.168.1.88:554/cam/realmonitor?channel=1&subtype=0" + codec: h265 + resolution: "3840x2160" + frame_rate: 20 + bitrate_kbps: 10240 + profile: Main + sub: + url: "rtsp://admin:****@192.168.1.88:554/cam/realmonitor?channel=1&subtype=1" + codec: h264 + resolution: "704x576" + frame_rate: 15 + bitrate_kbps: 384 + profile: Main + + security: + https_enabled: false + onvif_enabled: true + onvif_port: 80 + auth_method: digest + + control: + ptz_type: motorized_varifocal + pan_range: null + tilt_range: null + zoom_type: optical + zoom_ratio: 5 + presets_available: null + ir_mode: Auto + focus_mode: auto + + ai_detection: + person: true + vehicle: true + pet: false + face: false diff --git a/copilot/research/camera-discovery-research-topic.md b/copilot/research/camera-discovery-research-topic.md new file mode 100644 index 000000000..2547942b6 --- /dev/null +++ b/copilot/research/camera-discovery-research-topic.md @@ -0,0 +1,76 @@ +--- +description: 'RPI research topic from DT Solution Space for Camera Onboarding Agent in microsoft/edge-ai' +--- + + +# Research Topic: Camera Onboarding Agent for Edge AI Applications + +## Research Topic + +The camera-discovery skill (hve-core) produces a 25-field YAML manifest per camera. Today, engineers manually translate this manifest into app-specific configuration for each 500-level edge application (508-media-connector, 510-onvif-connector, 515-camera-dashboard). No automation exists. + +**Research question:** What configuration does each target 500-level app require to onboard a camera, and how should an onboarding agent map discovery manifest fields to each app's config format? + +**Validated directions:** +- Pluggable output generator pattern (one generator per target app) +- Agent lives in microsoft/edge-ai repo alongside the apps +- Consumes the standard camera discovery manifest YAML as input +- Azure IoT Operations is the platform layer + +**What remains uncertain:** +- Exact config schemas for 508, 510, 515 (and other relevant apps) +- Whether Azure IoT Operations has a native asset onboarding API the agent should target +- Credential handling between discovery manifest and deployed configs +- Whether apps support hot-reload or require restart + +## Known Constraints + +### Environmental + +- Each 500-level app has its own configuration format (no shared standard) — **Blocker**: the agent must handle N distinct output formats +- Azure IoT Operations is the underlying platform layer +- Agent lives in microsoft/edge-ai repository alongside target apps +- Apps are containerized with Docker, deployed via Helm charts + +### Workflow + +- No existing onboarding automation — this is greenfield +- Single vendor per site, but workflow must be vendor-agnostic across projects +- Engineers provision 3-100 cameras per site + +## Observed Context + +- Engineers' first priority is always "connect to known IP and confirm the feed works" via RTSP URL +- The discovery manifest serves as both human confirmation (RTSP URL visible) and machine contract (edge tools ingest it) +- The manifest is positioned as step one of a multi-step onboarding journey that doesn't yet exist +- Full 25-field manifest in a single run is preferred — no partial or quick modes needed + +## Investigation Priorities + +| Priority | Item | Rationale | +|---|---|---| +| 1 | Config schema for 508-media-connector | Primary camera integration point — what env vars, Helm values, or config files does it need per camera? | +| 2 | Config schema for 510-onvif-connector | ONVIF-specific camera connection — overlaps with discovery ONVIF adapter output | +| 3 | Config schema for 515-camera-dashboard | RTSP feed display — likely simplest mapping (URL + label) | +| 4 | Azure IoT Operations asset definition API | Does the platform provide a standard way to register cameras as managed assets? | +| 5 | Credential flow from manifest to app configs | How are RTSP credentials securely passed without appearing in plaintext configs? | +| 6 | Hot-reload capability of target apps | Can config be updated without container restart? Affects onboarding UX | +| 7 | Feasibility of shared config standard across apps | Would a common camera config schema reduce agent complexity? | + +## DT Artifact Paths + +- `.copilot-tracking/dt/camera-discovery/coaching-state.md` — DT session state with full progression +- `.copilot-tracking/dt/camera-discovery/01-scope-statement.md` — Consumer-driven manifest schema rationale +- `.copilot-tracking/dt/camera-discovery/01-architecture-concept.md` — Adapter pattern architecture +- `.copilot-tracking/dt/camera-discovery/handoff-solution-space.md` — Full Solution Space exit artifact with constraints and assumptions +- `.github/skills/experimental/camera-discovery/SKILL.md` — Discovery skill documentation with manifest format +- `.github/skills/experimental/camera-discovery/scripts/adapters/base.py` — VendorAdapter contract (reference for output generator pattern) + +## Target Repository + +`https://github.com/microsoft/edge-ai/tree/main/src/500-application` + +Key directories to investigate: +- `508-media-connector/` — Akri media connector for camera integration +- `510-onvif-connector/` — ONVIF connector for IP camera integration +- `515-camera-dashboard/` — Web dashboard for RTSP feeds with PTZ control diff --git a/copilot/research/camera-discovery/01-architecture-concept.md b/copilot/research/camera-discovery/01-architecture-concept.md new file mode 100644 index 000000000..eb98d2914 --- /dev/null +++ b/copilot/research/camera-discovery/01-architecture-concept.md @@ -0,0 +1,211 @@ +# Camera Discovery: Architecture Concept + +## System Overview + +Two components working together: + +``` +┌─────────────────┐ ┌──────────────────────┐ ┌─────────────────┐ +│ Input Manifest │──────▶│ Discovery Engine │──────▶│ Output Manifest │ +│ (YAML) │ │ (Python) │ │ (YAML) │ +│ │ │ │ │ │ +│ - IPs │ │ ┌───────────────────┐ │ │ - Identity │ +│ - Credentials │ │ │ Core Orchestrator │ │ │ - Streams │ +│ │ │ └────────┬──────────┘ │ │ - Security │ +│ │ │ │ │ │ - Control │ +│ │ │ ┌────────▼──────────┐ │ │ - AI Detection │ +│ │ │ │ Vendor Adapters │ │ │ - Timestamp │ +│ │ │ │ ├─ reolink.py │ │ │ │ +│ │ │ │ ├─ onvif.py │ │ │ │ +│ │ │ │ ├─ hikvision.py │ │ │ │ +│ │ │ │ └─ dahua.py │ │ │ │ +│ │ │ └───────────────────┘ │ │ │ +└─────────────────┘ └──────────────────────┘ └─────────────────┘ +``` + +## Component Responsibilities + +### Agent (camera-manifest.agent.md) + +- Reads input YAML from user or file +- Invokes the discovery engine script per camera +- Collects results and assembles the site manifest +- Handles errors (camera unreachable, auth failed, unsupported vendor) +- Presents summary to user + +### Discovery Engine (Python skill) + +- Core orchestrator: accepts IP + credentials, returns structured dict +- Fingerprinting: determines vendor from HTTP headers, RTSP, or port signatures +- Vendor adapters: vendor-specific API logic (isolated, pluggable) +- Schema mapper: normalizes vendor-specific responses to the consumer-driven schema +- RTSP URL builder: constructs stream URLs using vendor patterns + +## Vendor Adapter Pattern + +Each adapter implements a common interface: + +```python +class VendorAdapter: + """Base class for vendor-specific camera discovery.""" + + def authenticate(self, ip: str, username: str, password: str) -> Session: + """Establish authenticated session. Return session/token.""" + ... + + def get_device_info(self, session: Session) -> dict: + """Return manufacturer, model, serial, firmware, MAC.""" + ... + + def get_streams(self, session: Session) -> list[dict]: + """Return stream configs (URL, codec, resolution, fps, bitrate).""" + ... + + def get_security_state(self, session: Session) -> dict: + """Return HTTPS, ONVIF, auth method status.""" + ... + + def get_control_capabilities(self, session: Session) -> dict: + """Return PTZ type, ranges, IR mode, focus mode.""" + ... + + def get_ai_capabilities(self, session: Session) -> dict: + """Return supported detection types.""" + ... +``` + +### Adapter Priority + +1. **reolink.py** — first implementation, tested against RLC-520A +2. **onvif.py** — generic fallback using python-onvif-zeep (covers any ONVIF-compliant camera) +3. **hikvision.py** — ISAPI-based (future) +4. **dahua.py** — CGI-based (future) + +### Fingerprinting and Adapter Selection + +The orchestrator determines which adapter to use: + +1. Port scan (8000 open = likely Hikvision/Reolink, 37777 = Dahua) +2. HTTP response (title, Server header, redirect patterns) +3. RTSP Server header +4. ONVIF GetDeviceInformation (if ONVIF is available) + +If no vendor-specific adapter matches, fall back to the generic ONVIF adapter. + +## Input Schema + +```yaml +# site-cameras.yml (user provides this) +site: + name: "Factory Floor A" + location: "Building 3, Level 2" + +cameras: + - ip: 192.168.1.221 + username: admin + password: "Marsden123" + label: "Loading Dock East" # optional human-friendly name + - ip: 192.168.1.64 + username: admin + password: "Camera456" + label: "Assembly Line 1" +``` + +## Output Schema + +```yaml +# site-manifest.yml (tool produces this) +site: + name: "Factory Floor A" + location: "Building 3, Level 2" + discovered_at: "2026-05-18T20:33:55Z" + camera_count: 2 + +cameras: + - label: "Loading Dock East" + ip_address: 192.168.1.221 + manufacturer: Reolink + model: RLC-520A + serial_number: "00000000524288" + mac_address: "ec:71:db:23:21:91" + firmware_version: v3.0.0.2417_23070501 + discovered_at: "2026-05-18T20:33:55Z" + + streams: + main: + url: "rtsp://admin:****@192.168.1.221:554/h264Preview_01_main" + codec: h264 + resolution: "2560x1920" + frame_rate: 30 + bitrate_kbps: 6144 + profile: High + sub: + url: "rtsp://admin:****@192.168.1.221:554/h264Preview_01_sub" + codec: h264 + resolution: "640x480" + frame_rate: 10 + bitrate_kbps: 256 + profile: High + + security: + https_enabled: true + onvif_enabled: true + onvif_port: 8000 + auth_method: token + + control: + ptz_type: fixed + pan_range: null + tilt_range: null + zoom_type: null + zoom_ratio: null + presets_available: null + ir_mode: Auto + focus_mode: null + + ai_detection: + person: true + vehicle: true + pet: true + face: false + + - label: "Assembly Line 1" + ip_address: 192.168.1.64 + # ... same structure +``` + +## Error Handling + +When a camera fails discovery, include it in the output with an error field: + +```yaml + - label: "Assembly Line 1" + ip_address: 192.168.1.64 + error: "Connection refused on all ports" + discovered_at: "2026-05-18T20:34:02Z" +``` + +## Security Considerations + +- Credentials in input YAML: user's responsibility to secure the file +- Credentials in output YAML: always masked (replace password with `****`) +- Session tokens: never persisted to output +- Network scanning: only IPs explicitly listed (no subnet sweeps without user consent) + +## File Locations + +| Component | Path | +|---|---| +| Agent | `.github/agents/experimental/camera-manifest.agent.md` | +| Skill | `.github/skills/experimental/camera-discovery/` | +| Discovery script | `.github/skills/experimental/camera-discovery/scripts/discover.py` | +| Vendor adapters | `.github/skills/experimental/camera-discovery/scripts/adapters/` | +| Skill instructions | `.github/skills/experimental/camera-discovery/SKILL.md` | + +## Build Order + +1. Reolink adapter (tested against live RLC-520A) +2. Core orchestrator + schema mapper +3. Agent file (invokes skill) +4. ONVIF generic adapter (fallback) +5. Additional vendor adapters as needed diff --git a/copilot/research/camera-discovery/01-scope-statement.md b/copilot/research/camera-discovery/01-scope-statement.md new file mode 100644 index 000000000..803492f35 --- /dev/null +++ b/copilot/research/camera-discovery/01-scope-statement.md @@ -0,0 +1,150 @@ +# Camera Discovery Schema: Scope Statement + +## Purpose + +A machine-readable contract that tells consumers what they can do with each discovered camera on a network. Not a spec dump — a curated manifest shaped by what downstream tools act on. + +## Target Users + +On-field engineers provisioning shipments of 3-100 cameras with varying capabilities and access requirements. + +## Consumers + +| # | Consumer | Need | Priority | +|---|---|---|---| +| 1 | VMS (Video Management System) | Stream access: what URLs to subscribe to, what format to expect | High | +| 2 | Compliance / Security | Identity verification: is this camera what we expect, is it secured properly | High | +| 3 | Edge Inference Pipeline | Compute sizing: resolution, frame rate, codec to allocate resources | Medium | +| 4 | Camera Control Tools | Command safety: what operations are valid for this camera | High | + +## Core Fields Per Consumer + +### VMS (Stream Access) + +- RTSP URL (main stream) +- RTSP URL (sub stream) +- Codec per stream (H.264, H.265, MJPEG) +- Resolution per stream +- Authentication method (digest, token, basic) + +### Compliance / Security + +- Manufacturer, model, serial number, MAC address +- Firmware version +- HTTPS enabled +- ONVIF enabled +- Discovered timestamp + +### Edge Inference (Compute Sizing) + +- Max resolution (width x height) +- Max frame rate +- Codec +- Bitrate (or bitrate mode: CBR/VBR) +- Stream URL to subscribe to + +### Camera Control (Command Safety) + +- PTZ type (fixed, motorized varifocal, mechanical PTZ, ePTZ) +- Pan range and speed +- Tilt range and speed +- Zoom type (optical, digital, none) and ratio +- Preset positions available +- IR control mode (auto, manual, off) +- Focus mode (auto, manual) + +## Schema Design Principles + +- **Timestamp every manifest**: include `discovered_at` so consumers know how fresh the data is. +- **Current state only**: no desired-state or drift detection. Consumers own comparison logic. +- **Null means not supported**: use `null` for capabilities the camera lacks (PTZ fields on fixed cameras). +- **Curated, not exhaustive**: include only fields that consumers act on. Expand when real consumers request more. +- **Discoverable from network**: every field must be derivable from API/ONVIF/RTSP probes (no manual spec entry required). + +## Scope Boundaries + +### In Scope + +- Discover cameras from IP + credentials (single or batch) +- Produce a YAML manifest per camera with ~20-25 curated fields +- Support multi-camera site manifests (batch discovery) +- Schema optimized for VMS, compliance, inference, and control consumers + +### Out of Scope (for now) + +- Pushing configuration to cameras +- Desired-state management or policy enforcement +- Drift detection logic (consumers own that) +- Manual spec entry for fields not discoverable from the network +- Camera firmware updates or management + +## Lifecycle + +The manifest is a **snapshot**: generated during onboarding, re-runnable on demand. It is not a continuously monitored document. Re-run the tool to get a fresh timestamp and updated state. + +## Input Shape + +Minimal seed: IP address(es) + credentials. Tool discovers everything else. + +```yaml +# Input example +cameras: + - ip: 192.168.1.221 + username: admin + password: "********" + - ip: 192.168.1.64 + username: admin + password: "********" +``` + +## Output Shape + +One manifest per camera (or combined site manifest). Fields sourced from live discovery. + +```yaml +# Output example (single camera) +discovered_at: "2026-05-18T20:33:55Z" +manufacturer: Reolink +model: RLC-520A +serial_number: "00000000524288" +mac_address: "ec:71:db:23:21:91" +firmware_version: v3.0.0.2417_23070501 +ip_address: 192.168.1.221 + +streams: + main: + url: "rtsp://admin:****@192.168.1.221:554/h264Preview_01_main" + codec: h264 + resolution: "2560x1920" + frame_rate: 30 + bitrate_kbps: 6144 + profile: High + sub: + url: "rtsp://admin:****@192.168.1.221:554/h264Preview_01_sub" + codec: h264 + resolution: "640x480" + frame_rate: 10 + bitrate_kbps: 256 + profile: High + +security: + https_enabled: true + onvif_enabled: true + onvif_port: 8000 + auth_method: token + +control: + ptz_type: fixed + pan_range: null + tilt_range: null + zoom_type: null + zoom_ratio: null + presets_available: null + ir_mode: Auto + focus_mode: null + +ai_detection: + person: true + vehicle: true + pet: true + face: false diff --git a/copilot/research/camera-discovery/coaching-state.md b/copilot/research/camera-discovery/coaching-state.md new file mode 100644 index 000000000..d98f2f0f7 --- /dev/null +++ b/copilot/research/camera-discovery/coaching-state.md @@ -0,0 +1,53 @@ +```yaml +project: + name: "Camera Discovery for Edge Provisioning" + slug: "camera-discovery" + created: "2026-05-18" + initial_request: "Review the camera discovery research and help me think through the work done / planned work from a design thinking coach perspective" + initial_classification: "" + +current: + method: 6 + space: "solution" + phase: "handoff-complete" + +methods_completed: [1, 2, 3, 4, 5, 6] + +transition_log: + - from_method: null + to_method: 1 + rationale: "Project initialized" + date: "2026-05-18" + - from_method: 1 + to_method: 6 + rationale: "Accelerated progression — existing implementation and research evidence used to compress Methods 2-6" + date: "2026-05-26" + - type: lateral + from_method: 6 + to: rpi-researcher + rationale: "Solution Space complete: handoff to RPI Researcher with validated onboarding agent concept" + date: "2026-05-26" + +hint_calibration: + level: 1 + pattern_notes: "User responds concisely, prefers direct progression" + +session_log: + - date: "2026-05-18" + method: 1 + summary: "Project initialization. Reviewing existing camera discovery agent research and live discovery test (Reolink RLC-520A) to assess scope through DT lens." + - date: "2026-05-26" + method: 2-6 + summary: "Accelerated progression through Methods 2-6 using existing implementation evidence. Discovered: single-vendor per site, workflow consistency is primary value, RTSP feed confirmation is first user action, manifest feeds real edge tools (508/510/515), no onboarding automation exists today. Converged on Camera Onboarding Agent concept for edge-ai repo." + +artifacts: + - path: ".copilot-tracking/dt/camera-discovery/01-scope-statement.md" + method: 1 + type: "scope-statement" + - path: ".copilot-tracking/dt/camera-discovery/01-architecture-concept.md" + method: 1 + type: "concept-sketch" + - path: ".copilot-tracking/dt/camera-discovery/handoff-solution-space.md" + method: 6 + type: "handoff-artifact" +``` diff --git a/copilot/research/camera-discovery/handoff-solution-space.md b/copilot/research/camera-discovery/handoff-solution-space.md new file mode 100644 index 000000000..ffde73c96 --- /dev/null +++ b/copilot/research/camera-discovery/handoff-solution-space.md @@ -0,0 +1,75 @@ +--- +exit_point: "concept-validated" +dt_method: 6 +dt_space: "solution" +handoff_target: "researcher" +date: "2026-05-26" +--- + + +# Solution Space Exit Handoff: Camera Discovery → Onboarding Agent + +## Artifacts + +| Path | Type | Confidence | +|---|---|---| +| .copilot-tracking/dt/camera-discovery/01-scope-statement.md | scope-statement | validated | +| .copilot-tracking/dt/camera-discovery/01-architecture-concept.md | concept-sketch | validated | +| .github/skills/experimental/camera-discovery/scripts/adapters/ | implementation | validated | +| .github/skills/experimental/camera-discovery/SKILL.md | documentation | validated | + +## Validated Concept: Camera Onboarding Agent + +**Location:** microsoft/edge-ai repository + +**Purpose:** Consume the camera discovery manifest (YAML) and generate app-specific configuration for each deployed 500-level edge application, eliminating manual camera onboarding. + +**Architecture:** Pluggable output generator pattern (mirrors the discovery adapter pattern): + +- One input contract: camera discovery manifest +- Multiple output generators: one per target 500-level app +- Target apps: 508-media-connector, 510-onvif-connector, 515-camera-dashboard (and others as needed) + +**User journey:** Run discovery (hve-core skill) → get manifest → onboarding agent generates configs → deploy to edge site + +## Constraints + +| Constraint | Source | Confidence | Category | Severity | +|---|---|---|---|---| +| Single vendor per site; multi-vendor across projects | User interview (DT coaching session) | validated | Workflow | Minor | +| Each 500-level app has its own config format (no shared standard) | User interview + GitHub repo analysis | validated | Environmental | Blocker | +| Azure IoT Operations is the platform layer | User interview | validated | Environmental | Minor | +| No existing onboarding automation | User interview | validated | Workflow | Blocker | +| Agent must live in edge-ai repo alongside apps it configures | User decision | validated | Environmental | Minor | +| Discovery manifest is the input contract (25-field YAML) | Implementation evidence | validated | Physical | Minor | + +## Assumptions + +| Assumption | Confidence | Status | Impact | +|---|---|---|---| +| Discovery manifest format is stable enough to build on | validated | validated | high | +| 500-level apps have documented/discoverable config schemas | assumed | untested | high | +| Pluggable output generator pattern scales to new apps | assumed | untested | medium | +| Engineers want full manifest (not partial/quick modes) | validated | validated | low | +| RTSP URL in manifest is sufficient for feed confirmation | validated | validated | low | +| Onboarding agent can read app README/env templates for schema discovery | assumed | untested | medium | + +## Validated Patterns + +| Pattern | Evidence | +|---|---| +| Engineers prioritize "connect and confirm feed" as first action | Direct user statement: "highest priority is to connect a camera at a known IP and check out its feed" | +| Consistent workflow across sites is the primary value of vendor abstraction | User confirmed portability across projects (not mixed-fleet) is the driver | +| Full 25-field discovery in one shot preferred over partial modes | User stated "better to have all the needed data from the full discovery" | +| Manifest serves as machine-readable contract for edge tools | User confirmed media connector and 500-level apps consume manifest data | +| Discovery manifest is step one of a multi-step onboarding journey | User stated "the discovery manifest is a first step to smooth camera onboarding" | + +## Technical Unknowns + +| Unknown | Priority | Source | +|---|---|---| +| Config format for each 500-level app (env vars, Helm values, MQTT topics) | high | Each app has its own format — needs per-app investigation | +| Azure IoT Operations asset definition schema for camera onboarding | high | Platform integration not yet mapped | +| Whether apps can hot-reload config or require restart after onboarding | medium | Deployment workflow unknown | +| How credentials flow from manifest to app configs securely | high | Security boundary between discovery and deployment | +| Whether a shared config standard across apps is feasible/desired | medium | Could simplify agent but requires cross-app coordination |