Merge new develop commits

.claude/commands/README.md

@@ -0,0 +1,49 @@
+# Claude Code slash commands for the mcp-server test suite
+
+Three AI-assisted workflows wrapping `mcp-server/run-tests.sh` and the meshtastic MCP tools. Each one has a twin in `.github/prompts/` for Copilot users.
+
+| Slash command         | What it does                                                              | Copilot equivalent                       |
+| --------------------- | ------------------------------------------------------------------------- | ---------------------------------------- |
+| `/test [args]`        | Runs the test suite (auto-detects hardware) and interprets failures       | `.github/prompts/mcp-test.prompt.md`     |
+| `/diagnose [role]`    | Read-only device health report via the meshtastic MCP tools               | `.github/prompts/mcp-diagnose.prompt.md` |
+| `/repro <test> [n=5]` | Re-runs one test N times, diffs firmware logs between passes and failures | `.github/prompts/mcp-repro.prompt.md`    |
+
+## Why two surfaces
+
+The Claude Code commands and Copilot prompts cover the same three workflows but each speaks its host's idiom:
+
+- **Claude Code** (`/test`) uses `$ARGUMENTS` for pass-through, has direct access to Bash + all MCP tools registered in the user's settings, and runs in the terminal context.
+- **Copilot** (`/mcp-test`) runs in VS Code's agent mode; it has terminal + MCP access too but typically asks the operator to confirm inputs interactively.
+
+A contributor using either IDE gets equivalent assistance. Keep the two in sync when behavior changes — the diff of intent should be minimal.
+
+## House rules
+
+- **No destructive writes without explicit operator approval.** Skills that could reflash, factory-reset, or reboot a device must describe the action and stop — the operator authorizes.
+- **Interpret failures, don't just echo them.** The skill body should pull firmware log lines from `mcp-server/tests/report.html` (the `Meshtastic debug` section, attached by `tests/conftest.py::pytest_runtest_makereport`) and classify the failure.
+- **Keep MCP tool calls sequential per port.** SerialInterface holds an exclusive port lock; two parallel tool calls on the same port deadlock.
+- **Never speculate about root cause.** If the evidence doesn't support a classification, say "unknown" and list what you'd need to disambiguate.
+
+## Adding a new command
+
+1. Write the Claude Code version at `.claude/commands/<name>.md` with YAML frontmatter:
+
+   ```yaml
+   ---
+   description: one-line purpose (used for auto-invocation by the model)
+   argument-hint: [optional-hint]
+   ---
+   ```
+
+2. Write the Copilot equivalent at `.github/prompts/mcp-<name>.prompt.md` with:
+
+   ```yaml
+   ---
+   mode: agent
+   description: ...
+   ---
+   ```
+
+3. Add the row to the table above. Cross-link in both bodies.
+
+4. Smoke-test on Claude Code first (`/<name>` should appear in autocomplete), then in VS Code Copilot (`/mcp-<name>` in Chat).

.claude/commands/diagnose.md

@@ -0,0 +1,68 @@
+---
+description: Produce a device health report using the meshtastic MCP tools (device_info, list_nodes, get_config, short serial log capture)
+argument-hint: [role=all|nrf52|esp32s3|<port>]
+---
+
+# `/diagnose` — device health report
+
+Call the meshtastic MCP tool bundle and format a structured health report for one or all detected devices. Zero guesswork for the operator.
+
+## What to do
+
+1. **Enumerate hardware.** Call `mcp__meshtastic__list_devices(include_unknown=True)`. For each entry where `likely_meshtastic=True`, capture `port`, `vid`, `pid`, `description`.
+
+2. **Filter by `$ARGUMENTS`**:
+   - No args, `all` → every likely-meshtastic device.
+   - `nrf52` → only devices with `vid == 0x239a`.
+   - `esp32s3` → only devices with `vid == 0x303a` or `vid == 0x10c4`.
+   - A `/dev/cu.*` path → only that one port.
+   - Anything else → treat as a substring match against the `port` string.
+
+3. **For each selected device, in sequence (NOT parallel — SerialInterface holds an exclusive port lock):**
+   - `mcp__meshtastic__device_info(port=<p>)` — captures `my_node_num`, `long_name`, `short_name`, `firmware_version`, `hw_model`, `region`, `num_nodes`, `primary_channel`.
+   - `mcp__meshtastic__list_nodes(port=<p>)` — count of peers, which ones have `publicKey` set, SNR/RSSI distribution.
+   - `mcp__meshtastic__get_config(section="lora", port=<p>)` — region, preset, channel_num, tx_power, hop_limit.
+   - Optionally, if the device seems unhappy (fails to connect, `num_nodes==1` when ≥2 are plugged in, missing firmware*version), open a short firmware log window: `mcp__meshtastic__serial_open(port=<p>, env=<inferred-env>)`, wait 3s, `serial_read(session_id=<s>, max_lines=100)`, `serial_close(session_id=<s>)`. The env should be inferred from the VID map in `mcp-server/run-tests.sh` (nrf52 → rak4631, esp32s3 → heltec-v3) unless `MESHTASTIC_MCP_ENV*<ROLE>` is set.
+
+4. **Hub health** (call once, not per-device): `mcp__meshtastic__uhubctl_list()` — enumerates every USB hub the host can see. Note which hubs advertise `ppps=true` and which hub hosts each Meshtastic device (cross-reference by VID). Flag it in the report if:
+   - No hub advertises PPPS → `tests/recovery/` can't run on this setup; hard-recovery via `uhubctl_cycle` isn't available.
+   - A Meshtastic device is on a non-PPPS hub → note it; operator may want to move the device to a PPPS hub to unlock auto-recovery.
+   - `uhubctl_list` raises `ConfigError: uhubctl not found` → just say `uhubctl not installed` in the report; don't treat as a fault.
+
+5. **Render per-device report** as:
+
+   ```text
+   [nrf52 @ /dev/cu.usbmodem1101]      fw=2.7.23.bce2825, hw=RAK4631
+     owner       : Meshtastic 40eb / 40eb
+     region/band : US, channel 88, LONG_FAST
+     tx_power    : 30 dBm, hop_limit=3
+     peers       : 1 (esp32s3 0x433c2428, pubkey ✓, SNR 6.0 / RSSI -24 dBm)
+     primary ch  : McpTest
+     hub         : 1-1.3 port 2 (PPPS, uhubctl-controllable)
+     firmware    : no panics in last 3s; NodeInfoModule emitted 2 broadcasts
+   ```
+
+   Keep it scannable. If a field is missing or abnormal (no pubkey for a known peer, region=UNSET, num_nodes inconsistent with the hub, device on non-PPPS hub), flag it inline with a short `⚠︎ <one-line reason>`.
+
+6. **Cross-device correlation** (only when >1 device is inspected):
+   - Do both sides see each other in `nodesByNum`? If one does and the other doesn't, that's asymmetric NodeInfo — flag it.
+   - Do the LoRa configs match? (region, channel_num, modem_preset should all agree; mismatch = no mesh)
+   - Do the primary channel NAMES match? Mismatch = different PSK = no decode.
+
+7. **Recorder slice (cheap, always available).** The mcp-server runs an autouse log recorder that's been collecting from every connected device. Pull two short slices to surface anything weird that's already happened:
+   - `mcp__meshtastic__logs_window(start="-2m", level="WARN|ERROR|CRIT", max_lines=20)` — recent firmware errors. If empty, say "no recent errors"; don't manufacture concern.
+   - `mcp__meshtastic__telemetry_timeline(window="1h", field="free_heap", max_points=60)` — heap trend. If `slope_per_min < -50`, flag it and recommend `/leakhunt window=6h` for a deeper read; otherwise just note the current free heap.
+   - If `recorder_status` shows `running:false` or `files.telemetry.last_ts` is null, note "recorder has no telemetry yet — enable `set_debug_log_api(True)` to populate" and skip this step gracefully.
+
+8. **Suggest next actions only for specific, recognisable failure modes**:
+   - Stale PKI pubkey one-way → "run `/test tests/mesh/test_direct_with_ack.py` — the retry + nodeinfo-ping heals this in the test path."
+   - Region mismatch → "re-bake one side via `./mcp-server/run-tests.sh --force-bake`."
+   - Device unreachable, reachable via DFU → `touch_1200bps(port=...)` + `pio_flash`. If not even DFU responds AND the device is on a PPPS hub, escalate to `uhubctl_cycle(role=..., confirm=True)`.
+   - CP2102-wedged-driver on macOS → see the note in `run-tests.sh`.
+   - Heap slope strongly negative → "run `/leakhunt window=6h` for a full timeline + classification."
+
+## What NOT to do
+
+- No writes. No `set_config`, no `reboot`, no `factory_reset`. This is a read-only diagnostic skill — if the operator wants to change state, they'll ask explicitly.
+- No `flash` / `erase_and_flash`. Those are separate escalations.
+- No holding SerialInterface across tool calls — open, query, close; next device. The port lock is exclusive.

.claude/commands/leakhunt.md

@@ -0,0 +1,103 @@
+---
+description: Hunt for memory leaks (and other slow degradations) by reading the persistent recorder's heap timeline + log slice over a window
+argument-hint: [window=1h] [field=free_heap] [variant=local]
+---
+
+<!-- markdownlint-disable MD029 -->
+
+# `/leakhunt` — read the recorder, classify a memory leak
+
+Use the always-on recorder (`mcp-server/.mtlog/`) to read a heap timeline plus the matching log slice and produce a one-page verdict: **steady / slow leak / fragmentation / OOM-imminent**. No firmware changes, no special build flags — the LocalStats telemetry packet that the firmware already broadcasts every ~60 s carries `heap_free_bytes` and `heap_total_bytes`.
+
+## Two signal paths — pick the right one
+
+| Path                  | Build flag       | Cadence        | Per-thread attribution | Cost                      |
+| --------------------- | ---------------- | -------------- | ---------------------- | ------------------------- |
+| LocalStats packet     | (default)        | ~60 s          | No                     | Free — always on          |
+| `[heap N]` log prefix | `-DDEBUG_HEAP=1` | every log line | Yes (Thread X leaked)  | Bigger flash + log volume |
+
+Both feed the same `telemetry_timeline(field="free_heap")` query — when DEBUG_HEAP is on, the recorder synthesizes telemetry rows from log prefixes (tagged `source: debug_heap`), so a single timeline call gets whichever signal is available. **For a slow leak diagnosis, the default path is plenty** (60 s cadence over 6 h = 360 points; linear regression over that nails sub-100-byte/min slopes). **DEBUG_HEAP is for attribution** — when the slope is real and you need to know which thread is leaking.
+
+## What to do
+
+1. **Parse `$ARGUMENTS`**: optional `window` (default `1h`, accepts `30m`/`6h`/`-3d`/etc.), optional `field` (default `free_heap`; alternates: `total_heap`, `battery_level`, anything in the LocalStats variant), optional `variant` (default `local`; alternates: `device`, `environment`, `power`, `airQuality`, `health`).
+
+2. **Verify the recorder is alive** — call `mcp__meshtastic__recorder_status`. Check:
+   - `running == True`
+   - `files.telemetry.lines > 0` (at least one telemetry packet recorded — if zero, the device hasn't broadcast LocalStats yet OR `set_debug_log_api` has never been on; tell the operator to run `mcp__meshtastic__set_debug_log_api(enabled=True)` and wait one device-update interval)
+   - `files.telemetry.last_ts` within the last 5 minutes (if older, the device is silent — log that, not "leak detected")
+
+3. **Detect whether DEBUG_HEAP is active** — `mcp__meshtastic__logs_window(start="-2m", grep=r"\\[heap \\d+\\]", max_lines=3)`. If any line matches, the firmware has the prefix → DEBUG_HEAP is on, expect higher-cadence data and `heap_event` rows. If zero matches over the last 2 minutes, you're on the LocalStats-only path.
+
+4. **Pull the timeline** — `mcp__meshtastic__telemetry_timeline(window=$window, variant=$variant, field=$field, max_points=200)`. Read:
+   - `samples` — how many raw points contributed
+   - `min`, `max` — total swing
+   - `slope_per_min` — units per minute (linear regression over the whole window)
+
+5. **Pull the log context for the same window** — `mcp__meshtastic__logs_window(start="-${window}", grep="Heap status|leaked heap|freed heap|out of memory|Alloc an err|panic|abort", max_lines=200)`. These are the strings the firmware emits when something memory-related happens (`DEBUG_HEAP` builds emit `"Heap status:"` and `"leaked heap"` lines; production builds emit `"Alloc an err"` on failure and `"out of memory"` on OOM).
+
+6. **Pull marker events** so we know if the operator labeled phases — `mcp__meshtastic__events_window(start="-${window}", kind="mark|connection_lost|connection_established")`. If a `connection_lost` overlaps a sharp drop, that's not a leak; that's a reboot.
+
+6a. **(DEBUG_HEAP only) Per-thread attribution** — `mcp__meshtastic__logs_window(start="-${window}", grep="leaked heap", max_lines=200)`. Each row has a structured `heap_event` field with `{kind, thread, before, after, delta}`. Aggregate by thread: sum the `delta` over the window per thread name. The thread with the largest cumulative negative delta is your suspect. Note the count too — a thread with 50× small leaks is different from 1× big leak.
+
+7. **Classify** based on what the data says, NOT on what you wish it said. Use these rules in order:
+   - **Insufficient data** (< 5 samples): say so. Suggest a longer window or longer wait. Stop.
+   - **Reboot mid-window**: if any `connection_lost` event is present AND `free_heap` jumped UP at that timestamp, the device rebooted. Note it; pre-reboot trend may be a leak but you only have part of the curve.
+   - **OOM-imminent**: any `Alloc an err=` or `out of memory` line in the log slice. This trumps everything; flag urgently.
+   - **Slow leak**: `slope_per_min < -50` AND `max - min > 1000` AND no reboot. The heap is monotonically (or near-monotonically) declining. Estimate time-to-zero: `min / -slope_per_min` minutes. Surface it.
+   - **Fragmentation suspect**: `slope_per_min` close to zero (|x| < 50) BUT min trends down across the window AND the log slice shows `Alloc an err` warnings WITHOUT total OOM. Means free total is OK but largest contiguous block is shrinking. Recommend a `DEBUG_HEAP` build to confirm.
+   - **Steady**: |slope_per_min| < 50, no error lines. Heap is fine.
+   - **Recovery curve**: slope is POSITIVE — heap recovered. Either a workload completed or GC fired. Note it; not a leak.
+
+8. **Report**:
+
+   ```text
+   /leakhunt window=6h field=free_heap variant=local
+   ────────────────────────────────────────────────────
+   recorder        : running, telem last_ts 8s ago
+   build           : DEBUG_HEAP=ON (per-line prefix detected)
+   samples         : 14,200 over 6h (cadence ~1.5s, log-line synth)
+   free_heap       : min 92,344 / max 124,008 / range 31,664
+   slope           : -82 bytes/min (negative — heap declining)
+   reboots         : none in window
+   OOM events      : none
+   error lines     : 3× "Alloc an err=ESP_ERR_NO_MEM" at +4h12m, +5h08m, +5h44m
+   thread leaks    : (DEBUG_HEAP) MeshPacket -3,124 B over 18 events
+                                  Router       -1,408 B over 4 events
+                                  others       -240 B
+   verdict         : SLOW LEAK — primary suspect MeshPacket thread
+   est. time-to-OOM: ~1,127 min (~18.8 h) at current slope
+   evidence        : (3 log line citations with uptimes)
+   ```
+
+   Then: **what to do next.**
+   - SLOW LEAK, **DEBUG_HEAP off** → recommend rebuilding with the flag and re-running this skill. Concrete one-liner the operator can copy:
+     ```text
+     mcp__meshtastic__build(env="<env>", build_flags={"DEBUG_HEAP": 1})
+     mcp__meshtastic__pio_flash(env="<env>", port="<port>", confirm=True)
+     ```
+     After flash, set debug_log_api back on and wait one window; re-run `/leakhunt`.
+   - SLOW LEAK, **DEBUG_HEAP on** → cite the top-leaking thread name from step 6a. Point at the corresponding source file (`grep -rn "ThreadName(\"<name>\")" src/`); the operator decides what to fix.
+   - FRAGMENTATION SUSPECT → propose pre-allocating any per-packet buffers; or rebuilding with `CONFIG_HEAP_TASK_TRACKING=y` on ESP32 to see who's holding the largest blocks.
+   - OOM-IMMINENT → flag for immediate attention; don't wait for the next telemetry interval.
+   - STEADY → say so; stop. Don't invent problems.
+
+## What NOT to do
+
+- Don't assume a leak from a single dip. LocalStats fires every ~60 s and the firmware naturally allocates+frees on each broadcast cycle; one packet sees the trough. Look at the slope, not the deltas.
+- Don't recommend code changes. This skill diagnoses; the operator decides what to fix.
+- Don't enable `set_debug_log_api` automatically — if it's off, telemetry isn't reaching pubsub anyway, and the recorder will be empty. Tell the operator to flip it on and wait, then re-run.
+- Don't run heavy workloads to "trigger the leak." The recorder is passive; we read what's there.
+
+## Companion: `mark_event` for stress runs
+
+If the operator wants to test under stimulus (e.g. blast 50 broadcasts and see what the heap does), they can frame the experiment with markers:
+
+```text
+mark_event("burst-start")
+… run the workload …
+mark_event("burst-end")
+/leakhunt window=15m
+```
+
+The markers land in both `events.jsonl` and `logs.jsonl`, so the report can show "free_heap dipped 8 KB during the burst window, recovered to baseline within 2 LocalStats cycles" → not a leak.