V3 Map Service: Worker + device fetch/render + app rebuild button - #1
Merged
Conversation
Scopes phase 1 of the parked v3 map-service spec to a self-contained Cloudflare Worker: ports build_region_map.py's OSM pipeline to TS and defines the binary wire format the device will later parse. Firmware and app changes are deferred to later phases per the working agreement of one reviewed diff per layer.
Ports scripts/build_region_map.py's Overpass fetch -> stitch -> filter -> Douglas-Peucker -> quantize pipeline to a Cloudflare Worker (TypeScript) serving GET /map?lat&lon&radius as a binary blob per the wire format in docs/superpowers/plans/2026-07-05-v3-map-service-phase1.md. Includes Workers Cache API caching, per-IP rate limiting, and 34 unit/integration tests with mocked Overpass responses (no live network calls). Independent review (code-review skill, high effort) surfaced and fixed: - Rate limiter's overflow safety valve wiped all tracked IPs instead of sweeping expired ones, resetting counters exactly under high load. - Town labels were mangled for non-ASCII names (byte-truncated UTF-16 code units instead of UTF-8 encoding). - Layer fetches ran concurrently instead of matching the Python original's serialized+2s-delay Overpass politeness pattern (CLAUDE.md landmine MatixYo#7); now serialized with an injectable delay, plus an in-isolate single-flight guard so concurrent requests for the same location share one pipeline run. - BudgetExceededError (a caller-controllable outcome) returned 502 instead of 400. Deployment, firmware LittleFS integration, and the app's rebuild-map button are deferred to later phases per the working agreement of one reviewed diff per layer (see the phase-1 plan doc).
Records the device-side streaming-read architecture decision (no full blob in RAM given the ESP32-C3's ~70KB free heap vs. the wire format's 96KB budget) before implementation starts.
Wires the device up to the Phase 1 Cloudflare Worker (PR #1, worker/): - include/ui/region_map_blob.h: native-testable, Arduino-free wire-format decoder (mirrors worker/src/encode.ts field-by-field). - include/ui/region_map_source.h + src/ui/region_map_source.cpp: indirection layer between the renderer and its data source. Prefers /map.bin on LittleFS when present and valid, falls back to the compile-time-baked arrays otherwise. Never loads a fetched blob fully into RAM (the ESP32-C3's ~70KB free heap can't hold the wire format's 96KB budget) -- accessors seek+read only the records the renderer asks for. - src/ui/region_map.cpp: drawRegionMap()/drawSpan() rewired through the new source accessors (64-vert chunked reads) instead of the raw baked externs. - src/services/map_service.{h,cpp}: HTTPS fetch mirroring adsb_client.cpp's poll-during-blocking-I/O idiom; streams the response straight to /map.bin.tmp, validates the header/size, then atomically renames into place. Worker URL is NVS-backed (services::map::saveServiceUrl), empty by default -- this fork doesn't bake in a specific Cloudflare account's URL. - src/services/web_app.cpp: POST /api/map/rebuild; /api/aircraft gains mapSource ("baked"/"fetched"/"hidden") and mapServiceUrl fields; /api/settings accepts a mapServiceUrl field. - webapp/index.html: map service URL input, "rebuild map for my location" button with inline status, and a map-source readout in the drawer. - test/native/test_map_blob: 7 tests for the wire-format decoder. Verification note: this sandbox's egress policy blocks PlatformIO's package registry (confirmed via the proxy status endpoint), so `pio test`/`pio run` themselves can't run here. Verified instead via a hand-built harness that compiles/links test/native/* exactly like [env:native] would (all 54 tests pass, including the new ones) plus manual review/syntax-checking of the Arduino-bound files against stub headers matching the real APIs. Two bugs found in review and fixed before this commit: rebuildForLocation() was deleting the old /map.bin before attempting the rename (a rename failure would've left no map at all instead of the old one), and saveServiceUrl() was dead code with no caller (the rebuild button could never have worked end-to-end without a firmware rebuild to hardcode a URL). Real hardware was never touched -- no `-t upload`, per the working agreement of always warning before flashing Selma's physical device.
selmapi
changed the title
…r the live 2026-07-06 deploy) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A cold Worker request can take 37s-3min, far past the device's 15s HTTP timeout, so first-time rebuilds always failed. The Worker now returns 202 immediately on a cache miss and builds in the background; add RebuildResult::kBuilding for that case so the caller can retry later instead of treating it as a network error. Timeout stays at 15s -- 202 arrives right away, and cache hits measure ~1.3s on-device. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
kBuilding now surfaces as {"ok":false,"building":true,"error":"..."} so
the phone app can distinguish "still building, retry" from a hard
failure. Other results unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
When /api/map/rebuild responds with building:true, show inline status
("building map on server... attempt N of 8") and automatically re-POST
every 45s, up to 8 attempts. Stops early on ok:true (success + map-source
refresh) or on a non-building error. Vanilla JS, matches the existing
terminal-style status conventions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- include/config.h: kDefaultMapServiceUrl now points at the owner's deployed Worker (opened for public use by this fork), instead of shipping empty/disabled by default. - scripts/build_webapp.py: pin the gzip header's OS byte to 0xFF. Python 3.14 changed its default from 0x03 to 0xFF, which alone made the committed webapp_gz.h non-byte-stable across Python versions (landmine MatixYo#6). Regenerated the header (also picks up the new auto-poll JS); verified byte-identical across two consecutive regenerations. - README.md: document the map service (public URL, ~1-2min first build with automatic retry, or self-host from worker/). - CLAUDE.md: note the OS-byte pin under landmine MatixYo#6. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Cache-miss requests no longer hold the connection for the 37s+ (sometimes
multi-minute, when Overpass throttles Cloudflare egress IPs) pipeline the
device's 15s HTTP timeout can never survive. Instead a miss starts (or
joins) the single-flight background build, registers the full
pipeline->cache.put chain via ctx.waitUntil, and immediately returns
202 {"status":"building","retryAfterSeconds":45} with Retry-After: 45.
Cache hits still return the 200 blob unchanged; 400/404/405/429 behavior
is unchanged. Pipeline failures clear the in-flight entry so the next
poll retries fresh -- deliberately no persistent failure state (no KV/DO).
Documented honestly (code comment + README): Cloudflare docs say waitUntil
extends an invocation at most 30s past the response -- on paper too short
for this pipeline -- but observed live behavior (2026-07-05) is that
disconnected-client builds run to completion; each 45s poll also re-arms
the extension on a fresh ExecutionContext as a hedge.
wrangler.toml: [observability] enabled = false -- the service is being
opened to other users of this fork and request URLs contain their home
coordinates; do not retain request logs.
Tests (plain vitest, injected deps preserved): 202-then-200 flow via a
captured waitUntil promise, in-flight joiner dedup, failure-clears-flight
retry, plus a test-only resetInFlightState() to isolate module state.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rvice-radar-wqa0ip
selmapi
marked this pull request as ready for review
July 6, 2026 04:46
Owner
Author
|
Independent review (Opus): APPROVE — no findings above Low. Verified: wire format encoder/decoder field-by-field parity, LittleFS tmp/rename lifecycle, 202-vs-15s-timeout path, worker single-flight lifecycle (no leaks/races), webapp poll loop (no timer stacking), landmines #1/MatixYo#5/MatixYo#6/MatixYo#8/MatixYo#10 compliance, Overpass UA+politeness preserved. Suites: worker 35/35 + tsc clean, native 54/54, supermini build SUCCESS with clean tree after regeneration. 🤖 Generated with Claude Code |
rebuildForLocation() streams a new map to /map.bin.tmp and renames it over /map.bin, but region_map_source.cpp keeps /map.bin open for the process lifetime so the renderer can seek/read records on demand. LittleFS can't rename over (or reliably remove) an open file, so every rebuild with an existing map failed with kWriteError even after a successful fetch. Add mapSourceRelease() to close the handle and fall back to the baked arrays, called as late as possible (right before the rename block) so the old map stays drawable through the whole streaming phase. Both the success and the still-failing-after-retry kWriteError paths now call mapSourceInit() so the source is always re-attached rather than left detached until reboot. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Full V3 "Map Service" feature (see
docs/superpowers/specs/2026-07-05-v3-map-service-design.md), landed as three reviewed layers in one branch:Phase 1 —
worker/: Cloudflare Worker OSM pipeline portGET /map?lat&lon&radiusreturns a binary blob (magic/version/center/counts + verts/spans/towns) perdocs/superpowers/plans/2026-07-05-v3-map-service-phase1.md.scripts/build_region_map.py: Overpass fetch, way stitching, water-noise filters, Douglas-Peucker with the same budget ladder, int16 quantization, interstates-only owner decision.Phase 2 — device: fetch, store, render
include/ui/region_map_blob.h: native-testable, Arduino-free wire-format decoder mirroringworker/src/encode.ts.include/ui/region_map_source.h/.cpp: indirection layer between the renderer and its data — prefers/map.binon LittleFS when present/valid, falls back to the compile-time-baked arrays otherwise. Streams records from flash rather than loading a fetched blob into RAM (the ESP32-C3's ~70KB free heap can't hold the wire format's 96KB budget).src/ui/region_map.cpprewired through the new source accessors (chunked reads) instead of the raw baked externs.src/services/map_service.h/.cpp: HTTPS fetch mirroringadsb_client.cpp's poll-during-blocking-I/O idiom; streams to/map.bin.tmp, validates, atomically renames into place. Worker URL is NVS-backed, empty by default (no hardcoded Cloudflare account).Phase 3 — app: rebuild button
POST /api/map/rebuild;/api/aircraftgainsmapSource/mapServiceUrl;/api/settingsacceptsmapServiceUrl.webapp/index.html: map-service-URL input, "rebuild map for my location" button with inline status, map-source readout.An independent review of Phase 2+3 found and fixed two real bugs before this push:
rebuildForLocation()was deleting the old/map.binbefore attempting the rename (a rename failure would've left no map at all instead of the old one), andsaveServiceUrl()was dead code with no caller anywhere — the rebuild button could never have succeeded end-to-end without a firmware rebuild to hardcode a URL. Both are fixed.Verification
cd worker && npm test(34/34) +npx tsc --noEmit— run directly, both clean.403onapi.registry.platformio.org), sopio test -e native/pio run -e superminicannot run here. Verified instead via a hand-built harness that compiles/linkstest/native/*exactly like[env:native]would — 54/54 tests pass, independently re-run during review, not just trusted from the implementation pass. The webapp's generatedinclude/web/webapp_gz.hwas independently regenerated (bypassing the blocked PlatformIO pre-script by running the same gzip logic directly) and confirmed byte-identical. Arduino-bound files (map_service.cpp,web_app.cpp,region_map_source.cpp) were syntax-checked against hand-written stub headers matching the real ESP32 Arduino APIs where feasible; full LovyanGFX/WiFiManager stubbing was out of scope for the time budget, so those two files' edits (small, pattern-matching existing adjacent code) were verified by manual line-by-line review instead of a compiler.-t upload, per the working agreement of always warning Selma before flashing her physical device.Explicitly deferred
worker/README.mddocumentswrangler deployfor whoever hosts it)./api/settingspath is wired up).Test plan
cd worker && npm test— 34/34 passcd worker && npx tsc --noEmit— cleanpio testcan't reach its registry here) — 54/54 passpio test -e native/pio run -e superminiin an environment with PlatformIO registry access, to confirm the sandbox workaround didn't miss anything