Add #5: bulk upload_schedule service + canonical schedule hash (RFC-005)#104
Open
eman wants to merge 11 commits into
Open
Add #5: bulk upload_schedule service + canonical schedule hash (RFC-005)#104eman wants to merge 11 commits into
eman wants to merge 11 commits into
Conversation
Implements the firmware side of RFC-005 (dhw-sensor-apps#5), built on the #92 write-operation layer: - New pure schedule_codec (payload parse/validate, 42-byte layer image builder, canonical FNV-1a-64 hash) with 29 host tests; golden hash vectors are shared verbatim with the dhw-sensor-apps scheduler suite as a cross-language contract. - New upload_schedule service: full-state clear-and-set upload of the entire 7x5 grid in one call. Per layer: mandatory fresh read, skip when the image already matches (a no-change re-upload costs zero BLE writes), whole-layer write + commit for the rest, settle, readback confirm. New PARTIAL terminal status; settled event gains layers_written / layers_skipped / schedule_hash. 150 s watchdog. Uploads supersede queued grid ops and vice versa. - New schedule_hash text sensor (diagnostic): canonical hash of the cached grid + enabled flag, republished after every settled schedule op; "unknown" until fully cached. External schedulers poll it to verify sync without a read-back. - ScheduleService: public write_layer_image_async + cached-image/hash helpers. write_bench.py gains an upload subcommand with local expected-hash computation. - 30 new write-op simulator assertions (happy path, skip-identical, partial, supersede, disconnect, enabled flag); all 156 pass alongside the existing suites. - Fix host build under g++ 14: control_service.h now includes <cmath> (NAN macro ordering). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The mock ESP_LOG macros discarded their arguments, so every TAG, log-only lambda capture, and log-only parameter warned as unused in the host build (~200 warnings; the firmware build was already clean). The mock loggers now route arguments into an unevaluated sink so log-only values count as used, the mock component/helpers headers unname their parameters, and helpers.h gains the format_hex_pretty stub the sink made name-checked. The remaining dozen genuinely-unused captures and callback parameters are unnamed at their sites, one size_t/int sign-compare in get_mode_async is cast like its neighbors, and WriteOperationService drops its never-used session_ reference. Host suite: 372 tests, zero warnings. Firmware build: zero warnings.
The workflow only triggered on pushes to main and PRs targeting main, so branches without an open PR got no checks. Trigger on all branch pushes (plus manual dispatch), keep the PR trigger, and cancel superseded runs per ref so a push to a PR branch doesn't queue duplicates.
find_free_single_event_slot() treated every cached slot as occupied, so expired one-shot events accumulated until the 35-slot pool exhausted (the RFC-005 nowcast path writes one event per pre-heat burst). The resolver now takes a reusable_before_ts and treats disabled events and events that ended before it as free — the new event's begin timestamp serves as "now", and writing over an expired slot both reuses and clears it. Default 0 preserves the old behavior for other callers. Updated the auto-slot test to seed a live (future-end) event and added a reuse case: an enabled-but-expired slot 0 is picked and overwritten. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The aggregate Weekly Schedule JSON truncates at HA's 255-char state cap (a real 20-entry grid already loses layer 4), so an external scheduler recovering from state loss could not learn the full grid before performing a destructive full-state upload — the bench run had to brute-force a hidden entry against the canonical hash. Adds five optional schedule_layer_0..4 text sensors, each publishing its layer as 7 compact cells ([start_min,end_min] or 0, ~86 chars worst case, always under the cap) via the pure codec::layer_image_to_json. Republished alongside schedule_hash after every settled schedule op; "unknown" until the layer is cached. Together with the hash sensor this gives a race-free, poll-only cold-start recovery path sized to the data. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…o feature-issue-5-bulk-upload
Adapts the bulk-upload additions to the review changes: upload payload grammar errors settle invalid via the bridge, submit_upload_schedule gains the origin parameter, and the WriteResult carries both the upload summary fields and the requested_* echoes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Firmware side of [RFC-005] Based on
feature-issue-92's write-operation layer.What
schedule_codec(new, pure, host-testable): v1 bulk-payload parser/validator, 42-byte layer-image builder, and the canonical FNV-1a-64 schedule hash. 29 host tests; the golden hash vectors are shared verbatim with the Python scheduler suite — a cross-language contract test.upload_scheduleservice (data,op_id): full-state clear-and-set upload of the entire 7×5 grid in one HA call, replacing the 35-clear + N-set sequence. Per layer: mandatory fresh read → skip when the image already matches (a no-change re-upload costs zero BLE writes) → whole-layer write + commit → settle → readback confirm. Newpartialterminal status; the settled event gainslayers_written,layers_skipped,schedule_hash. 150 s watchdog. Uploads supersede queued grid ops and vice versa (same resource keys).schedule_hashtext sensor (diagnostic): canonical hash of the cached grid + enabled flag, republished after every settled schedule op;unknownuntil all five layers + overview are cached. The external scheduler polls this to verify sync without a read-back (the JSON sensor truncates at HA's 255-char cap; the hash is 19 chars).ScheduleService: publicwrite_layer_image_async(same proven path asset_entry_async) + cached-image/hash helpers.tools/write_bench.py: newuploadsubcommand that builds the payload from CLI entry specs and prints the locally computed expected hash for comparison against the settled event.control_service.hnow includes<cmath>(NAN macro ordering broke the test suite under g++ 14).ESP_LOG*macros now route arguments into an unevaluated sink so log-only values count as used, the dozen genuinely unused captures/params are unnamed at their sites, andWriteOperationServicedrops a never-usedsession_member. Host and firmware builds are both at zero warnings.mainand PRs targetingmain, so this branch had no checks until a PR existed. It now runs on pushes to every branch, with per-ref cancellation of superseded runs.Payload grammar (v1)
Full-state semantics: absent cells are cleared; duplicate cells, range violations,
start >= end, >35 entries, and unknown versions are rejected pre-wire with a terminalrejectedevent. CSV keeps the existing service idiom and the ESP32-C3 string-args workaround; ~540 bytes worst case.Tests
schedule-v155forecast) parses here and hashes to the identical value the Python side computes (v1:d6700dc4d90b5c43)Hardware verification (ESP32-C3 + real pump)
OTA-deployed and bench-driven with
write_bench.py; every call settled with exactly one terminal event.accepted, detailno-op, all 5 layers skipped, zero BLE writes, hash matched the locally computed valueaccepted,layers_written: 4only, settled hash matched predictionaccepted, layer 4 rewritten (clear-and-set removed the added entry), hash back to baselineThe device hash also survived an adversarial check: the one grid entry hidden by the 255-char JSON truncation was recovered by brute-forcing candidates against
schedule_hash, and the reconstructed grid verified bit-for-bit. That recovery only worked because a single entry was hidden; the general cold-start read-back gap is filed as dhw-sensor-apps#7.RFC-005 Phase 3 cross-validation (scheduler v2 against this firmware, live)
Ran the
feature/rfc-005-scheduler-v2daemon in-process against real HA, real InfluxDB forecast curves, and this firmware: capability probe, weekly replan (27-entry bulk upload, hash convergence,in_sync: true), daily refinement plus trigger dedupe, forced resync (firmware no-op), nowcast single event (written, deduped, cleared), away/home inhibit round trip. Two scheduler-side defects were found and fixed on that branch (hash entity id is area/name-derived in HA, not node-derived; HA requires every declared service field). Firmware behaved to contract throughout.Deployment note
HA's ESPHome integration marks every declared service field required and returns HTTP 400 for calls that omit one. Because this PR's services all declare
op_id, existing HA REST callers that send onlydata(the current production scheduler, any user automations) will start failing with 400 the moment this firmware deploys. Callers must sendop_id(empty string is fine). Deploy scheduler v2 together with this firmware, or addop_id: ""to existing automations first. Native-API callers are unaffected.Not done here: version bump/release (left to the release flow — CHANGELOG entry added under Unreleased).