Skip to content

Add #5: bulk upload_schedule service + canonical schedule hash (RFC-005)#104

Open
eman wants to merge 11 commits into
feature-issue-92from
feature-issue-5-bulk-upload
Open

Add #5: bulk upload_schedule service + canonical schedule hash (RFC-005)#104
eman wants to merge 11 commits into
feature-issue-92from
feature-issue-5-bulk-upload

Conversation

@eman

@eman eman commented Jul 19, 2026

Copy link
Copy Markdown
Owner

Firmware side of [RFC-005] Based on feature-issue-92's write-operation layer.

What

  • schedule_codec (new, pure, host-testable): v1 bulk-payload parser/validator, 42-byte layer-image builder, and the canonical FNV-1a-64 schedule hash. 29 host tests; the golden hash vectors are shared verbatim with the Python scheduler suite — a cross-language contract test.
  • upload_schedule service (data, op_id): full-state clear-and-set upload of the entire 7×5 grid in one HA call, replacing the 35-clear + N-set sequence. Per layer: mandatory fresh read → skip when the image already matches (a no-change re-upload costs zero BLE writes) → whole-layer write + commit → settle → readback confirm. New partial terminal status; the settled event gains layers_written, layers_skipped, schedule_hash. 150 s watchdog. Uploads supersede queued grid ops and vice versa (same resource keys).
  • schedule_hash text sensor (diagnostic): canonical hash of the cached grid + enabled flag, republished after every settled schedule op; unknown until all five layers + overview are cached. The external scheduler polls this to verify sync without a read-back (the JSON sensor truncates at HA's 255-char cap; the hash is 19 chars).
  • ScheduleService: public write_layer_image_async (same proven path as set_entry_async) + cached-image/hash helpers.
  • tools/write_bench.py: new upload subcommand that builds the payload from CLI entry specs and prints the locally computed expected hash for comparison against the settled event.
  • Host-build fix: control_service.h now includes <cmath> (NAN macro ordering broke the test suite under g++ 14).
  • Host-build warnings eliminated (follow-up commit): the mock ESP_LOG* macros now route arguments into an unevaluated sink so log-only values count as used, the dozen genuinely unused captures/params are unnamed at their sites, and WriteOperationService drops a never-used session_ member. Host and firmware builds are both at zero warnings.
  • CI trigger fix (follow-up commit): the workflow only ran on pushes to main and PRs targeting main, so this branch had no checks until a PR existed. It now runs on pushes to every branch, with per-ref cancellation of superseded runs.

Payload grammar (v1)

data   := "v1," enabled ( ";" entry )*      enabled := "0" | "1" | "-"
entry  := layer "," day "," sh "," sm "," eh "," em

Full-state semantics: absent cells are cleared; duplicate cells, range violations, start >= end, >35 entries, and unknown versions are rejected pre-wire with a terminal rejected event. CSV keeps the existing service idiom and the ESP32-C3 string-args workaround; ~540 bytes worst case.

Tests

  • 29 codec host tests (grammar accept/reject table, layer images, canonicalization, golden vectors)
  • 30 new write-op simulator assertions: happy path, skip-identical (0 write frames on re-upload), partial (failed layer + skipped layers), upload-supersedes-upload, upload-supersedes-queued-entry-op, disconnect mid-op (exactly one terminal event), enabled-flag application
  • Full suite: 372 tests across all suites pass, zero warnings; no regressions in the existing 126 write-op cases. CI green on the Linux GCC runner.
  • Cross-checked against live data: the scheduler's real 28-entry payload (from the deployed schedule-v155 forecast) parses here and hashes to the identical value the Python side computes (v1:d6700dc4d90b5c43)

Hardware verification (ESP32-C3 + real pump)

OTA-deployed and bench-driven with write_bench.py; every call settled with exactly one terminal event.

Test Result Settle
Upload of the pump's identical 20-entry grid accepted, detail no-op, all 5 layers skipped, zero BLE writes, hash matched the locally computed value 2.2 s
Upload with one added entry accepted, layers_written: 4 only, settled hash matched prediction 5.2 s
Re-upload of the original grid accepted, layer 4 rewritten (clear-and-set removed the added entry), hash back to baseline 5.4 s

The device hash also survived an adversarial check: the one grid entry hidden by the 255-char JSON truncation was recovered by brute-forcing candidates against schedule_hash, and the reconstructed grid verified bit-for-bit. That recovery only worked because a single entry was hidden; the general cold-start read-back gap is filed as dhw-sensor-apps#7.

RFC-005 Phase 3 cross-validation (scheduler v2 against this firmware, live)

Ran the feature/rfc-005-scheduler-v2 daemon in-process against real HA, real InfluxDB forecast curves, and this firmware: capability probe, weekly replan (27-entry bulk upload, hash convergence, in_sync: true), daily refinement plus trigger dedupe, forced resync (firmware no-op), nowcast single event (written, deduped, cleared), away/home inhibit round trip. Two scheduler-side defects were found and fixed on that branch (hash entity id is area/name-derived in HA, not node-derived; HA requires every declared service field). Firmware behaved to contract throughout.

Deployment note

HA's ESPHome integration marks every declared service field required and returns HTTP 400 for calls that omit one. Because this PR's services all declare op_id, existing HA REST callers that send only data (the current production scheduler, any user automations) will start failing with 400 the moment this firmware deploys. Callers must send op_id (empty string is fine). Deploy scheduler v2 together with this firmware, or add op_id: "" to existing automations first. Native-API callers are unaffected.

Not done here: version bump/release (left to the release flow — CHANGELOG entry added under Unreleased).

eman and others added 11 commits July 18, 2026 20:41
Implements the firmware side of RFC-005 (dhw-sensor-apps#5), built on
the #92 write-operation layer:

- New pure schedule_codec (payload parse/validate, 42-byte layer image
  builder, canonical FNV-1a-64 hash) with 29 host tests; golden hash
  vectors are shared verbatim with the dhw-sensor-apps scheduler suite
  as a cross-language contract.
- New upload_schedule service: full-state clear-and-set upload of the
  entire 7x5 grid in one call. Per layer: mandatory fresh read, skip
  when the image already matches (a no-change re-upload costs zero BLE
  writes), whole-layer write + commit for the rest, settle, readback
  confirm. New PARTIAL terminal status; settled event gains
  layers_written / layers_skipped / schedule_hash. 150 s watchdog.
  Uploads supersede queued grid ops and vice versa.
- New schedule_hash text sensor (diagnostic): canonical hash of the
  cached grid + enabled flag, republished after every settled schedule
  op; "unknown" until fully cached. External schedulers poll it to
  verify sync without a read-back.
- ScheduleService: public write_layer_image_async + cached-image/hash
  helpers. write_bench.py gains an upload subcommand with local
  expected-hash computation.
- 30 new write-op simulator assertions (happy path, skip-identical,
  partial, supersede, disconnect, enabled flag); all 156 pass alongside
  the existing suites.
- Fix host build under g++ 14: control_service.h now includes <cmath>
  (NAN macro ordering).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The mock ESP_LOG macros discarded their arguments, so every TAG,
log-only lambda capture, and log-only parameter warned as unused in the
host build (~200 warnings; the firmware build was already clean). The
mock loggers now route arguments into an unevaluated sink so log-only
values count as used, the mock component/helpers headers unname their
parameters, and helpers.h gains the format_hex_pretty stub the sink
made name-checked. The remaining dozen genuinely-unused captures and
callback parameters are unnamed at their sites, one size_t/int
sign-compare in get_mode_async is cast like its neighbors, and
WriteOperationService drops its never-used session_ reference.

Host suite: 372 tests, zero warnings. Firmware build: zero warnings.
The workflow only triggered on pushes to main and PRs targeting main, so
branches without an open PR got no checks. Trigger on all branch pushes
(plus manual dispatch), keep the PR trigger, and cancel superseded runs
per ref so a push to a PR branch doesn't queue duplicates.
find_free_single_event_slot() treated every cached slot as occupied, so
expired one-shot events accumulated until the 35-slot pool exhausted
(the RFC-005 nowcast path writes one event per pre-heat burst). The
resolver now takes a reusable_before_ts and treats disabled events and
events that ended before it as free — the new event's begin timestamp
serves as "now", and writing over an expired slot both reuses and
clears it. Default 0 preserves the old behavior for other callers.

Updated the auto-slot test to seed a live (future-end) event and added
a reuse case: an enabled-but-expired slot 0 is picked and overwritten.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The aggregate Weekly Schedule JSON truncates at HA's 255-char state
cap (a real 20-entry grid already loses layer 4), so an external
scheduler recovering from state loss could not learn the full grid
before performing a destructive full-state upload — the bench run had
to brute-force a hidden entry against the canonical hash.

Adds five optional schedule_layer_0..4 text sensors, each publishing
its layer as 7 compact cells ([start_min,end_min] or 0, ~86 chars
worst case, always under the cap) via the pure
codec::layer_image_to_json. Republished alongside schedule_hash after
every settled schedule op; "unknown" until the layer is cached.
Together with the hash sensor this gives a race-free, poll-only
cold-start recovery path sized to the data.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adapts the bulk-upload additions to the review changes: upload payload
grammar errors settle invalid via the bridge, submit_upload_schedule
gains the origin parameter, and the WriteResult carries both the upload
summary fields and the requested_* echoes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant