deployd-api: hiqlite 0.14 WAL serialization change may brick boot on pre-existing single-node state

[bartekus]

Summary

The hiqlite 0.14 bump that landed in #558 (a2235ad7) changed the
on-disk WAL/logs-store serialization: hiqlite-wal dropped rkyv in
favour of bincode/serde. deployd-api-rs runs hiqlite as a
single node with an embedded Raft store, and its boot path treats a
start_node failure as fatal. An existing deployd data directory written
by a pre-0.14 hiqlite therefore risks a hard boot failure after the
upgrade, and hiqlite's auto-heal feature does not cover this case.

This is an operational upgrade caveat, not a code bug. Filing it so the
next deployd rollout against a persistent data dir has a tracked home
before someone hits it.

Why deployd is exposed

• Single-node topology. store.rs pins one node:
  HQL_NODES = "1 127.0.0.1:7001 127.0.0.1:7002"
  (platform/services/deployd-api-rs/src/store.rs:78).
• Fatal boot on store error. init_db propagates a start_node
  error with ? and has no deployd-level move-aside / retry:
  let client = hiqlite::start_node(config).await?;
  (platform/services/deployd-api-rs/src/store.rs:22-28). main.rs:44
  awaits init_db at startup, so a store-open error aborts the process.
• auto-heal does not cover a format change. deployd enables
  hiqlite's auto-heal feature (Cargo.toml:17,
  features = ["sqlite", "backup", "s3", "auto-heal"]), and that feature
  does extend into the WAL layer (hiqlite/Cargo.toml:24:
  auto-heal = ["hiqlite-wal/auto-heal"]). But the WAL-layer heal path
  (hiqlite-wal/src/wal.rs:110-157) only recovers from a bad CRC /
  torn write: it rolls the last log ID back to the last valid record
  and expects the truncated tail to be re-synced from another cluster
  member. A serialization-format change is not a CRC failure (the
  reader cannot parse the old record framing at all), and even if the
  heal path engaged, deployd is single-node and has no peer to
  re-sync from.

Impact

A deployd instance whose hiqlite data dir persists across the 0.14
upgrade (any PVC-backed dev/staging deployment via the Azure AKS or
Hetzner K3s paths) may fail to boot with a store-open / integrity error
and no automatic recovery. deployd's Raft store is deployment
orchestration state (a cache derivable from the live cluster + Helm
releases), not an irreplaceable source of truth, so the data loss is
tolerable; the failure mode (a crash-looping pod on rollout) is the
concern.

Mitigation (rollout runbook)

• If S3 backups are configured (DEPLOYD_BACKUP_*): restore the
  latest snapshot into a fresh data dir via the existing
  HQL_BACKUP_RESTORE=s3:<key> path (store.rs:86-110; validated by the
  restore_from_env_var ignore-gated test at store.rs:439-455). This
  reconstitutes state_machine/db/deployd.db from a sqlite snapshot and
  sidesteps replaying the incompatible WAL.
• Otherwise: wipe / reprovision the deployd hiqlite data dir on the
  0.14 rollout. deployd re-derives its orchestration state.

Optional durable fix (defer until scheduled)

If a persistent, migration-safe deployd store is wanted later:

• Detect an unreadable / format-incompatible store at boot and move it
  aside + reinit with a clear diagnostic, rather than crash-looping
  (the pattern axiomregent already uses with its port sidecar).
• Or document a mandatory pre-upgrade S3 snapshot + fresh-dir restore
  step in the deployd chart upgrade notes.

Either is a code/docs change to a spec-governed path
(platform/services/deployd-api-rs/), so it should graduate to a draft
spec when prioritized, not be bolted on here.

Not yet verified

• The exact hiqlite version deployd ran before chore(deps): consolidate non-spec-spine Dependabot bumps #558 (the pre-bump
  pin), and thus the precise from/to WAL format delta. The rkyv-drop
  fact comes from the 0.14 changelog investigation during chore(deps): consolidate non-spec-spine Dependabot bumps #558; the
  local hiqlite source confirms hiqlite-wal now serializes via
  bincode/serde with no rkyv.
• Whether any live deployd instance currently holds pre-0.14 state
  (project is pre-alpha, single-developer, no public releases), i.e.
  whether this is latent or immediate.

References

• chore(deps): consolidate non-spec-spine Dependabot bumps #558 (a2235ad7): the hiqlite 0.14 consolidation bump.
• Related MSRV fallout already fixed in the same PR: deployd Dockerfile
  FROM rust:1.95 (platform/services/deployd-api-rs/Dockerfile:12),
  since hiqlite 0.14 raised its MSRV.