Add daemon health check with loading guts vs reindex notification/escalation

[dream-hermes-agent]

Problem

When the verusd container starts (especially after a no-bootstrap restart), verus getinfo returns -28 Loading block index... for an extended period. Two very different states produce the same RPC response:

• Loading block index (normal): Daemon is rebuilding the in-memory UTXO set from existing chainstate on disk. LoadBlockIndexDB: loaded guts appears in debug.log. Daemon will recover on its own within minutes.
• Reindex (serious): Daemon is replaying all blocks from disk, or chainstate is corrupt/missing. LoadBlockIndexDB: reindexing blocks or ERROR: ReadBlockFromDisk in debug.log. May indicate data loss or a chain fork.

Currently, monitoring/playbooks cannot distinguish between the two without reading debug.log directly.

Desired behavior

During startup health checks (e.g. 09-hygiene.yml, 10-shutdown-vrsc.yml, or a monitoring cron):

1. If verus getinfo returns -28 Loading block index...:

   • Check debug.log for LoadBlockIndexDB: loaded guts vs LoadBlockIndexDB: reindexing
   • If loading guts: log only, no alert. Wait and retry.
   • If reindexing: send notification (Telegram/email), escalate.

2. A separate playbook or script should expose this check so it can be used by both health checks and external monitoring.

Implementation suggestions

• Add a scripts/check-daemon-startup-state.sh that:
  1. Runs verus getinfo
  2. If -28, reads tail -1 debug.log for state detection
  3. Returns loading_guts | reindex | ready | unknown
• Use this in 09-hygiene.yml readiness checks
• Wire Telegram notification for reindex state to the on-call channel

References

• debug.log indicators:
  • Normal: LoadBlockIndexDB: loaded guts
  • Reindex: LoadBlockIndexDB: reindexing blocks or Reindexing in log
• RPC error code -28 = RPC_IN_WARMUP