History Publish (Testnet): validator dies during catchup — bucket entry XDR parse failure in restore_from_has_parallel (disk_bucket.rs:228)

[tomerweller]

Symptom

The History Publish (Testnet) job Test History Publish now fast-fails (~38–82 s) by the validator process dying during catchup, distinct from the checkpoint-publish timeout/runner-cancel mode tracked by #3741. The node downloads all buckets, begins restore_from_has_parallel (disk-backed mode), loads levels 0–10, then aborts parsing a bucket entry:

INFO  henyey_history::catchup::buckets: apply_buckets START — RSS 166 MB
INFO  henyey_history::catchup::buckets: Applying buckets to build state at ledger 4378879 (disk-backed mode)
...
INFO  henyey_bucket::bucket_list: restore_from_has_parallel: loaded level curr+snap level=10 curr_entries=5774175 snap_entries=0 elapsed_ms=6718
ERROR henyey_app::run_cmd: Node stopped with error error=catchup failed: Failed to restore live bucket list:
      bucket serialization error: Failed to parse bucket entry at offset 4483092: xdr value invalid
Error: catchup failed: Failed to restore live bucket list: bucket serialization error:
      Failed to parse bucket entry at offset 4483092: xdr value invalid
##[error]Process completed with exit code 1.

Evidence

Run	Created	Duration	Conclusion
33169000459	2026-08-28T11:55:26Z	38 s	failure (bucket-parse death, confirmed above)
33067408027	2026-08-27T11:26:55Z	82 s	failure (fast-fail; log expired, same mode by duration)

• Head sha: 129190c0 (current origin/main HEAD) — NOT the deployed validator binary 0ac84d42.
• Each daily run catches up to the then-current testnet checkpoint (here target_ledger=4378879), i.e. a different bucket set each run. A parse failure recurring across days on different checkpoints argues a systematic deserialization defect, not a one-off corrupt download.
• The older Aug-25/26 History Publish failures were ~180–246 s (the History Publish (Testnet): checkpoint-publish timeout persists on a35b1066 despite #3732 #3741 stall/timeout mode), so at least two distinct failure modes are now present in this workflow.

Code path (file:line)

• crates/bucket/src/disk_bucket.rs:228 — stellar_xdr::BucketEntry::from_xdr(record.body, Limits::none()) returns Err → crates/bucket/src/disk_bucket.rs:232-235 formats Failed to parse bucket entry at offset {}: {} (BucketError::Serialization).
• Record framing comes from self.records.next_record() (disk_bucket.rs:212); the raw mark_bytes+body are fed to the hasher before parse (:222-224), so a framing/offset mis-split would corrupt record.body.
• Propagates via crates/history/src/catchup/buckets.rs:450 → HistoryError::CatchupFailed("Failed to restore live bucket list: …").

Hypotheses (to be triaged)

1. Record framing / offset drift in next_record() — a mis-read length prefix would hand from_xdr misaligned bytes and produce "xdr value invalid" at a plausible-looking offset. Offset 4483092 is deep in the largest (level-10, 5.77 M-entry) bucket.
2. A BucketEntry variant henyey's XDR can't decode on a recent testnet bucket (protocol/type the deserializer rejects). Would be a genuine parity gap.
3. Recently introduced on main (present on 129190c0, deployed 0ac84d42 predates it) — bisect 0ac84d42..129190c0 over crates/bucket/ and the vendored stellar_xdr pin.

Why this matters beyond CI

restore_from_has_parallel is the same code path used by the operator-only cold-catchup recovery for the current mainnet outage (#3905 / #3702: recovery = wipe + cold catchup, which restores the live bucket list from a HAS). If this deserialization defect also affects mainnet buckets on 129190c0, a binary carrying it could fail the recovery catchup. (Not yet confirmed on mainnet buckets — mainnet cold-catchup last succeeded 2026-07-01 on 0ac84d42.) Flagging so a fix lands before any redeploy off 0ac84d42.

Suggested next steps

• Pull the failing testnet bucket for checkpoint 4378879 and reproduce offline: parse level-10 curr bucket, locate the record at offset 4483092, and inspect whether framing or the entry body is at fault.
• Compare next_record() framing against stellar-core's bucket record format for the entry preceding offset 4483092.
• Add a regression test around disk_bucket.rs record framing on a real testnet bucket fixture.

Distinct from #3741 (checkpoint-publish timeout/cancel) — filing separately per the different subsystem + root cause; Related to #3741.

---

Filed by monitor-tick (validator is separately down under #3905; this is a CI-surfaced correctness finding on 129190c0).

[tomerweller]

Recurrence — 2026-08-29T08:14Z

Same bucket-parse-death signature recurred on a fresh checkpoint.

• Run: 33242679469 — Test History Publish job failure
• headSha: 129190c0
• Duration: 37s (08:14:25Z → 08:15:02Z) — fast-fail, consistent with the 38–82s signature
• Checkpoint: checkpoint_seq=4393535, target_ledger=4393535
• Error:

  ERROR henyey_app::run_cmd: Node stopped with error
    error=catchup failed: Failed to restore live bucket list:
    bucket serialization error: Failed to parse bucket entry at offset 6083448: xdr value invalid
  

Failure lands in restore_from_has_parallel immediately after level 10 loads
(curr_entries=5774175 snap_entries=0 elapsed_ms=6990), i.e. during the
disk-backed bucket-list restore — same path as the prior occurrences. All 42
buckets downloaded cleanly before the parse died, so this is a bucket-decode
defect, not a download/network issue.

Offset differs from prior reports (checkpoint-specific), confirming this is not
a single corrupt archived bucket but a systematic decode failure across
checkpoints/days.

Note: this is the same restore_from_has_parallel code path the mainnet
validator would exercise on the operator-authorized cold-catchup recovery for
the #3702/#3905 downtime — worth weighting accordingly.

[tomerweller]

Recurrence — 2026-08-30T08:14Z

Same bucket-parse fast-fail on main HEAD 129190c0.

• Run: 33301114001 — job Test History Publish failure
• Duration: ~42s (08:14:56Z → 08:15:38Z) — within the 38–82s fast-fail band
• Error: catchup failed: Failed to restore live bucket list: bucket serialization error: Failed to parse bucket entry at offset 15444880: xdr value invalid
• Checkpoint: checkpoint_seq=4410815, target_ledger=4410815
• Phase: all 38/38 buckets downloaded; died inside restore_from_has_parallel after level 9 loaded (apply_buckets START — RSS 162 MB → ERROR ~6.9s later)

New offset (15444880) and checkpoint vs the 08-29 recurrence (offset differs day-to-day, consistent with a parse defect keyed on the specific archived bucket bytes at each fresh testnet checkpoint, not a fixed corrupt file). No code change on main since last recurrence (HEAD unchanged at 129190c0).

[tomerweller]

Recurrence — 2026-08-31T08:20Z

Same bucket-parse fast-fail on main HEAD 129190c0 (surfaced in-window by /monitor-tick check 11).

• Run: 33372410789 — job Test History Publish failure
• Duration: ~47s (08:20:03Z → 08:20:50Z) — within the 38–82s fast-fail band
• Checkpoint: checkpoint_seq=4428159, target_ledger=4428159 (fresh checkpoint vs prior recurrences)
• Phase: all 37/37 buckets downloaded; died inside restore_from_has_parallel right after level-10 loaded (curr_entries=5774175 snap_entries=0)
• Error:

  ERROR henyey_app::run_cmd: Node stopped with error
    error=catchup failed: Failed to restore live bucket list:
    bucket serialization error: Failed to parse bucket entry at offset 19033816: xdr value invalid
  

Offset trend (varies every run, but the failing level is consistently level-10 = the largest curr bucket):

Date	Run	Checkpoint	Offset
08-29	33242679469	4393535	(37s)
08-30	33301114001	4410815	15444880
08-31	33372410789	4428159	19033816

Continues to recur daily on the scheduled testnet HP run, each on a fresh checkpoint with a different offset — consistent with a systematic deserialization defect in disk-backed restore_from_has_parallel, not one-off corrupt downloads. No new action beyond this log; root cause tracking stays here.

[tomerweller]

Recurrence — daily data point (monitor-tick).

• Run: 33486374656, workflow History Publish (Testnet), job Test History Publish
• Head: 129190c0
• Conclusion: failure (NOT cancelled) — this is the History Publish (Testnet): validator dies during catchup — bucket entry XDR parse failure in restore_from_has_parallel (disk_bucket.rs:228) #3941 fast-fail mode, distinct from the History Publish (Testnet): checkpoint-publish timeout persists on a35b1066 despite #3732 #3741 timeout-cancel
• Duration: 40s (in the 38–82s fast-fail band)
• Signature (unchanged, new offset):

INFO ... restore_from_has_parallel: loaded level curr+snap level=10 curr_entries=5774175 snap_entries=0 elapsed_ms=7530
ERROR henyey_app::run_cmd: Node stopped with error error=catchup failed: Failed to restore live bucket list: bucket serialization error: Failed to parse bucket entry at offset 5500004: xdr value invalid
Error: catchup failed: Failed to restore live bucket list: bucket serialization error: Failed to parse bucket entry at offset 5500004: xdr value invalid

Death occurs immediately after the level-10 curr restore (snap_entries=0), parsing an entry at offset 5500004 — same restore_from_has_parallel / disk_bucket.rs path. Still reproduces every scheduled run (08-29, 08-30, 08-31, 09-01). Same code path as the #3905/#3702 cold-catchup recovery.

[tomerweller]

Recurrence — 2026-09-02T08:18Z (daily data point, monitor-tick)

Fresh in-window failure of the daily History Publish (Testnet) run — same fast-fail bucket-parse death, on a new checkpoint at a new offset, which keeps arguing a systematic deserialization defect (not a one-off corrupt download).

• Run: 33607982645 — failure, headSha 129190c0, ~40 s fast-fail (08:18:17 → 08:18:51Z)
• Checkpoint: checkpoint_seq=4462655 (prior day 09-01 was 4445375)
• Offset: Failed to parse bucket entry at offset 1891160: xdr value invalid (prior day 09-01 was offset 5500004)
• Site: restore_from_has_parallel, right after level-10 curr_entries=5774175 / level-9 loaded (crates/bucket/src/disk_bucket.rs:228 BucketEntry::from_xdr → crates/history/src/catchup/buckets.rs:450)
• Error: catchup failed: Failed to restore live bucket list: bucket serialization error: Failed to parse bucket entry at offset 1891160: xdr value invalid

Distinct new (checkpoint, offset) pair vs every prior day (08-29 ckpt 4393535/off 6083448; 09-01 ckpt 4445375/off 5500004) → confirms the parse failure is not pinned to a specific checkpoint or byte offset. No mainnet cold-catchup regression is confirmed from this testnet path.

[tomerweller]

Recurrence — 2026-09-03T08:18Z (daily data point, monitor-tick)

Fresh in-window failure of the daily History Publish (Testnet) run — same fast-fail bucket-parse death during catchup, on main HEAD 129190c0. Surfaced by /monitor-tick check 11.

• Run: 33732631119, workflow History Publish (Testnet), job Test History Publish, attempt 1
• Fast-fail window: started 08:18:21Z → completed 08:18:59Z (~38s) — inside the established 38–82s band
• Signature (unchanged):

  ERROR henyey_app::run_cmd: Node stopped with error error=catchup failed: Failed to restore live bucket list: bucket serialization error: Failed to parse bucket entry at offset 671856: xdr value invalid
  

• Parse offset: 671856 (offset continues to vary per checkpoint across runs — consistent with a parser/writer-side defect rather than one specific corrupt entry)
• Death occurs in restore_from_has_parallel after loading all bucket levels (last logged: level=10 curr_entries=5774175), same code path as the OFFLINE 20 min: near-tip stall at a 63-mod-64 boundary escalated to a FATAL pre-close hash mismatch at L64069504 on 0ac84d42 (state wipe demanded; restarted without wipe) #3905/OFFLINE: restore-from-disk near-tip back-fill stall (NOT binary-specific) — wedges on both f850e6da and 0ac84d42 #3702 cold-catchup recovery.

VE-Mainnet (33722461780, 06:16Z) and Archive-Done (33726745446, 07:10Z) both green in-window; this is the only red main-branch workflow. No re-file; appending to the recurrence series only.

[tomerweller]

Recurrence — 2026-09-04T08:18Z (daily data point, monitor-tick)

Fresh in-window failure of the daily History Publish (Testnet) job — same #3941 bucket-parse death mode (distinct from the #3741 cancel mode).

• Run: 33852737573 — job Test History Publish, conclusion failure (not cancelled)
• Head: 129190c0 (HEAD/origin/main; VE-Mainnet green on this sha)
• Mode: fast-fail validator process death during catchup (~25s: validator pid up 08:17:54Z, died 08:18:19Z)
• Checkpoint: 4497215 (SCP trust anchor f140b9c0…), disk-backed bucket apply
• Fatal: catchup failed: Failed to restore live bucket list: bucket serialization error: Failed to parse bucket entry at offset 1842908: xdr value invalid
• Locus: restore_from_has_parallel — died immediately after loading level 10 (curr_entries=5774175 snap_entries=0, elapsed 5899ms); levels 0–4, 9, 10 loaded cleanly first.

Offset/checkpoint continue to vary per run (offset 1842908 today), consistent with a non-deterministic parse-boundary failure in the parallel HAS restore rather than a fixed-offset corruption. No new issue filed; recurrence logged here per policy.