Unable to sync complete blob history (mainnet)

[YuXiaoCoder]

Description

I need the complete Blob data, so I want to achieve this through the backfill function.

Version

./lighthouse --version
Lighthouse v8.1.3-176cce5
BLS library: blst-portable
BLS hardware acceleration: true
SHA256 hardware acceleration: true
Allocator: jemalloc (4K)
Profile: maxperf
Specs: mainnet (true), minimal (true), gnosis (true)

Present Behaviour

Can catch up with the latest block, but cannot perform backfill

Apr 22 10:59:47.556 INFO  Sync state updated                            old_state: Synced, new_state: Syncing Historical Blocks
Apr 22 10:59:48.418 INFO  New block received                            slot: 14166897, root: 0xb3d4bbdab80267b5c2abd14448e2b0dd9adde1b3a10285cd8934ad9cd6babcce
Apr 22 10:59:53.000 INFO  Downloading historical blocks                 distance: "14005024 slots (277 weeks 6 days)", est_time: "--"
Apr 22 10:59:53.000 INFO  Synced                                        peers: "82", exec_hash: "0x6afee3db7166d9112ab708ad7c3ef264176f620225e97849c8e3dca2d578c3a5 (verified)", finalized_root: 0xf74bfcdd1fb60ad29f23b537c5bd9b692dff9abc57b9e8fb90af984cff32c47e, finalized_epoch: 442713, epoch: 442715, block: "0xb3d4bbdab80267b5c2abd14448e2b0dd9adde1b3a10285cd8934ad9cd6babcce", slot: 14166897
Apr 22 11:00:00.704 INFO  New block received                            slot: 14166898, root: 0xf54746271d9badb31129969d10eeef2b2f42377de2dfd6569a426cf106b8df5c
Apr 22 11:00:05.000 INFO  Synced                                        peers: "77", exec_hash: "0x79ca551da007d9713129ad0ca136b13fa3da4f2d941c80fb58dd171a0a776460 (verified)", finalized_root: 0xf74bfcdd1fb60ad29f23b537c5bd9b692dff9abc57b9e8fb90af984cff32c47e, finalized_epoch: 442713, epoch: 442715, block: "0xf54746271d9badb31129969d10eeef2b2f42377de2dfd6569a426cf106b8df5c", slot: 14166898
Apr 22 11:00:13.139 INFO  New block received                            slot: 14166899, root: 0xdaa7050b799901c957a4952b9fc3c79e7965b61f46cc1848b1afdc325d7f73f8
Apr 22 11:00:17.000 INFO  Synced                                        peers: "75", exec_hash: "0xcaf4712bdefc1f46b2ea40366b5caf81005e6a1637ddd918d66d3c12164a5fe2 (verified)", finalized_root: 0xf74bfcdd1fb60ad29f23b537c5bd9b692dff9abc57b9e8fb90af984cff32c47e, finalized_epoch: 442713, epoch: 442715, block: "0xdaa7050b799901c957a4952b9fc3c79e7965b61f46cc1848b1afdc325d7f73f8", slot: 14166899
Apr 22 11:00:24.745 INFO  New block received                            slot: 14166900, root: 0x6e6a8b3470acddd6b91251b2ff4411504df4592655b3726543874aefae357630
Apr 22 11:00:29.000 INFO  Synced                                        peers: "73", exec_hash: "0xbe0a1e497faede3a5d41fc08b6c29ebe00d833a378064bfc88709d70aca89d71 (verified)", finalized_root: 0xf74bfcdd1fb60ad29f23b537c5bd9b692dff9abc57b9e8fb90af984cff32c47e, finalized_epoch: 442713, epoch: 442715, block: "0x6e6a8b3470acddd6b91251b2ff4411504df4592655b3726543874aefae357630", slot: 14166900
Apr 22 11:00:37.506 INFO  New block received                            slot: 14166901, root: 0x51c0c8ca192c4aab0aed7fc86c5406f1f6f652d9e567906eacf547c52181ca27
Apr 22 11:00:41.000 INFO  Synced                                        peers: "76", exec_hash: "0x8138d802ddc57ee339a7793a4f4fb0cf9be1c5a82589be3d16d44ceece45c8bf (verified)", finalized_root: 0xf74bfcdd1fb60ad29f23b537c5bd9b692dff9abc57b9e8fb90af984cff32c47e, finalized_epoch: 442713, epoch: 442715, block: "0x51c0c8ca192c4aab0aed7fc86c5406f1f6f652d9e567906eacf547c52181ca27", slot: 14166901
Apr 22 11:00:45.302 ERROR Backfill sync failed                          error: BatchDownloadFailed(Epoch(437652))

Expected Behaviour

Backfilling blobs are possible

Steps to resolve

Please describe the steps required to resolve this issue, if known.

[michaelsproul]

This feature is still somewhat experimental. Do you see backfill sync failing and restarting periodically?

The network is not really designed to let you sync historical blobs beyond the DA window, because almost all peers prune them. So your node is fighting an uphill battle to ban most of the nodes on the network and latch onto the few nodes that do have historical blob data. If you can peer your node with a node that you know has all the blobs, that will help.