[Tracking] Support HTTP retry for the multi-poll backend

[kingcrimsontianyu]

Our easy thread pool backend supports HTTP retry that allows the transfer to restart in case of certain types of errors. It is non-trivial to add this feature to our new multi-poll backend due to different code structure. This issue tracks the PRs to implement retry logic in multi-poll.

• Currently with the easy thread pool backend, if a transfer of a subrange times out, the retry logic will kick in and redo the transfer. However, the offset counters in the BounceBufferH2D and CallbackContext are not reset prior to the retry, which will cause the new transfer to start at the previous offset, eventually triggering the data overflow exception. This PR fixes this bug by resetting the offset counters before retry attempts. Multi-poll retry support (1/n): Fix a bug in request retry #1021
• Functionally this PR adds nothing new: it refactors and then extracts the core retry logic to be shared by the easy thread pool and multi-poll backends. Multi-poll retry support (2/n): Refactor and extract the retry logic #1022
• This small PR replaces the use of std::cout with KVIKIO_LOG_WARN to display the retry notices. Multi-poll retry support (3/n): Use KvikIO logger to display retry notices instead of the standard output #1023
• This PR supports runtime change of remote I/O backend. Multi-poll retry support (4/n): Support runtime change of remote I/O backend #1025
• This PR carries out the main implementation of the retry support for the multi-poll backend. The basic idea is that if a transfer fails but qualifier for retry, we move it to the end of the reactor-local queue _pending, and the transfer has a timer running for the backoff, so as to determine if the transfer should stay in _pending (backoff elapsed) or be deferred to the next iteration (backoff still ongoing). Multi-poll retry support (5/n): Main implementation #1026
• This PR does the following 2 things Fix multi poll backend deadlock #1027:
  • Previously, the multi poll backend occasionally hangs (to my experience, the frequency of hang was low for cudf-polars TPC-H SF-1K benchmarks, about 1 out of 30 runs). Interestingly, switching to CU_STREAM_NON_BLOCKING makes query 2 hang definitively, manifesting the bug. The hang happens due to this pattern: (1) In reactor's I/O thread, a shard's mutex in BounceBufferCachePerThreadAndContext::try_get() is used to acquire a buffer, which may call cuMemHostAlloc, (2) In cuLaunchHostFunc, the same mutex is used to return the buffer to the pool. Apparently cuLaunchHostFunc holds some internal mutex that blocks the progress of cuMemHostAlloc. The effective way to break the deadlock is to move CUDA call in BounceBufferCachePerThreadAndContext::try_get() out of the locked region.
  • This PR also makes similar changes to StreamCachePerThreadAndContext and BounceBufferCachePerThreadAndContext to make CUDA calls in non-locked region.