Skip to content

Handle benchmark errors gracefully instead of panicking#118

Open
qdrant-cloud-bot wants to merge 1 commit intodevfrom
fix/handle-retry-exhaustion-gracefully
Open

Handle benchmark errors gracefully instead of panicking#118
qdrant-cloud-bot wants to merge 1 commit intodevfrom
fix/handle-retry-exhaustion-gracefully

Conversation

@qdrant-cloud-bot
Copy link
Copy Markdown

Summary

  • Replace .unwrap() on run_benchmark() result in main() with proper error handling that prints the error and exits with code 1
  • Previously, when all upsert retries were exhausted (e.g. during chaos testing with frequent node kills + resharding), the error propagated to main() and hit .unwrap(), producing an ugly panic with stack trace instead of a clean exit

Context

Observed on qdrant-chaos-testing-three (2026-03-25 17:00–19:00 UTC): the combination of kill-random-node cron (~every 10 min), resharding, and rolling updates caused upsert timeouts. After exhausting all 5 retries, bfb panicked with:

called `Result::unwrap()` on an `Err` value: Error in the response: The operation was cancelled Timeout expired

Test plan

  • cargo check passes
  • Run bfb against a stopped/unreachable cluster and verify it prints a clean error message and exits with code 1 instead of panicking

Made with Cursor

Replace `.unwrap()` on `run_benchmark()` result with proper error
handling. Previously, when all upsert retries were exhausted (e.g.
during chaos testing node kills), the error would propagate to main()
and hit `.unwrap()`, producing an ugly panic with stack trace instead
of a clean error message and exit code.

Made-with: Cursor
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant