Add Python benchmark engine (async valkey-glide, redis-py, valkey-py)

[jamesx-improving]

Implements the Python engine (#11), at parity with the Java (reference), Ruby, and C# engines.

What

• New async engine under python/ (pip install -e ., python -m resp_bench), driving three async drivers, one client per connection on a single asyncio event loop:
  • valkey-glide-python — GLIDE async (import glide)
  • redis-py — redis.asyncio
  • valkey-py — valkey.asyncio (Valkey fork of redis-py)
  • recording — in-memory driver for server-free tests
• Consumes the shared driver/workload JSON and emits the exact NDJSON schema (metadata/phase/totals/metrics, unit:"us", uppercased command keys, HDR compressed base64).

Concurrency model

connections = N → N clients (one per connection — the client == connection invariant) driven by N worker coroutines via asyncio.gather, each awaiting one command at a time. This is the faithful async analogue of the Java/Ruby "one in-flight request per connection" model, keeping results comparable across engines. pipeline_depth > 1 is not yet implemented; a >1 request logs a warning and runs at depth 1.

Cross-engine parity

• JavaRandom LCG port (incl. int32-overflow rejection) — verified byte-identical to Java's java.util.Random (seed-0 anchor) and to the Ruby engine (seed 12345).
• sequential_int uses a per-phase shared counter (matches Java's forkForThread); uniform_rand seeded per worker seed+idx.
• Key formatting %0Nd honoring key_size_bytes; leaky-bucket rate limiter; HdrHistogram (1, 600_000_000, 3).

Harness

• Makefile python-build/test/run/clean/info targets.
• Registered the Python driver_ids in scripts/run_benchmark_matrix.py (DRIVER_ENGINE_MAP) and scripts/generate_graphs.py (DRIVER_LANGUAGE_MAP).
• Driver configs under configs/drivers/{default,high-throughput}/ + example-*-standalone.json.
• CI benchmark-python job (matrix over the three drivers) added to generate-graphs needs; drivers.json python list.
• Doc fix: corrected the HdrHistogram range in docs/ADDING_LANGUAGE.md from 3_600_000_000 (1 hour) to 600_000_000 (600s) — every engine uses 600s.

Tests

34 unit + integration tests (cd python && python -m pytest): parity anchors (JavaRandom, key generator), rate limiter, config loader, HDR encode/decode round-trip, NDJSON schema, and end-to-end runs via the recording driver (no server needed).

Notes

• valkey-glide driver_id is valkey-glide-python (not the bare valkey-glide, which is Java's) — matches the valkey-glide-ruby / valkey-glide-csharp convention.
• Live-server runs (glide/redis-py/valkey-py against a real Valkey) were not run in CI-less local dev; the recording-driver e2e covers the engine paths. GLIDE advanced/custom-CA TLS should be validated in a live run.

[jeremyprime]

Note that an earlier attempt (https://github.com/jduo/resp-bench/tree/python-engine) also added Python sync if we want to see how that was done and consider here.

[jamesx-improving]

Note that an earlier attempt (https://github.com/jduo/resp-bench/tree/python-engine) also added Python sync if we want to see how that was done and consider here.

Thanks — I went through that branch before opening this PR. Borrowed from it: the valkey-py driver, which is included here (hence three drivers in the title).

Skipped its sync mode deliberately: #11 calls for the async clients, and that path is ThreadPoolExecutor thread-per-connection (GIL-bound, same shape as Ruby), so it'd add a second non-comparable execution model. Happy to add it as a follow-up if you want sync-client numbers — it'd be additive, async staying the default.

[jeremyprime]

Note that an earlier attempt (https://github.com/jduo/resp-bench/tree/python-engine) also added Python sync if we want to see how that was done and consider here.

Thanks — I went through that branch before opening this PR. Borrowed from it: the valkey-py driver, which is included here (hence three drivers in the title).

Skipped its sync mode deliberately: #11 calls for the async clients, and that path is ThreadPoolExecutor thread-per-connection (GIL-bound, same shape as Ruby), so it'd add a second non-comparable execution model. Happy to add it as a follow-up if you want sync-client numbers — it'd be additive, async staying the default.

That's fine, just wanted to make sure we referenced the other implementation. We can add sync in the future as needed.

[Aryex]

Overall looks good. My main concerns that need to be resolves are that:

1. Need to ensure GLIDE client is shared across threads/connections. This is the recommended way to use GLIDE for the best performance.
2. In a similar vein, we need to ensure peer clients are configured correctly. Since there are many, I would suggest picking only 1, maybe 2, for this PR to start.

[jamesx-improving]

Overall looks good. My main concerns that need to be resolves are that:

1. Need to ensure GLIDE client is shared across threads/connections. This is the recommended way to use GLIDE for the best performance.
2. In a similar vein, we need to ensure peer clients are configured correctly. Since there are many, I would suggest picking only 1, maybe 2, for this PR to start.

Thanks — this was a genuinely useful review; the measured reproductions made every item easy to confirm. I reproduced all of them locally before changing anything, and all 12 are addressed in 4096bac.

On sharing the GLIDE client: I have kept one client per connection for now, because sharing a multiplexing client across workers is what ikolomi#11 explicitly declined — the reasoning there was that one-client-per-transport-connection is the cross-engine comparison baseline, and that transport sharing should be reached via pipelining instead. I do not think I should reverse that inside a language-engine PR. That said your point stands on its own merits: N separate GlideClients each with their own Tokio runtime is not how GLIDE is meant to be used, and it under-represents it. Your other comment also corrected my premise — only lettuce and redis-rb are actually 1:1 among the incumbents — which weakens the comparability argument. Could we settle it as a methodology question with @ikolomi on the original issue? An opt-in shared_client flag (default off, preserving the 1:1 baseline) would give both numbers, but it needs his agreement rather than just ours. The current choice and its rationale are documented in the meantime.

On peer drivers: agreed, and your own findings are the argument for it — the two peers were not configured equivalently. Dropped valkey-py from this PR; it is now GLIDE + redis-py only, with redis-py's protocol, retry policy, parser and command timeout all pinned explicitly rather than inherited, and those settings recorded in the metrics metadata. valkey-py can return as a follow-up once there is a pattern for holding peer configuration equivalent. @jeremyprime — this is the driver borrowed from jduo's branch, so flagging that it is deferred rather than dropped on the merits.

The four correctness bugs (worker cancellation, loop starvation, warmup, config validation) have tests; the suite is now 59 tests. Two of your findings led somewhere broader than the original report: the starvation hole also existed on the success path (any non-suspending driver let one connection monopolise a duration phase — [460010, 0, 0, 0] on the recording driver, now even), and the new ERROR status did not actually stop the matrix runner scoring a bad cell as ok, so the CLI now exits non-zero on any ERROR phase.

Given the severity of the starvation and validation issues I would rather you re-reviewed than merged on the earlier approval.

[jeremyprime]

Looks like we are following the conclusion of ikolomi#11 (i.e. client==connection, and use pipelining for shared transport). My only remaining concern I have is that Java/Ruby/C# all support pipelining and can set pipeline_depth>1, so I think we need the same pipeline support in the Python client.

[Aryex]

@jamesx-improving +1 on the pipeline_depth support

[yipin-chen]

I suggest to support both sync and async benchmark.

[Aryex]

Do we have sync clients? this could be added in a separate PR

[jeremyprime]

The GLIDE Ruby client is sync and already added so resp-bench. GLIDE Python has sync and async, so we should either add sync support in this PR or create a backlog item to add it later, all depending on how easy it is to add (can also reference the previous branch to add Python to resp-bench here).