The default/ driver profile lets each client fall back to its own library
default, which for GLIDE is 250ms and for most peers is unbounded — so GLIDE's
latency tail is silently clipped where its competitors' is not. Chasing that
turned up a bigger problem: two clients never read command_timeout_ms at all,
so high-throughput's command_timeout_ms: 10000 is a no-op for them.
Impact: no published number appears to be wrong, but the exposure grows with load
Nothing needs retracting today. Every timeout ceiling sits one to three orders
of magnitude above any latency we have actually recorded, so no run examined has
had a request clipped:
| Data |
Worst latency observed |
Relevant ceiling |
Margin |
| Node AWS sweep — raw NDJSON, ~33M requests, up to 200 conns |
max 7.4–8.1ms, 0 errors |
250ms (GLIDE) |
~30× |
| Java charts — 100 clients, SET p99.9 |
GLIDE 4.0ms (worst driver: lettuce 23.2ms) |
250ms (GLIDE) |
~62× |
| Ruby charts — 100 clients, GET p99.9 |
GLIDE 1.8ms |
500ms (hardcoded) |
~274× |
The Node figure is fully verified — I have the raw NDJSON and can confirm 0 errors
across the whole sweep. The Java and Ruby figures are inferred from the committed
PNG charts only, since no raw NDJSON is checked in, so I cannot confirm their
error counts were zero. Treat those two rows as strong evidence, not proof.
Where it will bite. The distortion is zero until p99.9 reaches the hundreds of
milliseconds, then it turns on sharply and is unbounded and one-directional — it
always flatters GLIDE, via two compounding effects:
- GLIDE's
max and p99.9 cannot exceed the ceiling; its peers' can.
- A timed-out request returns immediately, so the slot issues its next request
while a peer's slot is still blocked — inflating GLIDE's throughput, not just
truncating its tail.
The conditions that trigger it are exactly the ones the AWS runner exists to
measure: a loaded or remote server, cross-AZ latency, large payloads, high
connection counts, or a single GC/failover stall. A cross-AZ p99.9 in the 300ms
range would silently convert GLIDE's slow requests into errors while leaving every
peer's tail intact — and the run would still report status: succeeded.
One thing is actively wrong right now, though it misstates configuration rather
than results: configs/drivers/high-throughput/*.json declares
command_timeout_ms: 10000 for both Ruby drivers, while Ruby GLIDE actually runs
at a hardcoded 500ms and redis-rb at no timeout at all (§2). Anyone reading those
configs to interpret Ruby high-throughput numbers is being misled by a factor
of 20.
Surfaced by @Aryex reviewing #27.
1. The default/ profile has no timeout floor
No default/ config in the repo sets command_timeout_ms — all 16, across every
engine:
configs/drivers/default/ command_timeout_ms
valkey-glide.json None <- Java GLIDE
jedis.json None
lettuce.json None
redisson.json None
spring-data-*.json (x7) None
stackexchange-redis.json None
valkey-glide-csharp.json None
valkey-glide-ruby.json None
redis-rb.json None
valkey-glide-node.json None
ioredis.json None
iovalkey.json None
That is defensible — "out-of-box configuration" is a legitimate comparison axis.
The problem is what the libraries actually default to. Measured against
valkey 9.0.2, one driver at a time on an idle server, each issuing
DEBUG SLEEP 0.5 (@Aryex):
glide TimeoutError: timed out (after 252ms)
ioredis ok after 502ms
iovalkey ok after 502ms
Confirmed against the libraries directly:
| Client |
Default command timeout |
@valkey/valkey-glide |
DEFAULT_REQUEST_TIMEOUT_IN_MILLISECONDS = 250 |
ioredis |
commandTimeout = undefined (unbounded) |
iovalkey |
same as ioredis |
So in the default/ profile a GLIDE request slower than ~250ms becomes an error
while the identical request on a peer completes. Three consequences in the
recorded output, since measure() records the error path as a request:
- GLIDE's
summary.max and p99.9 cannot exceed ~250ms; its peers' can.
- The timed-out request is counted in both
totals.requests and totals.errors.
- Throughput skew: an erroring slot issues its next request immediately,
where a peer's slot is still blocked on the slow one.
Per-driver detail behind the Node row in the impact table:
driver requests errors max latency
ioredis 12,016,468 0 7.7 ms
iovalkey 11,810,263 0 7.4 ms
valkey-glide-node 8,989,511 0 8.1 ms
2. Two clients ignore command_timeout_ms entirely
This is the more serious half. Handling is inconsistent per engine:
| Engine / driver |
Behaviour |
| Node — all three |
Applied when set (requestTimeout / commandTimeout); library default otherwise |
| Java — glide, jedis, lettuce |
Applied when set (requestTimeout, timeoutMillis, withTimeout) |
| C# — StackExchange.Redis |
Applied when set (SyncTimeout + AsyncTimeout) |
| C# — valkey-glide-csharp |
Never reads CommandTimeoutMs. Silently ignored. |
| Ruby — both drivers |
DriverConfig does not parse command_timeout_ms at all. The GLIDE client hardcodes timeout: 500; redis-rb sets none. |
Evidence:
# C#: the property is parsed but only one of the two clients reads it.
$ grep -rn 'CommandTimeoutMs' csharp/src/
csharp/src/RespBench/Config/DriverConfig.cs:33: public int? CommandTimeoutMs { get; set; }
csharp/src/RespBench/Client/Impl/StackExchangeRedisBenchmarkClient.cs:65,67,68
# ...no hit in ValkeyGlideBenchmarkClient.cs
# Ruby: the field never reaches the model.
$ grep -rn 'command_timeout_ms' ruby/lib/ # no output
# ruby/lib/resp_bench/config/driver_config.rb:8 — no command_timeout_ms attribute
attr_accessor :schema_version, :description, :driver_id, :mode,
:tls, :auth, :specific_driver_config
# ruby/lib/resp_bench/client/impl/valkey_glide_client.rb:23 — hardcoded
timeout: 500 # request timeout in milliseconds (protobuf uint32)
Consequence: configs/drivers/high-throughput/*.json sets
command_timeout_ms: 10000 for both Ruby drivers and it does nothing — Ruby GLIDE
runs at a hardcoded 500ms. The config file states a timeout that is not applied,
which is worse than omitting it, because a reader comparing high-throughput
results assumes a uniform 10s ceiling.
3. high-throughput is itself inconsistent
12 of 16 high-throughput configs set 10000; the two C# ones do not:
configs/drivers/high-throughput/ command_timeout_ms
stackexchange-redis.json None <- inconsistent
valkey-glide-csharp.json None <- inconsistent (also ignored, see §2)
everything else (12 configs) 10000
For StackExchange.Redis this is a live gap: it does honor the field, so setting
it would take effect.
4. Docs don't mention it
docs/CONFIG_SPECIFICATION.md does not document command_timeout_ms. The schema
describes it as "Command timeout in milliseconds. Applied to all drivers that
support it." — the "that support it" escape hatch is doing a lot of work, and
there is no way for a user to tell which drivers those are.
Suggested fix
Roughly in priority order:
- Make every client honor the field. Add
command_timeout_ms to Ruby's
DriverConfig and apply it in both Ruby clients (replacing the hardcoded
500); read CommandTimeoutMs in the C# GLIDE client. Until this lands, any
config that sets the field for those drivers is misleading.
- Decide what
default/ means, and write it down. Either (a) keep it as
true out-of-box and document that latency tails are not comparable across
drivers in that profile, or (b) give it a uniform generous floor (e.g. 10000)
so the profile compares steady-state behaviour rather than divergent timeout
policies. Whichever way, apply it to all 16 configs, not per engine.
- Set
command_timeout_ms: 10000 in the two C# high-throughput configs,
so the profile is uniform.
- Document the field in
docs/CONFIG_SPECIFICATION.md, including a
per-driver table of what it maps to, and the library default when unset.
- Consider making a driver fail loudly when handed a config option it does
not implement, so silent no-ops like §2 cannot recur.
Notes
Pre-existing; not introduced by #27, whose Node configs deliberately follow the
existing convention (all default/ omit, high-throughput sets 10000). Filed
separately from that PR because a real fix touches Java, Ruby and C# driver code
and changes the meaning of every engine's published default/ numbers.
The
default/driver profile lets each client fall back to its own librarydefault, which for GLIDE is 250ms and for most peers is unbounded — so GLIDE's
latency tail is silently clipped where its competitors' is not. Chasing that
turned up a bigger problem: two clients never read
command_timeout_msat all,so
high-throughput'scommand_timeout_ms: 10000is a no-op for them.Impact: no published number appears to be wrong, but the exposure grows with load
Nothing needs retracting today. Every timeout ceiling sits one to three orders
of magnitude above any latency we have actually recorded, so no run examined has
had a request clipped:
max7.4–8.1ms, 0 errorsThe Node figure is fully verified — I have the raw NDJSON and can confirm 0 errors
across the whole sweep. The Java and Ruby figures are inferred from the committed
PNG charts only, since no raw NDJSON is checked in, so I cannot confirm their
error counts were zero. Treat those two rows as strong evidence, not proof.
Where it will bite. The distortion is zero until p99.9 reaches the hundreds of
milliseconds, then it turns on sharply and is unbounded and one-directional — it
always flatters GLIDE, via two compounding effects:
maxandp99.9cannot exceed the ceiling; its peers' can.while a peer's slot is still blocked — inflating GLIDE's throughput, not just
truncating its tail.
The conditions that trigger it are exactly the ones the AWS runner exists to
measure: a loaded or remote server, cross-AZ latency, large payloads, high
connection counts, or a single GC/failover stall. A cross-AZ p99.9 in the 300ms
range would silently convert GLIDE's slow requests into errors while leaving every
peer's tail intact — and the run would still report
status: succeeded.One thing is actively wrong right now, though it misstates configuration rather
than results:
configs/drivers/high-throughput/*.jsondeclarescommand_timeout_ms: 10000for both Ruby drivers, while Ruby GLIDE actually runsat a hardcoded 500ms and
redis-rbat no timeout at all (§2). Anyone reading thoseconfigs to interpret Ruby
high-throughputnumbers is being misled by a factorof 20.
Surfaced by @Aryex reviewing #27.
1. The
default/profile has no timeout floorNo
default/config in the repo setscommand_timeout_ms— all 16, across everyengine:
That is defensible — "out-of-box configuration" is a legitimate comparison axis.
The problem is what the libraries actually default to. Measured against
valkey 9.0.2, one driver at a time on an idle server, each issuing
DEBUG SLEEP 0.5(@Aryex):Confirmed against the libraries directly:
@valkey/valkey-glideDEFAULT_REQUEST_TIMEOUT_IN_MILLISECONDS = 250iorediscommandTimeout=undefined(unbounded)iovalkeySo in the
default/profile a GLIDE request slower than ~250ms becomes an errorwhile the identical request on a peer completes. Three consequences in the
recorded output, since
measure()records the error path as a request:summary.maxandp99.9cannot exceed ~250ms; its peers' can.totals.requestsandtotals.errors.where a peer's slot is still blocked on the slow one.
Per-driver detail behind the Node row in the impact table:
2. Two clients ignore
command_timeout_msentirelyThis is the more serious half. Handling is inconsistent per engine:
requestTimeout/commandTimeout); library default otherwiserequestTimeout,timeoutMillis,withTimeout)SyncTimeout+AsyncTimeout)CommandTimeoutMs. Silently ignored.DriverConfigdoes not parsecommand_timeout_msat all. The GLIDE client hardcodestimeout: 500;redis-rbsets none.Evidence:
Consequence:
configs/drivers/high-throughput/*.jsonsetscommand_timeout_ms: 10000for both Ruby drivers and it does nothing — Ruby GLIDEruns at a hardcoded 500ms. The config file states a timeout that is not applied,
which is worse than omitting it, because a reader comparing
high-throughputresults assumes a uniform 10s ceiling.
3.
high-throughputis itself inconsistent12 of 16
high-throughputconfigs set10000; the two C# ones do not:For StackExchange.Redis this is a live gap: it does honor the field, so setting
it would take effect.
4. Docs don't mention it
docs/CONFIG_SPECIFICATION.mddoes not documentcommand_timeout_ms. The schemadescribes it as "Command timeout in milliseconds. Applied to all drivers that
support it." — the "that support it" escape hatch is doing a lot of work, and
there is no way for a user to tell which drivers those are.
Suggested fix
Roughly in priority order:
command_timeout_msto Ruby'sDriverConfigand apply it in both Ruby clients (replacing the hardcoded500); readCommandTimeoutMsin the C# GLIDE client. Until this lands, anyconfig that sets the field for those drivers is misleading.
default/means, and write it down. Either (a) keep it astrue out-of-box and document that latency tails are not comparable across
drivers in that profile, or (b) give it a uniform generous floor (e.g. 10000)
so the profile compares steady-state behaviour rather than divergent timeout
policies. Whichever way, apply it to all 16 configs, not per engine.
command_timeout_ms: 10000in the two C#high-throughputconfigs,so the profile is uniform.
docs/CONFIG_SPECIFICATION.md, including aper-driver table of what it maps to, and the library default when unset.
not implement, so silent no-ops like §2 cannot recur.
Notes
Pre-existing; not introduced by #27, whose Node configs deliberately follow the
existing convention (all
default/omit,high-throughputsets 10000). Filedseparately from that PR because a real fix touches Java, Ruby and C# driver code
and changes the meaning of every engine's published
default/numbers.