Skip to content

command_timeout_ms is honored inconsistently across engines, and ignored entirely by two clients #29

Description

@jamesx-improving

The default/ driver profile lets each client fall back to its own library
default, which for GLIDE is 250ms and for most peers is unbounded — so GLIDE's
latency tail is silently clipped where its competitors' is not. Chasing that
turned up a bigger problem: two clients never read command_timeout_ms at all,
so high-throughput's command_timeout_ms: 10000 is a no-op for them.

Impact: no published number appears to be wrong, but the exposure grows with load

Nothing needs retracting today. Every timeout ceiling sits one to three orders
of magnitude above any latency we have actually recorded, so no run examined has
had a request clipped:

Data Worst latency observed Relevant ceiling Margin
Node AWS sweep — raw NDJSON, ~33M requests, up to 200 conns max 7.4–8.1ms, 0 errors 250ms (GLIDE) ~30×
Java charts — 100 clients, SET p99.9 GLIDE 4.0ms (worst driver: lettuce 23.2ms) 250ms (GLIDE) ~62×
Ruby charts — 100 clients, GET p99.9 GLIDE 1.8ms 500ms (hardcoded) ~274×

The Node figure is fully verified — I have the raw NDJSON and can confirm 0 errors
across the whole sweep. The Java and Ruby figures are inferred from the committed
PNG charts only
, since no raw NDJSON is checked in, so I cannot confirm their
error counts were zero. Treat those two rows as strong evidence, not proof.

Where it will bite. The distortion is zero until p99.9 reaches the hundreds of
milliseconds, then it turns on sharply and is unbounded and one-directional — it
always flatters GLIDE
, via two compounding effects:

  1. GLIDE's max and p99.9 cannot exceed the ceiling; its peers' can.
  2. A timed-out request returns immediately, so the slot issues its next request
    while a peer's slot is still blocked — inflating GLIDE's throughput, not just
    truncating its tail.

The conditions that trigger it are exactly the ones the AWS runner exists to
measure: a loaded or remote server, cross-AZ latency, large payloads, high
connection counts, or a single GC/failover stall. A cross-AZ p99.9 in the 300ms
range would silently convert GLIDE's slow requests into errors while leaving every
peer's tail intact — and the run would still report status: succeeded.

One thing is actively wrong right now, though it misstates configuration rather
than results: configs/drivers/high-throughput/*.json declares
command_timeout_ms: 10000 for both Ruby drivers, while Ruby GLIDE actually runs
at a hardcoded 500ms and redis-rb at no timeout at all (§2). Anyone reading those
configs to interpret Ruby high-throughput numbers is being misled by a factor
of 20.

Surfaced by @Aryex reviewing #27.

1. The default/ profile has no timeout floor

No default/ config in the repo sets command_timeout_ms — all 16, across every
engine:

configs/drivers/default/          command_timeout_ms
  valkey-glide.json                    None      <- Java GLIDE
  jedis.json                           None
  lettuce.json                         None
  redisson.json                        None
  spring-data-*.json  (x7)             None
  stackexchange-redis.json             None
  valkey-glide-csharp.json             None
  valkey-glide-ruby.json               None
  redis-rb.json                        None
  valkey-glide-node.json               None
  ioredis.json                         None
  iovalkey.json                        None

That is defensible — "out-of-box configuration" is a legitimate comparison axis.
The problem is what the libraries actually default to. Measured against
valkey 9.0.2, one driver at a time on an idle server, each issuing
DEBUG SLEEP 0.5 (@Aryex):

glide      TimeoutError: timed out (after 252ms)
ioredis    ok after 502ms
iovalkey   ok after 502ms

Confirmed against the libraries directly:

Client Default command timeout
@valkey/valkey-glide DEFAULT_REQUEST_TIMEOUT_IN_MILLISECONDS = 250
ioredis commandTimeout = undefined (unbounded)
iovalkey same as ioredis

So in the default/ profile a GLIDE request slower than ~250ms becomes an error
while the identical request on a peer completes. Three consequences in the
recorded output, since measure() records the error path as a request:

  • GLIDE's summary.max and p99.9 cannot exceed ~250ms; its peers' can.
  • The timed-out request is counted in both totals.requests and totals.errors.
  • Throughput skew: an erroring slot issues its next request immediately,
    where a peer's slot is still blocked on the slow one.

Per-driver detail behind the Node row in the impact table:

driver               requests   errors   max latency
ioredis             12,016,468        0        7.7 ms
iovalkey            11,810,263        0        7.4 ms
valkey-glide-node    8,989,511        0        8.1 ms

2. Two clients ignore command_timeout_ms entirely

This is the more serious half. Handling is inconsistent per engine:

Engine / driver Behaviour
Node — all three Applied when set (requestTimeout / commandTimeout); library default otherwise
Java — glide, jedis, lettuce Applied when set (requestTimeout, timeoutMillis, withTimeout)
C# — StackExchange.Redis Applied when set (SyncTimeout + AsyncTimeout)
C# — valkey-glide-csharp Never reads CommandTimeoutMs. Silently ignored.
Ruby — both drivers DriverConfig does not parse command_timeout_ms at all. The GLIDE client hardcodes timeout: 500; redis-rb sets none.

Evidence:

# C#: the property is parsed but only one of the two clients reads it.
$ grep -rn 'CommandTimeoutMs' csharp/src/
csharp/src/RespBench/Config/DriverConfig.cs:33:    public int? CommandTimeoutMs { get; set; }
csharp/src/RespBench/Client/Impl/StackExchangeRedisBenchmarkClient.cs:65,67,68
# ...no hit in ValkeyGlideBenchmarkClient.cs

# Ruby: the field never reaches the model.
$ grep -rn 'command_timeout_ms' ruby/lib/          # no output
# ruby/lib/resp_bench/config/driver_config.rb:8 — no command_timeout_ms attribute
attr_accessor :schema_version, :description, :driver_id, :mode,
              :tls, :auth, :specific_driver_config
# ruby/lib/resp_bench/client/impl/valkey_glide_client.rb:23 — hardcoded
timeout: 500 # request timeout in milliseconds (protobuf uint32)

Consequence: configs/drivers/high-throughput/*.json sets
command_timeout_ms: 10000 for both Ruby drivers and it does nothing — Ruby GLIDE
runs at a hardcoded 500ms. The config file states a timeout that is not applied,
which is worse than omitting it, because a reader comparing high-throughput
results assumes a uniform 10s ceiling.

3. high-throughput is itself inconsistent

12 of 16 high-throughput configs set 10000; the two C# ones do not:

configs/drivers/high-throughput/          command_timeout_ms
  stackexchange-redis.json                     None    <- inconsistent
  valkey-glide-csharp.json                     None    <- inconsistent (also ignored, see §2)
  everything else (12 configs)                10000

For StackExchange.Redis this is a live gap: it does honor the field, so setting
it would take effect.

4. Docs don't mention it

docs/CONFIG_SPECIFICATION.md does not document command_timeout_ms. The schema
describes it as "Command timeout in milliseconds. Applied to all drivers that
support it."
— the "that support it" escape hatch is doing a lot of work, and
there is no way for a user to tell which drivers those are.

Suggested fix

Roughly in priority order:

  1. Make every client honor the field. Add command_timeout_ms to Ruby's
    DriverConfig and apply it in both Ruby clients (replacing the hardcoded
    500); read CommandTimeoutMs in the C# GLIDE client. Until this lands, any
    config that sets the field for those drivers is misleading.
  2. Decide what default/ means, and write it down. Either (a) keep it as
    true out-of-box and document that latency tails are not comparable across
    drivers in that profile, or (b) give it a uniform generous floor (e.g. 10000)
    so the profile compares steady-state behaviour rather than divergent timeout
    policies. Whichever way, apply it to all 16 configs, not per engine.
  3. Set command_timeout_ms: 10000 in the two C# high-throughput configs,
    so the profile is uniform.
  4. Document the field in docs/CONFIG_SPECIFICATION.md, including a
    per-driver table of what it maps to, and the library default when unset.
  5. Consider making a driver fail loudly when handed a config option it does
    not implement, so silent no-ops like §2 cannot recur.

Notes

Pre-existing; not introduced by #27, whose Node configs deliberately follow the
existing convention (all default/ omit, high-throughput sets 10000). Filed
separately from that PR because a real fix touches Java, Ruby and C# driver code
and changes the meaning of every engine's published default/ numbers.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions