Fix XPENDING reply schema for empty pending lists - #4653
Merged
zuiderkwast merged 1 commit intoSep 11, 2026
Merged
zuiderkwast merged 1 commit into
zuiderkwast merged 1 commit into
Conversation
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (1)
Included review availability: Your plan provides up to 10 included reviews per hour; 9 remain after this review. 📝 WalkthroughWalkthroughThe XPENDING command declaration now includes a reply schema for empty pending-message summaries. The schema defines a zero count and null values for the minimum ID, maximum ID, and consumer list. ChangesXPENDING reply schema
Priority: ⬇️ Low Estimated code review effort: 1 (Trivial) | ~5 minutes Merge Risk: ⚪ Minimal · up to This change documents the valid empty XPENDING response without altering command behavior. 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
sarthakaggarwal97
added
the
run-extra-tests
sarthakaggarwal97
changed the title
XPENDING returns a zero count followed by three null values when the consumer group's pending entries list is empty. Model that response as a separate schema alternative so the reply validator accepts it without weakening validation of non-empty summaries. Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
sarthakaggarwal97
force-pushed
the
fix-xpending-empty-reply-schema
branch
from
September 10, 2026 23:37
fa21b7c to
457ef99
Compare
sarthakaggarwal97
marked this pull request as ready for review
September 10, 2026 23:41
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## unstable #4653 +/- ##
============================================
- Coverage 80.12% 80.03% -0.09%
============================================
Files 189 189
Lines 97253 97253
============================================
- Hits 77923 77837 -86
- Misses 19330 19416 +86 🚀 New features to boost your workflow:
|
zuiderkwast
approved these changes
Sep 11, 2026
zuiderkwast
pushed a commit
to Tarte12/valkey
that referenced
this pull request
Sep 15, 2026
When a consumer group has no pending messages, `XPENDING key group`
returns:
[0, null, null, null]
The reply schema only described non-empty summaries and extended
replies, so the reply-schema validator rejected this valid response.
This adds a dedicated schema variant for the empty summary while
preserving strict validation for non-empty responses. There is no change
to command behavior.
This started surfacing consistently after valkey-io#4629 added stream tests that
call `XPENDING` after clearing the pending entries list.
Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
This was referenced Sep 16, 2026
madolson
added a commit
that referenced
this pull request
Sep 16, 2026
`process_file` runs inside a `multiprocessing.Pool` worker, and on a
reply schema mismatch it re-raised the `jsonschema` exception. That
exception holds a reference to a `TypeChecker` lambda in
`jsonschema._types`, which cannot be pickled, so the parent process
reports `MaybeEncodingError`/`PicklingError` and the actual mismatch is
only visible in the worker's own output, hundreds of lines earlier in
the CI log. This raises a `RuntimeError` carrying the message instead,
so the failure the validator exits on is the failure it found.
<details>
<summary>Details</summary>
## Problem
`utils/req-res-log-validator.py:239` ends the schema-mismatch handler
with a bare `raise`. The pool is created at `:334` and `process_file` is
the worker body from `:186`.
Reproduced standalone with the pinned `jsonschema==4.17.3`:
```
pickle of jsonschema.ValidationError: PicklingError: Can't pickle <function <lambda> at 0x7f6d8c90add0>: attribute lookup <lambda> on jsonschema._types failed
real message the worker saw: None is not of type 'string'
bare `raise` (upstream today): parent saw MaybeEncodingError: Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object at 0x7f6d8c73b450>'. Reason: 'PicklingError("Can't pickle <function <lambda> at 0x7f6d8c90add0>: attribute lookup <lambda> on jsonschema._types failed")'
RuntimeError (this patch) : parent saw RuntimeError: JSON schema validation error on fake.log: None is not of type 'string'
```
This is not specific to any one command. Every reply-schema mismatch
presents the same way, which is why the `reply-schemas-validator` job's
failure output does not name the schema that failed. Contrast #2676,
where the worker raised a `ValueError` and the parent reported it
verbatim.
It surfaced while triaging a Daily failure where `XPENDING` on an empty
PEL was rejected by a stale schema (fixed separately in #4653). The
schema was the bug; the pickling error was what made it expensive to
find.
## Scope
Only the `jsonschema` handler at `:239` is changed. The two handlers at
`:210` and `:214` also re-raise from the worker, but they carry
`json.decoder.JSONDecodeError` and whatever `Request`/`Response`
construction throws, which pickle fine, so they are left alone rather
than widened speculatively.
## Alternative rejected
Catching `MaybeEncodingError` in the parent and reformatting it would
keep the worker code unchanged, but the parent has no access to the
original exception's fields by then, only the pickling failure's text.
The information has to be flattened before it crosses the process
boundary.
</details>
This was generated by AI but verified, with love, by a human.
---------
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
arshidkv12
pushed a commit
to arshidkv12/valkey
that referenced
this pull request
Sep 16, 2026
…#4703) `process_file` runs inside a `multiprocessing.Pool` worker, and on a reply schema mismatch it re-raised the `jsonschema` exception. That exception holds a reference to a `TypeChecker` lambda in `jsonschema._types`, which cannot be pickled, so the parent process reports `MaybeEncodingError`/`PicklingError` and the actual mismatch is only visible in the worker's own output, hundreds of lines earlier in the CI log. This raises a `RuntimeError` carrying the message instead, so the failure the validator exits on is the failure it found. <details> <summary>Details</summary> ## Problem `utils/req-res-log-validator.py:239` ends the schema-mismatch handler with a bare `raise`. The pool is created at `:334` and `process_file` is the worker body from `:186`. Reproduced standalone with the pinned `jsonschema==4.17.3`: ``` pickle of jsonschema.ValidationError: PicklingError: Can't pickle <function <lambda> at 0x7f6d8c90add0>: attribute lookup <lambda> on jsonschema._types failed real message the worker saw: None is not of type 'string' bare `raise` (upstream today): parent saw MaybeEncodingError: Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object at 0x7f6d8c73b450>'. Reason: 'PicklingError("Can't pickle <function <lambda> at 0x7f6d8c90add0>: attribute lookup <lambda> on jsonschema._types failed")' RuntimeError (this patch) : parent saw RuntimeError: JSON schema validation error on fake.log: None is not of type 'string' ``` This is not specific to any one command. Every reply-schema mismatch presents the same way, which is why the `reply-schemas-validator` job's failure output does not name the schema that failed. Contrast valkey-io#2676, where the worker raised a `ValueError` and the parent reported it verbatim. It surfaced while triaging a Daily failure where `XPENDING` on an empty PEL was rejected by a stale schema (fixed separately in valkey-io#4653). The schema was the bug; the pickling error was what made it expensive to find. ## Scope Only the `jsonschema` handler at `:239` is changed. The two handlers at `:210` and `:214` also re-raise from the worker, but they carry `json.decoder.JSONDecodeError` and whatever `Request`/`Response` construction throws, which pickle fine, so they are left alone rather than widened speculatively. ## Alternative rejected Catching `MaybeEncodingError` in the parent and reformatting it would keep the worker code unchanged, but the parent has no access to the original exception's fields by then, only the pickling failure's text. The information has to be flattened before it crosses the process boundary. </details> This was generated by AI but verified, with love, by a human. --------- Signed-off-by: Madelyn Olson <madelyneolson@gmail.com> Signed-off-by: arshidkv12 <arshidkv12@gmail.com>
arshidkv12
added a commit
to arshidkv12/valkey
that referenced
this pull request
Sep 16, 2026
Signed-off-by: arshidkv12 <arshidkv12@gmail.com> Bump minimum cmake version to 3.24 (valkey-io#4232) Currently the `src/CMakeLists.txt` is using [Link Features](https://cmake.org/cmake/help/latest/manual/cmake-generator-expressions.7.html#link-features) which was added in version 3.24. Increase `cmake_minimum_required` to 3.24 to reflect the actual minimum version required to run `cmake`. Signed-off-by: Bara' Hasheesh <bara.hasheesh@gmail.com> Signed-off-by: arshidkv12 <arshidkv12@gmail.com> Feature valkey path hash (valkey-io#4506) This document proposes adding **Path Hash**, a general-purpose, binary-safe data type backed by a radix tree, to Valkey. Each Valkey key stores one Path Hash, and each logical path maps to a `field -> value` map. The type provides exact lookup, longest-prefix matching, matching of all stored ancestor prefixes, atomic updates, and prefix traversal. It gains persistence and high availability through Valkey's existing RDB, AOF, primary-replica replication, and Cluster capabilities. The first target use case is a KV Cache placement index for AI inference: Worker KV Cache events continuously update which Workers contain which cached prefixes. A Router queries all reusable prefixes along a request's cumulative block-hash path and then combines the results with real-time load to select a Worker. The type itself does not interpret tokens, models, Workers, or cache tiers. Applications encode those semantics into binary-safe paths, fields, and values, so the type can also serve hierarchical routing, longest-prefix matching, autocomplete indexes, and other prefix-oriented workloads. Large-scale model inference services commonly reuse previously computed KV Cache through a prefix cache. When a request reaches a Router, the Router needs to know more than whether the prompt hits the cache. It also needs to know: 1. How deep the request's contiguous prefix matches on each Worker; 2. Whether the same prefix exists simultaneously on multiple Workers or cache tiers; 3. How placement changes continuously as blocks are created, migrated, evicted, and as Workers restart; 4. That multiple Routers must observe the same persistent and recoverable placement index; A regular Hash can store `block_hash -> workers`, but it cannot express "find all stored ancestor prefixes of this query" in a single query. Storing every prefix as a separate top-level key also inflates the key count, makes atomic updates across keys difficult, and introduces many network round trips. The ordering of a Sorted Set is likewise not equivalent to a byte-prefix relationship. A radix tree compresses paths by their shared byte prefixes, making it naturally suitable for longest-prefix matching and ancestor-prefix enumeration. Valkey already contains a mature internal `rax` implementation that supports path compression, binary-safe keys, values on internal nodes, insertion, deletion, and ordered iteration. Therefore, no new tree implementation dependency is required. - Provide a general-purpose, binary-safe prefix-index data type. - Query the longest prefix or all stored ancestor prefixes with one command. - Allow multiple fields on the same path to be updated independently. - Make writes idempotent and support atomic single-key batch application for event streams. - Use existing Valkey persistence, replication, failover, and Cluster semantics. - Reuse the path-compression and iteration capabilities of the current internal `rax`. - Provide stable and explicit response structures for RESP2 and RESP3 clients. - The type does not provide vector similarity or semantic-similarity matching; all matching is exact byte-prefix matching. - The type does not provide a global Path Hash spanning multiple Valkey keys. - The type does not replace the inference system's Worker snapshot, live-event, or event-reconciliation protocols. - The type does not guarantee synchronous replication or zero RPO; data reliability still follows the selected Valkey deployment model. - v1 does not provide a separate TTL for an individual path; TTL still applies to the top-level Valkey key. ```mermaid flowchart LR Client[Client] --> Router[Router] Router -->|prefix query| Valkey[Valkey Path Hash] Router -->|inference request| Worker[Inference Worker] Worker -->|KV events| Bridge[KV Event Bridge] Bridge -->|atomic batch| Valkey Worker -->|load report| Router ``` The data plane has two paths: - Update path: A Worker produces KV events such as block created, stored, and evicted. The Bridge converts each event into a complete path and writes it idempotently to Valkey through `PHSET` or batches updates to multiple paths through `PHMSET`. - Query path: The Router converts a prompt into a cumulative block-hash path, retrieves all matching prefixes and placements in one `PHPREFIXES` call, and then combines them with Worker load for routing. The Valkey Path Hash is a placement index shared by multiple Routers. The Bridge is responsible for protocol adaptation, event ordering, batch coalescing, retries, generation management, and short-term in-memory buffering rather than serving queries. Path Hash data is persisted through RDB, AOF, and primary-replica replication. The Bridge uses bounded in-memory buffering and retries for unacknowledged events. If it detects a sequence gap or buffer overflow, it should reconcile against a Worker snapshot before continuing to consume live events. A Path Hash placement index is generally rebuildable soft state. Losing an add usually produces a false miss and therefore reuses less cache. Losing a delete can produce a more dangerous stale positive. Applications should therefore include the Worker generation in the value and have the Router accept only the currently live generation. After a Worker restarts, an old generation will not participate in routing even if an old delete was lost; stale records can be reclaimed later by a background scan. This strategy reduces application-level risk during a failure window but does not change the RPO of Valkey's asynchronous replication itself. Each top-level Valkey key stores one tree: ```text Valkey key └── Path Hash ├── path P1 │ ├── field F1 -> value V1 │ └── field F2 -> value V2 ├── path P1 || P2 │ └── field F1 -> value V3 └── path P1 || P3 └── field F3 -> value V4 ``` Formally: ```text Tree: BinaryString -> Map<BinaryString, BinaryString> ``` - Top-level key: A regular Valkey key responsible for the namespace, Cluster slot, and tree-wide TTL. - path: A binary-safe byte string and the logical key in the Path Hash. - field: A binary-safe byte string identifying an owner or property that can be updated independently under the same path. - value: A binary-safe opaque value encoded by the application. An empty path is valid and represents the root payload. An internal navigation node is a "stored logical path" only when it has a non-empty field/value map. After its last field is deleted, the logical path disappears automatically, and `rax` can subsequently compress internal nodes that are no longer needed. The top-level Path Hash key may remain present with zero logical paths. Deleting the final path, or clearing the tree with `PHDELPREFIX key ""`, leaves an empty Path Hash object whose `PHCARD` is zero. This follows Stream semantics, where `XTRIM` may leave an empty stream; only `DEL` or `UNLINK` removes the top-level key. Empty Path Hash objects are preserved by RDB, AOF rewrite, replication, and `COPY`. If each path stored only one opaque value, updating placement for multiple Workers would require reading the entire value, modifying the list, and writing it back. Two Bridges or two event batches could then cause a lost update. A field/value map reduces the conflict granularity to an individual field: ```text path = cumulative-hash-path field = worker-a | hbm value = generation 7 | component mask | metadata ``` `worker-a` and `worker-b` can independently execute `PHSET` or `PHDEL` operations, and the commands are naturally idempotent. Let `p` and `q` be byte strings. `p` is a prefix of `q`, written `p ⪯ q`, if and only if there is a byte string `s` such that `q = p || s`. Matching does not interpret UTF-8, integers, tokens, or segments. For the set `S` of stored paths in the tree: ```text Exact(q) = q, provided q belongs to S Longest(q) = arg max |p|, where p belongs to S and p ⪯ q Prefixes(q) = all p in S satisfying p ⪯ q, ordered from shortest to longest ``` If an empty path is stored, it is the first match for every query. `PREFIXES` returns only ancestors of the query, not descendants for which the query is a prefix. The application first divides the prompt into blocks using a fixed page size. Suppose there are three blocks, `B1`, `B2`, and `B3`, and a chained cumulative hash is used: ```text C1 = Hash(B1) C2 = Hash(C1 || B2) C3 = Hash(C2 || B3) ``` Each cumulative hash is encoded as a fixed-width big-endian byte string. For example, a 64-bit hash uses 8 bytes: ```text P1 = BE64(C1) P2 = BE64(C1) || BE64(C2) P3 = BE64(C1) || BE64(C2) || BE64(C3) ``` The request query path is: ```text Q = BE64(C1) || BE64(C2) || BE64(C3) ``` The stored representation is therefore a "cumulative hash sequence." If `Hash(B1)`, `Hash(B2)`, and `Hash(B3)` were treated as three unrelated keys, they would have no ancestor relationship at the byte level, and the Path Hash could not derive the block order. The server does not enforce an 8-byte segment size. Fixed-width encoding is simply how the application ensures that no logical path ends in the middle of a hash. Suppose `page_size = 4` and the request contains 12 tokens: ```text B1 = token 1..4 B2 = token 5..8 B3 = token 9..12 ``` The placement tree has the following logical content: ```text P1 ├── worker-a|hbm -> generation=7, components=1111 └── worker-b|hbm -> generation=3, components=1111 P2 ├── worker-a|hbm -> generation=7, components=1111 └── worker-b|hbm -> generation=3, components=1111 P3 └── worker-b|hbm -> generation=3, components=1111 ``` The shared-prefix relationship in the tree is: ```mermaid flowchart TD Root[Root] P1[P1 cached by A and B] P2[P1 plus P2 cached by A and B] P3[P1 plus P2 plus P3 cached by B] Root --> P1 P1 --> P2 P2 --> P3 ``` The Router executes the following operation for `Q`. `LENGTHS` avoids repeatedly returning the shared bytes of `P1`, `P2`, and `P3`; the Router can divide the matched length by 8 to obtain the page depth directly: ```text PHPREFIXES kv:{model-id}:placement Q LENGTHS WITHVALUES COUNT 3 MAXLEN 24 ``` The response is conceptually equivalent to: ```text [ [8, [worker-a|hbm, value-a7, worker-b|hbm, value-b3]], [16, [worker-a|hbm, value-a7, worker-b|hbm, value-b3]], [24, [worker-b|hbm, value-b3]] ] ``` The Router aggregates by path length and obtains: ```text worker-a has a contiguous match of 2 pages = 8 tokens worker-b has a contiguous match of 3 pages = 12 tokens ``` It then combines those results with signals such as load, queue length, and cross-machine communication cost for routing. The largest cache hit does not necessarily determine the final Worker. This is also why the Router needs all ancestor prefixes rather than only a global `LONGEST` result. If the Router is considering only a set of healthy Workers, it can use `FIELDS` filtering to reduce the response size: ```text PHPREFIXES kv:{model-id}:placement Q LENGTHS FIELDS 2 worker-a|hbm worker-b|hbm COUNT 3 MAXLEN 24 ``` The Path Hash API accepts a complete path. It does not accept a `parent_hash` and then query the parent chain on the server. If a Worker event contains only the current block hash and parent hash, the Bridge has two options: 1. Maintain a short-lived `hash -> full path` cache and use it to construct the complete path; 2. Modify the event protocol so that the Worker carries either the cumulative hash-segment sequence or the fully encoded path directly. The second approach makes a stateless Bridge easier to implement, while the first reduces event size. This trade-off belongs to the inference event protocol and is not part of the general-purpose Path Hash data type. The `PH` command prefix stands for Path Hash. The public type name is Path Hash; the command group, ACL category, and `TYPE`/`SCAN TYPE` name use `path-hash`. The keyspace-notification configuration character remains `r`. In every command, `key` is the top-level Valkey key, while `path`, `query`, `field`, and `value` are binary-safe bulk strings. A command returns the standard `WRONGTYPE` error when an existing key has the wrong type. | Command | Syntax | Purpose | Time Complexity | |---|---|---|---| | `PHSET` | `PHSET key path [FNX \| FXX] FIELDS numfields field value [field value ...]` | Atomically set one or more fields under one path | `O(L + F·U)` | | `PHMSET` | `PHMSET key path FIELDS numfields field value [field value ...] [path FIELDS numfields field value ...]` | Atomically set fields grouped by path | `O(ΣL + M·U)` | | `PHGET` | `PHGET key path field [field ...]` | Read one or more fields by exact path | `O(L + F·U)` | | `PHMGET` | `PHMGET key path FIELDS numfields field [field ...] [path FIELDS numfields field ...]` | Read fields grouped by exact path | `O(ΣL + M·U)` | | `PHGETALL` | `PHGETALL key path` | Read all field/value pairs under a path | `O(L + O)` | | `PHEXISTS` | `PHEXISTS key path` | Test whether an exact logical path exists | `O(L)` | | `PHDEL` | `PHDEL key path [field [field ...]]` | Delete fields or an entire logical path | `O(L + F·U)` for specified fields; `O(L + S)` for the entire payload | | `PHLONGEST` | `PHLONGEST key query [LENGTH] [WITHVALUES \| FIELDS numfields field [field ...]]` | Return the longest stored prefix of a query | `O(L + O)` | | `PHPREFIXES` | `PHPREFIXES key query [LENGTHS] [WITHVALUES \| FIELDS numfields field [field ...]] [COUNT count] [MAXLEN max-path-bytes]` | Return all stored ancestor prefixes of a query | `O(L + O)` | | `PHDELPREFIX` | `PHDELPREFIX key prefix` | Delete all logical paths under a prefix | `O(L + Σ(Pᵢ + Fᵢ))` | | `PHSCAN` | `PHSCAN key cursor [PREFIX prefix] [COUNT count] [WITHVALUES]` | Incrementally traverse logical paths | `O(L + C + O)` | | `PHCARD` | `PHCARD key` | Return the number of logical paths | `O(1)` | Where: - `L` is the byte length of the input path or query; - `F` is the number of requested fields; - `S` is the number of stored fields in a payload removed in full; - `M` is the total number of fields across all path groups in a multi-path command; - `U` is the lookup cost within a node payload: linear for a small listpack and amortized constant time for a dict; - `O` is the number of bytes or fields actually returned; - `Pᵢ` is the byte length of the i-th path matched by a subtree deletion; - `Fᵢ` is the number of fields in that path's payload; - `C` is the number of paths examined by the current scan. ```text PHSET key path [FNX | FXX] FIELDS numfields field value [field value ...] ``` Atomically set one or more field/value pairs in the field map for `path`. `numfields` must be greater than zero and must equal the number of field/value pairs that follow. Without a condition, a missing tree or path is created automatically and existing fields are overwritten. - `FNX`: Apply the entire field/value group only if none of the specified fields exists under the path. - `FXX`: Apply the entire field/value group only if every specified field exists under the path. - `FNX` and `FXX` are mutually exclusive. - Conditions are evaluated against all specified fields before any write is performed. If any field fails the condition, none of the supplied field/value pairs is written. - If the key or path does not exist, `FNX` succeeds and creates it, while `FXX` fails without creating it. - If the same field is specified more than once, assignments are applied from left to right and the last value wins. Return: `OK` if the write succeeds; null if the condition is not satisfied. For example, the following command writes both fields only if neither `field-1` nor `field-2` exists under `path-a`. The path may already exist with other fields. If either specified field exists, neither field is modified: ```text PHSET tree path-a FNX FIELDS 2 field-1 value-1 field-2 value-2 ``` ```text PHMSET key path FIELDS numfields field value [field value ...] [path FIELDS numfields field value ...] ``` Atomically apply one or more path groups within one Path Hash. Each path is followed by `FIELDS`, a positive `numfields`, and exactly that many field/value pairs. Grouping reflects the underlying `path -> field/value map`, avoids repeating path bytes, and allows one path lookup per path group. Missing paths are created automatically and existing fields are overwritten. `PHMSET` does not support `FNX` or `FXX`; every assignment is unconditional. The complete grouped argument structure is validated before any mutation is applied. If the same `(path, field)` target appears more than once within or across groups, assignments are applied from left to right and the last value wins. Return: `OK` after all assignments have been applied. For example: ```text PHMSET tree path-a FIELDS 2 field-1 value-1 field-2 value-2 path-b FIELDS 1 field-3 value-3 ``` ```text PHGET key path field [field ...] ``` Perform one exact path lookup and read one or more fields from its payload. - When one field is requested, return its value as a bulk string, or null if the key, path, or field does not exist. - When multiple fields are requested, return an array of values in request order. A missing key or path produces null for every requested field, and each missing field produces null at its corresponding position. ```text PHMGET key path FIELDS numfields field [field ...] [path FIELDS numfields field ...] ``` Read one or more field groups from one Path Hash. Each path is followed by `FIELDS`, a positive `numfields`, and exactly that many fields. Each path lookup is exact and performed once per group; the command does not perform ancestor-prefix or descendant-prefix matching. The complete grouped argument structure is validated before reading results. Return one flat array whose elements correspond to all requested fields in path-group and field order. A missing key, path, or field produces null at the corresponding position. The command always returns an array, including when only one field is requested. Paths and fields do not need to be distinct, and duplicate requests are preserved. For example, table-cache rows stored under a `row` field can be fetched by primary key in one command: ```text PHMGET student:pk student-10001 FIELDS 2 name grade student-10002 FIELDS 1 name student-10003 FIELDS 1 name ``` ```text PHGETALL key path ``` Return all field/value pairs for the specified path. RESP2 returns a flat array: ```text [field-1, value-1, field-2, value-2, ...] ``` RESP3 returns a map: ```text {field-1: value-1, field-2: value-2, ...} ``` Field order is undefined in both protocols. If the path or key does not exist, return an empty array in RESP2 or an empty map in RESP3. ```text PHEXISTS key path ``` Perform an exact path lookup and return 1 if `path` is a stored logical path, or 0 if the key or path does not exist. Prefix ancestors and descendants do not count as an exact match. ```text PHDEL key path [field [field ...]] ``` - When fields are provided, delete only those fields and return the number of fields actually deleted. - When no field is provided, delete the entire payload for the path and return 1 or 0. - Deleting a path payload does not delete descendant paths. - After the last field is deleted, the path is automatically removed from the logical tree. - Deleting the final logical path leaves an empty Path Hash key. Use `DEL` or `UNLINK` to remove the top-level key. The command is an idempotent no-op for a target that does not exist. When fields are provided, the command takes `O(L + F·U)` time. When no field is provided, removing the complete payload synchronously releases all of its stored fields and takes `O(L + S)` time in the worst case. `PHDEL` is therefore an `@slow` command. ```text PHLONGEST key query [LENGTH] [WITHVALUES | FIELDS numfields field [field ...]] ``` Find the longest stored path that is a byte prefix of `query`. - By default, return the matching path; return null if there is no match. - `LENGTH`: Replace the matching path with its byte length, for cases where the caller already holds the query. - `WITHVALUES`: Return `[path, [field, value, ...]]` in RESP2 or `[path, {field: value, ...}]` in RESP3. Field order is undefined. - `FIELDS`: Read only the specified fields. The inner value array preserves request order and places null for a missing field. Return `[path, [value-or-null, ...]]` in both RESP2 and RESP3. - `WITHVALUES` and `FIELDS` are mutually exclusive. An empty path can be returned as a match if it is stored. ```text PHPREFIXES key query [LENGTHS] [WITHVALUES | FIELDS numfields field [field ...]] [COUNT count] [MAXLEN max-path-bytes] ``` Return all stored paths satisfying `path ⪯ query`, ordered from shortest to longest. - By default, return `[path-1, path-2, ...]`. - `LENGTHS`: Replace each matching path with its byte length to avoid repeating the shared bytes of a long query in the response. - `WITHVALUES`: Return `[[path-1, [field, value, ...]], ...]` in RESP2 or `[[path-1, {field: value, ...}], ...]` in RESP3. Field order within each payload is undefined. - `FIELDS`: Return `[[path-1, [value-or-null, ...]], ...]` in both RESP2 and RESP3, with each value array aligned to the requested field order and null for each missing field. - `COUNT`: Return at most `count` matches. This is a hard limit, not a hint. If more matches exist, retain the deepest `count` prefixes; return the selected results in ascending path-length order. - `MAXLEN`: Match only paths whose byte length does not exceed `max-path-bytes`. Apply this limit before `COUNT`. - Return an empty array when there is no match. `COUNT` must be an integer from `1` to `LONG_MAX`, and `MAXLEN` must be an integer from `0` to `LONG_MAX`, using the server build's `long` range (maximum `2147483647` on 32-bit builds or `9223372036854775807` on 64-bit builds). Invalid or out-of-range values return an error, even when the key does not exist. Omitting either option imposes no corresponding limit; `MAXLEN 0` permits only a stored empty root path. The command visits only ancestor nodes along the query path and does not scan the entire tree. For example, suppose the matching path lengths are `[8, 16, 24, 32]`: ```text PHPREFIXES key query LENGTHS COUNT 2 MAXLEN 24 ``` `MAXLEN 24` first produces `[8, 16, 24]`; `COUNT 2` then retains the two deepest matches, so the final result is `[16, 24]`. ```text PHDELPREFIX key prefix ``` Delete every logical path satisfying `prefix ⪯ path` and return the number of paths deleted. An empty prefix clears the entire tree but preserves the empty top-level Path Hash key. Use `DEL` or `UNLINK` to remove the key itself. This is an `@slow` command with time complexity `O(L + Σ(Pᵢ + Fᵢ))`: deleting each match copies and removes its complete path and synchronously releases every field in its payload. The implementation may collect matching paths in bounded-size chunks to avoid iterator invalidation and limit the number of temporary path copies. This chunking does not make the operation incremental: the complete matching subtree is deleted synchronously and atomically in one command invocation, without yielding between chunks. A large subtree can therefore cause high latency. ```text PHSCAN key cursor [PREFIX prefix] [COUNT count] [WITHVALUES] ``` Incrementally traverse paths in lexicographic order. The cursor is an opaque bulk string: pass `0` on the first call; the server returns `[next-cursor, entries]`, where `next-cursor = 0` means the traversal is complete. A nonzero cursor encodes the previous position and must not be interpreted by the client. - `PREFIX` restricts traversal to the specified subtree. - `COUNT` is a hint for the number of paths to examine in each call and does not guarantee an exact result count. - `WITHVALUES` returns each entry as `[path, [field, value, ...]]` in RESP2 or `[path, {field: value, ...}]` in RESP3. Field order within each payload is undefined. The outer response remains `[next-cursor, entries]` in both protocols. - As with Valkey `SCAN`, concurrent modifications can cause duplicates or omissions, and clients must process results idempotently. The implementation can encode a version number and the last returned path in the cursor, then use `raxSeek` to resume at the lexicographic position without retaining an iterator session on the server. ```text PHCARD key ``` Return the number of logical paths in the tree that carry a non-empty payload. Return 0 for either an empty Path Hash or a missing key; use `EXISTS` or `TYPE` when the caller needs to distinguish those states. ```c typedef struct RadixPayload { uint8_t encoding; /* listpack or dict */ uint32_t num_fields; void *data; } RadixPayload; typedef struct RadixObject { rax *index; /* path -> RadixPayload* */ uint64_t num_paths; uint64_t num_fields; } RadixObject; ``` ```text Valkey key └── RadixObject ├── rax index │ └── data pointer ─────────┐ └── counters │ ▼ RadixPayload └── listpack or dict └── field -> value ``` The `rax` data pointer points to a `RadixPayload`. `rax` already allows a key to terminate at an internal node that still has descendants, so `P1` and `P1 || P2` can both carry payloads. A small payload uses a listpack to reduce memory overhead when each block belongs to only a few Workers. After the entry or value threshold is exceeded, the payload is promoted one-way to a dict to provide near-`O(1)` field updates. It is not automatically demoted after deletion, avoiding encoding oscillation. - `raxNew` creates an empty tree; - `raxInsert` and `raxTryInsert` insert a path and payload pointer; - `raxFind` supports exact-path operations such as `PHGET` and `PHEXISTS`; - `raxRemove` removes a logical path after its payload becomes empty and recompresses compressible nodes; - `raxStart`, `raxSeek`, `raxNext`, and `raxPrev` support lexicographic scan, RDB save, and AOF rewrite; - `raxSize` provides the number of logical keys; - `raxAllocSize` can be included in `MEMORY USAGE`; - Path compression stores the shared bytes of many cumulative hash prefixes only once. Currently, `raxFind` returns data only when the query exactly matches a stored key. `raxSeek` provides lexicographic positioning but does not efficiently enumerate ancestors of the query. If a client calls `raxFind` separately for every byte prefix of the query, the worst-case complexity degrades from `O(L)` to `O(L²)`. Two helpers should be added at the internal `rax` layer, or a callback-based walk should be added and shared by both: ```c void *raxFindLongestPrefix( rax *rt, const unsigned char *query, size_t len, size_t *matched_len); int raxForEachPrefix( rax *rt, const unsigned char *query, size_t len, raxPrefixCallback callback, void *context); ``` The algorithm walks the query only once: ```text 1. Start at the head; if the root is a logical key, record length 0. 2. At a regular node, select the child for the next query byte. 3. At a compressed node, compare the entire compressed edge at once; stop immediately if any byte differs. 4. Whenever an iskey node is reached, record the number of bytes consumed and the data pointer. 5. LONGEST retains only the final record; PREFIXES invokes the callback for every record in order. 6. Stop when the query is exhausted or the required branch does not exist. ``` The time complexity is `O(L + K)`, where `K` is the number of logical paths actually matched; siblings and descendants are not scanned. The implementation must correctly handle a root key, a mismatch in the middle of a compressed edge, a key that is also an ancestor of another key, an empty query, and binary zero. The current `raxLowWalk` already contains most of the path-descent logic, but it is `static inline` and returns only the stopping node and split position. The implementation can extract a shared internal walker without changing the existing insertion semantics, or add a separate read-only prefix walker. The latter has a smaller change surface and is easier to validate with differential tests. `PHSCAN PREFIX p` can use `raxSeek(">=", p)` to locate the first candidate, then call `raxNext` until a key no longer starts with `p`. `PHDELPREFIX` cannot iterate and delete without accounting for iterator invalidation. The v1 implementation collects a bounded number of complete path copies, closes the iterator, removes those paths, and repeats until no match remains. The bound applies only to the number of temporary path copies. The outer loop still deletes the complete subtree synchronously and atomically in one event-loop turn, so chunking does not bound command latency. Its work includes copying and removing every matched path and releasing every field in the associated payloads. If truly nonblocking deletion of a large subtree is needed in the future, `rax` can gain a subtree-detach capability and hand detached nodes to the lazy-free thread. This is not required for v1. --------- Signed-off-by: bodong.ybd <bodong.ybd@alibaba-inc.com> Co-authored-by: Jacob Murphy <jkmurphy@google.com> Signed-off-by: arshidkv12 <arshidkv12@gmail.com> Report the real error when a reply schema fails validation (valkey-io#4703) `process_file` runs inside a `multiprocessing.Pool` worker, and on a reply schema mismatch it re-raised the `jsonschema` exception. That exception holds a reference to a `TypeChecker` lambda in `jsonschema._types`, which cannot be pickled, so the parent process reports `MaybeEncodingError`/`PicklingError` and the actual mismatch is only visible in the worker's own output, hundreds of lines earlier in the CI log. This raises a `RuntimeError` carrying the message instead, so the failure the validator exits on is the failure it found. <details> <summary>Details</summary> `utils/req-res-log-validator.py:239` ends the schema-mismatch handler with a bare `raise`. The pool is created at `:334` and `process_file` is the worker body from `:186`. Reproduced standalone with the pinned `jsonschema==4.17.3`: ``` pickle of jsonschema.ValidationError: PicklingError: Can't pickle <function <lambda> at 0x7f6d8c90add0>: attribute lookup <lambda> on jsonschema._types failed real message the worker saw: None is not of type 'string' bare `raise` (upstream today): parent saw MaybeEncodingError: Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object at 0x7f6d8c73b450>'. Reason: 'PicklingError("Can't pickle <function <lambda> at 0x7f6d8c90add0>: attribute lookup <lambda> on jsonschema._types failed")' RuntimeError (this patch) : parent saw RuntimeError: JSON schema validation error on fake.log: None is not of type 'string' ``` This is not specific to any one command. Every reply-schema mismatch presents the same way, which is why the `reply-schemas-validator` job's failure output does not name the schema that failed. Contrast valkey-io#2676, where the worker raised a `ValueError` and the parent reported it verbatim. It surfaced while triaging a Daily failure where `XPENDING` on an empty PEL was rejected by a stale schema (fixed separately in valkey-io#4653). The schema was the bug; the pickling error was what made it expensive to find. Only the `jsonschema` handler at `:239` is changed. The two handlers at `:210` and `:214` also re-raise from the worker, but they carry `json.decoder.JSONDecodeError` and whatever `Request`/`Response` construction throws, which pickle fine, so they are left alone rather than widened speculatively. Catching `MaybeEncodingError` in the parent and reformatting it would keep the worker code unchanged, but the parent has no access to the original exception's fields by then, only the pickling failure's text. The information has to be flattened before it crosses the process boundary. </details> This was generated by AI but verified, with love, by a human. --------- Signed-off-by: Madelyn Olson <madelyneolson@gmail.com> Signed-off-by: arshidkv12 <arshidkv12@gmail.com> Support filtering ACL LIST by user and role Signed-off-by: arshidkv12 <arshidkv12@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When a consumer group has no pending messages,
XPENDING <key> <group>returns:The reply schema only described non-empty summaries and extended replies, so the reply-schema validator rejected this valid response.
This adds a dedicated schema variant for the empty summary while preserving strict validation for non-empty responses. There is no change to command behavior.
This started surfacing consistently after #4629 added stream tests that call
XPENDINGafter clearing the pending entries list.