[daily-ci] REAL-DEFECT: XPENDING empty-PEL reply schema failed reply-schemas-validator; validator masks the error as a PicklingError

[madolson]

The reply-schemas-validator job failed once in the Daily runs because the XPENDING <key> <group> reply schema in src/commands/xpending.json did not describe the reply the server actually sends when the consumer group's pending entries list is empty. src/t_stream.c:3069-3072 returns [0, nil, nil, nil], but the schema required elements 1 and 2 to be ID strings and element 3 to be an array, so utils/req-res-log-validator.py rejected a valid reply and exited 1. The root cause is already fixed upstream by valkey-io/valkey#4653, and the failing run just predates it by 18 minutes.

Filing anyway for a second defect that is still unfixed: when a schema mismatch happens, the validator re-raises a jsonschema.ValidationError out of a multiprocessing.Pool worker, that exception cannot be pickled, and the parent reports MaybeEncodingError: ... PicklingError instead of the real error. That is why automated fingerprinting found no signature here, and every future reply-schema mismatch will present the same misleading way.

Details

Occurrence

Run 34545972977, job 103098549810: https://github.com/valkey-io/valkey/actions/runs/34545972977/job/103098549810

Failing step is validator, .github/workflows/daily.yml:2176-2177. The job is defined at daily.yml:2125 with timeout-minutes: 1440, so timeout is not a factor. All three test steps (test, module api test, sentinel tests) passed. The job died in post-processing of the .reqres logs.

The actual error, 650 lines before the ##[error]

JSON schema validation error on /home/runner/work/valkey/valkey/tests/tmp/server.10276.39/stdout.reqres: [0, None, None, None] is not valid under any of the given schemas
argv: ['XPENDING', 'testxadstream', 'testxadgrp1']

Traceback (most recent call last):
  File "/home/runner/work/valkey/valkey/./utils/req-res-log-validator.py", line 229, in process_file
    jsonschema.validate(instance=res.json, schema=req.schema, cls=schema_validator)
  File "/home/runner/.local/lib/python3.12/site-packages/jsonschema/validators.py", line 1121, in validate
    raise error
jsonschema.exceptions.ValidationError: [0, None, None, None] is not valid under any of the given schemas

Failed validating 'oneOf' in schema:
...
On instance:
    [0, None, None, None]

What the job actually reported at the tail

Traceback (most recent call last):
  File "/home/runner/work/valkey/valkey/./utils/req-res-log-validator.py", line 337, in <module>
    for result in pool.map(func, paths):
  File "/usr/lib/python3.12/multiprocessing/pool.py", line 367, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/lib/python3.12/multiprocessing/pool.py", line 774, in get
    raise self._value
multiprocessing.pool.MaybeEncodingError: Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object at 0x7f0275781010>'. Reason: 'PicklingError("Can't pickle <function <lambda> at 0x7f0276898b80>: attribute lookup <lambda> on jsonschema._types failed")'
##[error]Process completed with exit code 1.

utils/req-res-log-validator.py:230-239 catches the ValidationError, prints diagnostics, then re-raises it inside a pool worker. jsonschema.ValidationError holds a reference to a TypeChecker lambda in jsonschema._types, which is unpicklable, so the parent never sees the real exception. Suggested fix at line 239: sys.exit(1), or raise a plain RuntimeError(str(err)), instead of re-raising the jsonschema exception. This is still present at upstream/unstable.

Why the reply is valid

src/t_stream.c:3064-3072:

    if (justinfo) {
        addReplyArrayLen(c, 4);
        /* Total number of messages in the PEL. */
        addReplyLongLong(c, raxSize(group->pel));
        /* First and last IDs. */
        if (raxSize(group->pel) == 0) {
            addReplyNull(c);      /* Start. */
            addReplyNull(c);      /* End. */
            addReplyNullArray(c); /* Clients. */

The pre-fix src/commands/xpending.json had a two-branch oneOf, extended form and summary form, and the summary branch declared items[1]/items[2] as type: string with pattern: [0-9]+-[0-9]+ and items[3] as type: array. Nothing accepted nulls.

Why only 1 of 7 Daily runs

Deterministic, in a narrow window, not intermittent.

d46565268 "Implement XACKDEL and XDELEX command (#4629)", 2026-09-10, added the first tests under --log-req-res that call XPENDING <key> <group> on an empty PEL, at tests/unit/type/stream.tcl:777 and :799:

        assert_equal {} [lindex [r XPENDING testxadstream testxadgrp1] 1]

98f915212 "Fix XPENDING reply schema for empty pending lists (#4653)", 2026-09-11 00:01 -0700, added a third oneOf branch for the empty summary.

The failing run's head commit is 95e0e9ac7 (2026-09-10 13:32 -0700) and the run was created 2026-09-11T00:19Z. git merge-base --is-ancestor 98f915212 95e0e9ac7 confirms the fix is not in that tree: the Daily started 18 minutes after the fix merged but built a commit from before it. The other six Daily runs fall outside the window, either before #4629 or after #4653.

Ruled out

Timeout (1440 minutes, job ran to completion). Test failure (no [err] or [exception]; all runtest steps passed). Dependency drift (jsonschema==4.17.3 pinned in utils/req-res-validator/requirements.txt, installed cleanly). Build failure (make SERVER_CFLAGS='-Werror -DLOG_REQ_RES' succeeded). Runner or network flake.

Upstream

valkey-io/valkey#4653 fixes the root cause and I agree with its diagnosis and its added oneOf branch; its description already names #4629 as the trigger. No upstream issue covers the validator's error masking.

This was generated by AI but verified, with love, by a human.

[madolson]

Diagnosis holds on both counts. Reproduced the pickling mask standalone with the pinned jsonschema==4.17.3: the ValidationError holds a jsonschema._types lambda, so the parent reports MaybeEncodingError/PicklingError and the real mismatch is only visible in the worker's own stdout. agents/unstable is 66 commits behind and lacks valkey-io/valkey#4653, so the schema gap reproduces there too once a test calls the two-argument form on an empty PEL. Fix in #62.

No upstream issue covers the validator masking. valkey-io/valkey#2676 is the closest test-failure record for this job, but its worker raised a picklable ValueError so the parent reported it verbatim, which is the opposite case.

This was generated by AI but verified, with love, by a human.