DOC-6832 Add go-redis automatic pipelining page - #3621
Open
andy-stark-redis wants to merge 9 commits into
Open
Conversation
Document the experimental AutoPipeline() and AsyncAutoPipeline() APIs
(blocking and async faces), AutoPipelineConfig options, cluster slot
sharding, and caveats, based on go-redis PR #3867. Cross-link from the
Pipelines/transactions page.
Written against an unmerged draft PR, so two things are deliberately
provisional: the version note reads "v9.XX.0" as a placeholder because
no release ships the API yet, and the examples are static go blocks
rather than tested clients-example doctests because the TCE example set
cannot exist until a release lands. Both resolve at pickup, not now.
The config field list and defaults are also draft-derived and may move.
Learned: page docs an unreleased draft API; version and examples are placeholders to resolve at pickup
Constraint: the "v9.XX.0" version note is a placeholder; do not invent a real version until the release that ships #3867 is known
Directive: convert the static go code blocks to {{< clients-example >}} doctests once go-redis ships the API and an example set exists
Recheck: when go-redis PR #3867 merges and a release ships the AutoPipeline API
Ticket: DOC-6832
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
andy-stark-redis
added
do not merge yet
parked
Contributor
Contributor
go-redis PR #3942 (the master-bound umbrella that superseded #3867) merged on 2026-08-03 and shipped in v9.22.0 the same day, so the park trigger is met. Reconcile the page against the released source rather than the draft it was written from: - Version note, v9.XX.0 placeholder becomes v9.22.0. - AutoPipelineConfig becomes AutoPipelineOptions; AutoPipeline(nil) becomes a no-arg AutoPipeline() plus AutoPipelineWithOptions(), same for the async face. Configuration also shown via Options.AutoPipelineOptions. - Config fields, dropping MaxRetries (gone from the type; batch retries are governed by the client's own MaxRetries) and PipelinePoolSize (a client option, and only live when a pipeline buffer size is set), adding MaxBatchBytes and AdaptiveDelay, and recording the real defaults (MaxBatchSize 200/300, MaxConcurrentBatches 1, NumShards 0, MaxFlushDelay 0). - Widened scope, since Ring exposes all four methods and failover clients get them via *Client. - Cluster, noting the several-slot-routed-shards default and that multi-node commands are rejected from batches. - Caveats, since Do/DoRaw/DoRawWriteTo run outside the pipeline, and retries are capped by the client MaxRetries and disabled with -1. - bannerText, still experimental at release, so the banner stays but "not yet released" goes. Predicted-versus-actual, closing the park loop. The prose survived almost whole; every identifier in it moved. The page's architecture (two faces, blocking versus windowed, cluster slot sharding, the four caveats) needed no structural change, and the cluster ordering and context/blocking/Do caveats all read correctly against the released code. What churned was exclusively naming and field-level detail, and the park snapshot's two flagged placeholders were the cheapest items to fix. The one thing the snapshot got wrong in substance rather than in naming was PipelinePoolSize: it was recorded as "moved to the client Options", implying it still belonged on the page, but autopipeline.go never references the pipeline pool at all and the pool is only created when a pipeline buffer size is set, so the honest fix was to demote it to a tuning aside rather than relocate it as an autopipeline knob. Also worth recording, the trigger fired within 44 minutes of the merge (merged 16:59Z, v9.22.0 published 17:43Z), so "merged" and "released" were effectively the same event here; the tag still had to be checked separately, because the preceding tag was a beta the version note must not cite. The static go blocks stay static: go-redis has no autopipeline doctest upstream, so no TCE example set can exist yet. Learned: park snapshot got the page's shape and caveats right and every identifier wrong; incremental re-verification during the park kept the final reconcile cheap Constraint: batch retries are governed by the client's MaxRetries, not an autopipeline option; AutoPipelineOptions has no retry field Rejected: relocating PipelinePoolSize as an autopipeline config knob | autopipeline.go never touches the pipeline pool, which is only created when a pipeline buffer size is set Directive: keep the code blocks static go until go-redis ships an autopipeline doctest; a TCE example set cannot be authored from the docs side alone Gaps: the examples are not run-verified against a live server, and the throughput figures in the release notes were not reproduced Recheck: if automatic pipelining leaves experimental status, drop the bannerText and the note Ticket: DOC-6832 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
🧠 Redis MemoryFound 5 related items from repository history:
Memory updated at 6051201 |
andy-stark-redis
removed
do not merge yet
parked
andy-stark-redis
changed the title
Promote the cross-cutting lesson from unparking this PR into the skill that can act on it, rather than leaving it in a commit nobody re-reads. Three additions, each from something that actually bit on DOC-6832. Step 1 now says a trigger must name the source that will really ship the feature (the draft #3867 was one of a series that landed as the consolidated #3942, so a trigger watching the draft would have watched the wrong PR) and must resolve "released" as an ancestry check rather than by dates (merge and the v9.22.0 release were 44 minutes apart, and the preceding tag was a beta the version note must not cite). Step 3 now asks for the observed shape to be split into semantics and identifiers, because they decayed at completely different rates here, plus the two traps that cost the most on the reconcile: a mis-attributed field is indistinguishable from a renamed one unless you record where you saw it, and the release notes under-reported which client types expose the API where the source was authoritative. Limits records the measured outcome so a future author reads a moved identifier as normal rather than as evidence the page was written too early. Learned: the identifier/semantics split is the load-bearing distinction in a park snapshot; recording where an identifier was seen is what separates a rename from a mis-attribution Directive: keep the manifest's required-section format in _shared/park-manifest.md; if the identifier-list structure needs to become a hard requirement, add it there and point at it from here Ticket: DOC-6832 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
🧠 Redis MemoryFound 5 related items from repository history:
Memory updated at 6ca3d05 |
Three review points from Andy. Drop the upstream "face" terminology. go-redis' own source and release notes call the blocking and deferred APIs "the two faces", and writing the page from that source pulled the word in throughout, but it is that project's internal jargon rather than Go vocabulary a reader would recognize. The page now talks about two methods, with the section retitled "Blocking and asynchronous pipelining" and "Windowed usage" becoming "Asynchronous usage" to match. Where the text described a per-API default or cached instance, it now names the method it means, which is more precise than "face" was: the cached instances are per-API, not per-method, so that sentence says so explicitly. Remove the note shortcode that restated the bannerText. The banner already carries the experimental warning, so the note only added the version requirement, which now sits in the intro prose instead. Say "Sentinel" when mentioning NewFailoverClient. The bare word "failover" reads as the cross-region failover feature currently in preview for other clients; this is the long-standing Sentinel client (sentinel.go, "uses Redis Sentinel for automatic failover"), and go-redis has no geo-failover API at all. Naming Sentinel removes the ambiguity rather than dropping the sentence, since the scope statement is accurate and useful. Learned: NewFailoverClient is Sentinel failover and long released, unrelated to the previewed cross-region failover on the Lettuce and redis-py failover pages Directive: do not reintroduce "face" for the two autopipelining APIs; it is go-redis' internal jargon, not reader-facing Go terminology Constraint: the bannerText carries the experimental warning on this page; do not add a note shortcode that repeats it Ticket: DOC-6832 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
🧠 Redis MemoryFound 5 related items from repository history:
Memory updated at 5cd5fdd |
Includes Andy's own edits: the Sentinel failover sentence is gone (still
supported, but a legacy feature we aren't emphasizing), the intro is reworded,
the version requirement is its own paragraph, and "drain the results" became
"read the results afterwards".
Continuing the same jargon sweep, "a window of commands" becomes "a sequence of
commands" in both places it appeared. "Sequence" rather than "batch", the other
candidate, because this page already uses "batch" two dozen times for the thing
the engine flushes, and the asynchronous section's point is precisely that the
caller submits commands while the engine decides how to batch them; reusing the
word would blur that.
Two more carried-over words found by sweeping the page against the upstream
source rather than waiting to be told again. "is deferred and" is dropped from
the asynchronous bullet: it was upstream's predicate for that API, and with the
bullet already labelled Asynchronous and the next sentence explaining that
calls return immediately, it was redundant as well as jargon. "queue-and-flusher
shards" becomes "command queues, or shards, that the engine flushes separately",
since "flusher" names an internal goroutine that nobody tuning NumShards needs
to know about, while keeping the word "shards" tied to the field name.
Left alone deliberately: "coalesces" and "soft threshold rather than a hard cap"
also come from the upstream source, but both are ordinary technical English that
a reader parses without knowing go-redis internals, which is the line this sweep
is drawing.
Learned: the jargon a source-derived page inherits is not only coined metaphors like "face" but ordinary-looking nouns ("window", "flusher") that silently name upstream internals
Directive: sweep a source-derived page against the upstream vocabulary before review, not after; two rounds of this were reader-caught
Ticket: DOC-6832
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
🧠 Redis MemoryFound 5 related items from repository history:
Memory updated at e5b7424 |
Andy asked why one field had two default values. It does not, and the table row said so misleadingly: "Defaults to 200, or 300 when AutoPipeline() falls back to its built-in default" implied a per-method default when the truth is a precedence chain. newAutoPipeliner coerces any MaxBatchSize <= 0 to 200 regardless of which method built the autopipeliner, so 200 is the engine's only default. The 300 exists solely inside DefaultBlockingAutoPipelineOptions, which is consulted only when no options are supplied anywhere: explicit options beat Options.AutoPipelineOptions, which beats the per-method preset. The consequence the old wording hid is that supplying AutoPipelineOptions at all, even an empty struct, drops the blocking target from 300 back to 200 for an unset MaxBatchSize -- a silent change from a seemingly unrelated edit. The row now states the single default, 200, and a following paragraph explains the preset and the precedence. Prose is the right home because MaxBatchSize is the only field where the two presets differ at all: both set MaxConcurrentBatches to 1 and both leave MaxFlushDelay at 0, so this is one field's quirk rather than a general per-method-defaults story that a whole column of the table would need to carry. Not documented, because upstream does not say it: there is no stated rationale for 300 over 200. DefaultBlockingAutoPipelineOptions explains at length why the blocking preset keeps MaxConcurrentBatches at 1, but never ties the batch target to that reasoning, so the page describes the value without inventing a why. Learned: MaxBatchSize 200 is the engine default and 300 is a no-options-only preset, so supplying any AutoPipelineOptions silently drops the blocking target to 200 Constraint: do not restate 300 as a per-method default for MaxBatchSize; it applies only when no options are set anywhere Gaps: upstream gives no rationale for the 300 preset value; the page deliberately does not offer one Ticket: DOC-6832 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
🧠 Redis MemoryFound 5 related items from repository history:
Memory updated at 510e238 |
Andy suggested the singleton pattern as a framing readers of this page already know, with a link to the Wikipedia article. It fits exactly: each client keeps one blocking and one asynchronous autopipeliner, created lazily on first use and shared thereafter, which is what the old "cached separately, and each is shared by all of its callers" wording was groping towards. Naming the pattern also let a real trap be stated plainly instead of implied. The old text said "the first call's options win", which is true but leaves the reader to work out the consequence. Verified in the source: AutoPipeline() and AutoPipelineWithOptions() both resolve through &c.autopipeliner, so they are two entry points to one singleton rather than one shared and one bespoke instance. Calling AutoPipelineWithOptions() when the singleton already exists therefore returns the existing instance and silently discards the options you passed. The page now says so, and points out that closing is consequently how you apply different options, since the next call builds a fresh one. The Wikipedia link follows established practice here: 407 content pages already link to Wikipedia, and the italic-on-first-use form matches existing entries such as [*idempotent*]. Learned: AutoPipeline() and AutoPipelineWithOptions() share one cached instance per client, so the WithOptions variant silently ignores its options once the singleton exists Constraint: describe the blocking and asynchronous autopipeliners as two singletons per client; the WithOptions variants are entry points to them, not separate instances Ticket: DOC-6832 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
🧠 Redis MemoryFound 5 related items from repository history:
Memory updated at c867787 |
Ticket: DOC-6832 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
🧠 Redis MemoryFound 5 related items from repository history:
Memory updated at b2e883a |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a new Automatic pipelining page to the go-redis client docs, covering the
experimental
AutoPipeline()/AsyncAutoPipeline()APIs (blocking and deferred faces),their
WithOptionsvariants,AutoPipelineOptionsand its defaults, cluster slot sharding,and caveats. Cross-linked from the Pipelines and transactions page.
Unparked 2026-08-04. The page was written against the unmerged upstream draft
redis/go-redis#3867 and parked until the feature
shipped. The draft series was consolidated into the master-bound umbrella
redis/go-redis#3942, which merged on 2026-08-03
and shipped in go-redis v9.22.0 the same day. The page has been reconciled against the
released v9.22.0 source:
v9.22.0(was av9.XX.0placeholder).AutoPipelineConfig→AutoPipelineOptions;AutoPipeline(nil)→ no-argAutoPipeline()plus
AutoPipelineWithOptions(), and the same for the deferred face. Configuration viaOptions.AutoPipelineOptionsis now shown too.MaxRetriesdropped (gone from the type — batch retries aregoverned by the client's own
MaxRetries),PipelinePoolSizedemoted to a tuning aside (itis a client option, and the autopipeliner never touches the pipeline pool),
MaxBatchBytesand
AdaptiveDelayadded, and the real defaults recorded — including unpickingMaxBatchSize, where 200 and 300 are a precedence chain rather than two per-methoddefaults (the 300 preset applies only when no options are supplied anywhere).
Ringexposes all four methods, and failover clients get them via*Client.rejected from batches.
bannerTextkeeps the experimental warning but drops "not yet released"; thenoteshortcode that restated it is gone, with the version requirement moved into the intro.
faces" for the blocking and asynchronous methods, "a window of commands" (now "a
sequence"), "drain the results", and "queue-and-flusher shards".
emphasizing.
The code blocks remain static Go rather than tested
clients-exampledoctests, becausego-redis ships no autopipeline doctest for the examples pipeline to pull from.
Also records the cross-cutting lesson from this unpark in
.claude/skills/park/SKILL.md: apark snapshot's semantics and its identifiers decay at different rates, a trigger must watch
the source that will actually ship the feature and resolve "released" as an ancestry check
rather than by dates, and an identifier's recorded location is what distinguishes a rename
from a mis-attribution.
🤖 Generated with Claude Code
Note
Low Risk
Documentation and internal skill updates only; no runtime or security-sensitive code changes.
Overview
Adds a new Automatic pipelining page under go-redis client docs, reconciled against v9.22.0 after unparking from upstream go-redis work. It documents blocking
AutoPipeline()vs asynchronousAsyncAutoPipeline(), theWithOptionsvariants,AutoPipelineOptions(includingMaxBatchSize200 vs 300 precedence), singleton behavior, cluster slot routing, and operational caveats. The page keeps an experimentalbannerTextand uses static Go samples rather thanclients-exampledoctests.Pipelines and transactions gains a short cross-link to the new page.
.claude/skills/park/SKILL.mdis extended with lessons from this unpark: testable triggers should track the shipping source and “released” via tag ancestry; park snapshots should split semantics from identifiers (with location notes to tell rename from mis-attribution); and expectations that identifier lists go stale while page structure often survives.Reviewed by Cursor Bugbot for commit b2e883a. Bugbot is set up for automated code reviews on this repo. Configure here.