router: add multi-channel distribution backend for N-node AF2 writes#5
Open
riaarora-boop wants to merge 1 commit into
Open
router: add multi-channel distribution backend for N-node AF2 writes#5riaarora-boop wants to merge 1 commit into
riaarora-boop wants to merge 1 commit into
Conversation
Adds Router, a backend.Backend implementation that distributes S3 object writes across N AF2 channels (one per node) using a stable placement hash: WAL segment names (24 hex chars) are placed by their low-64-bit ordinal mod N so consecutive segments round-robin across slots; everything else uses fnv1a(bucket+"/"+key) mod N. Placement is deterministic — reads recompute the same slot with no location map. Writes to the owning node go through the local backend under a per-channel weight-1 semaphore (P=1). Writes to non-owning nodes are forwarded via an s3proxy leg with an optional deadline so a partitioned peer fails fast. CreateBucket/DeleteBucket are fanned out cluster-wide. Includes OwnerMap (shared JSON config, one selfIdx difference per node) and placement unit tests. Jira: CDM-557246 Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
Adds
Router, abackend.Backendimplementation that distributes S3 object writes across N AF2 channels (one per node).Placement: WAL segment names (24 hex chars) are placed by their low-64-bit ordinal
mod Nso consecutive segments round-robin across slots. Everything else usesfnv1a(bucket+"/"+key) mod N. Placement is deterministic — reads recompute the same slot with no per-object location map.Writes: owner node writes go through the local backend under a per-channel weight-1 semaphore (P=1 — only one byte-write in flight into a channel at any moment, keeping it out of the wedge). Non-owner requests are forwarded via an
s3proxyleg with an optional deadline so a partitioned peer fails fast.Bucket ops:
CreateBucket/DeleteBucketfan out cluster-wide so one create/delete from any node propagates to all.Note — v1 limitation (will be updated with JFL control plane, §14 of design doc): currently uses a single static
OwnerMaploaded at startup. Will be replaced with per-grant routing ((access_key_id, bucket)→OtterChannelGrant) once the JFLOtterControlThrift push + CRDB persistence is implemented.Test Plan:
go test ./backend/router/...passes (placement unit tests: WAL ordinal round-robin, fnv fallback, determinism)JIRA Issues:
CDM-557246 (subtask of CDM-557238)
Revert Plan:
Revert this PR. The Router is only used by the
ottersubcommand (PR 6); reverting either PR is sufficient to remove it from the binary.