refactor(storage): backend-neutral blob/queue/identity storage — swap backends with zero consumer changes - #81
Closed
Diego Colombo (colombod) wants to merge 1 commit into
Closed
Conversation
Diego Colombo (colombod)
changed the title
Diego Colombo (colombod)
changed the title
Diego Colombo (colombod)
force-pushed
the
storage-api-on-78
branch
from
August 26, 2026 10:43
e53acb7 to
e8954df
Compare
This was referenced Aug 26, 2026
…ackend-neutral protocols Transplant PR #76's storage-API isolation onto PR #78's hardened durable queue so the two stack cleanly (78 -> 76). Every store is now reached only through a backend-neutral Protocol + factory; no consumer constructs a concrete backend or touches an on-disk path. queue_manager/ FileSystemQueueManager implements the QueueManager Protocol. Built on #78's authoritative body (the durable per-record cursor model): the class body is byte-identical to #78's queue_manager.py apart from the class rename and its self- references. protocol.py carries #78's Record/Batch verbatim; factory.create_queue_manager is the only queue backend selector. blob_store/ FileSystemBlobStore implements the BlobStore Protocol. write() returns a BlobReference (uri + size + last_modified); the store gains scan()/list() (async BlobReference iterators) and a fenced delete(uri, if_unmodified=ref). Adds settings.blob_backend. identity_store/ FileSystemIdentityStore implements the IdentityStore Protocol; the commit-order and fail-closed-load contract lives in the protocol. The backing path is private -- callers use exists(). Also folds in the blob-key fix: process_event includes tool_call_id in the blob-key node_id, so two same-millisecond parallel events no longer collide on one blob and silently overwrite each other. Consumers (registry, main, blob_processor, pipeline) build stores via the factories and pass URIs/references, never paths. The boot-reclaim dry-run log derives the blob path from the QueueManager's own queues_dir (fixing a latent mismatch when it differs from settings.queues_path); the identity first-boot warnings no longer echo the store path. The sole remaining config-path read outside the storage layer is registry.queues_dir_path -- the resolver the WriterLease boot detector uses precisely because it must not construct a QueueManager. Full non-neo4j suite: 2082 passed. Neo4j subsets (queue durability, blob ingest, identity auth): green. 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
Diego Colombo (colombod)
force-pushed
the
storage-api-on-78
branch
from
August 26, 2026 17:06
e8954df to
301e7ef
Compare
Diego Colombo (colombod)
marked this pull request as draft
August 27, 2026 15:29
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this delivers
Storage becomes pluggable. Blob, queue, and identity persistence each move behind a backend-neutral
Protocol+ factory, so a new backend (Azure, S3, a database) can be dropped in by adding one implementation, with zero changes to any consumer. Today's filesystem behaviour is preserved exactly — the abstraction is what's new.Per package
write() -> BlobReference(callers hold an opaque reference, never a path); reads viascan()/list(); fenceddelete(uri, if_unmodified=ref)that refuses if the blob changed since it was observed, closing a TOCTOU delete race.settings.blob_backendselects the backend; consumers use thecreate_blob_store(settings)factory.Protocoldefines theRecord/Batchtypes;create_queue_manager()factory.exists()contract;create_identity_store(settings, kind)factory.Why it matters
Consumer code no longer knows how storage works — only the protocol. This is the seam that lets the deployment run on cloud object storage without rewriting the server, and it's the foundation the operability & data-quality work stacked on top (#79) builds on.
Supporting changes
tool_call_idis included in blob-key node_id construction.qm.queues_dir(fixes a latent mismatch withsettings.queues_path); three identity first-boot warnings no longer echo store paths.registry.queues_dir_path -> Path(get_settings().queues_path)— the WriterLease boot detector needs a resolver that does not construct a QueueManager.Verification
Full suite: 2082 non-neo4j + 86 neo4j = 2168 passed, 0 failed. All storage-artifact operations are isolated to the three backend packages — no
Path/glob/unlink/settings.*_pathreads in consumers.Stack
Merge after #78, before #79.
🤖 Generated with Amplifier
Co-Authored-By: Amplifier 240397093+microsoft-amplifier@users.noreply.github.com