Skip to content

Delete a session's data from Context Intelligence (server side) - #97

Draft
Diego Colombo (colombod) wants to merge 13 commits into
mainfrom
feat/session-data-delete
Draft

Delete a session's data from Context Intelligence (server side)#97
Diego Colombo (colombod) wants to merge 13 commits into
mainfrom
feat/session-data-delete

Conversation

@colombod

@colombod Diego Colombo (colombod) commented Sep 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

What this adds

Until now a user who contributed data to Context Intelligence had no way to delete it. This adds that ability on the server side.

Deleting a session removes the whole session graph — the root session and every session below it (its subsessions and forks) — together with the data those sessions point to: their blobs, their queue files, and their dead-letter records.

Three endpoints

  • GET /sessions/{session_id}/summary?workspace=<ws> (needs read access): reports what deleting this session's data would remove, without deleting anything.
  • DELETE /sessions/{session_id}?workspace=<ws>&apply=<bool> (needs write access): apply=false (default) returns the same preview and deletes nothing; apply=true deletes it and reports the counts.
  • GET /whoami (needs read access): a "who am I" read used by the delete bundle's agent to resolve the acting user for ownership warnings -- it returns {"contributor_id": "..."}, the same identity the server already stamps onto created_by / requested_by. null when auth is disabled, never a 500.

How delete behaves

  • Passing any session id deletes the whole graph it belongs to. Pass a subsession id and the server finds that graph's root first, then removes the whole graph. There is no single-subsession delete.
  • Nodes shared with other sessions (for example an agent several sessions used) are kept; only their links into the deleted graph are removed.
  • Deleting is permanent.
  • It is a two-step flow: preview first, then apply. Every applied delete is written to the server log.
  • The delete is refused with 409 if any session in the graph is still receiving data (its queue has records not yet fully written). The summary's started-at / last-change fields show when a session last changed — a change less than a minute ago may mean it is still live.

How it is built

  • Each delete lives on the storage class that owns the data: the blob store deletes blobs, the graph store deletes graph nodes, the queue manager deletes queue files. A small deletion service composes them; the routes stay thin (they only read the request, build the stores, call the service, and return JSON). No route or service touches Neo4j or the filesystem directly.
  • A read always uses the read-only Neo4j connection; only a real delete uses the admin connection.
  • A graph's blobs are found through the blob store (which is keyed by session id and lists every blob for a session), not by scanning graph nodes. This is both the right place for it and correct in real use, where a blob reference is stored nested inside a node's data field.

What is proven

  • Unit tests for each storage operation and the deletion service.
  • Real-Neo4j tests: whole graph removed, shared nodes survive, an unrelated graph reached only through a shared node is untouched, a still-receiving-data graph is refused.
  • A real-server end-to-end test: it posts real events (which the server turns into real blob files), then over HTTP calls the summary and the delete, and checks the graph, blobs, and queue files are gone while shared and unrelated data survive, and that the summary's blob count matches the real blobs on disk.

This reuses the approach proven earlier by the one-off session-purge work (find the graph, delete it in two phases, keep shared concept nodes).

Version

Server version bumped 6.7.1 -> 6.8.0 -> 6.9.0 (6.9.0 for the new GET /whoami read endpoint).

Status

Opening as a draft to share with the team and start review. The bundle-side user experience (finding sessions, confirming, deleting from the agent) is a separate follow-up and is not part of this PR.

@colombod @microsoft-amplifier
feat(blob): add session-scoped delete_session to BlobStore
d00daa5 
Add delete_session(session_id) -> int to the BlobStore Protocol and
AsyncDiskBlobStore, removing <root>/<session_id>/ and returning the
number of blobs removed. Idempotent for a session with no blobs.

At the storage Protocol level so the deletion service composes it
rather than touching the filesystem directly; shape stays compatible
with a later rebase onto the storage-refactor BlobStore.delete().

Part of context-intelligence session data delete (server, A1).

🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier)

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
@colombod @microsoft-amplifier
feat(graph): whole-family session-summary resolver (A-SUM)
b92df3d 
Add resolve_session_family(session_id) at the GraphStore Protocol level
(neo4j_store impl + in-memory GraphState equivalent). Given any session
id it expands up to the root and returns the whole family: session-id
set, distinct $blob_ref URIs across all family nodes, node/edge counts,
created_by, started_at, last_change, subsession count, workspace.
Shared :SST_CONCEPT nodes bound the traversal so a family's counts never
leak through a shared concept.

Add BlobStore.size(uri) at the Protocol level; services.total_blob_size
composes whole-family blob size via it. Shared component the delete
service reuses so preview and apply cannot disagree.

Part of context-intelligence session data delete (server, A-SUM).

🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier)

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
@colombod @microsoft-amplifier
feat(graph): surface working_dir in session-summary (post-rebase on #94)
954e112 
#94 now records working_dir on the Session node; the summary resolver
reads it from the root and returns it instead of always None. Covers
both the Neo4j and in-memory GraphState paths; neo4j test asserts it
surfaces from the root.

🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier)

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
@colombod @microsoft-amplifier
feat(graph): whole session-graph DETACH DELETE (A2) + rename family->…
99ddbf2 
…session_graph

A2: delete_session_graph(session_id) at the GraphStore Protocol level
(neo4j_store impl + in-memory GraphState parity). Reuses the A-SUM
resolution, partitions owned nodes vs shared :SST_CONCEPT boundary nodes,
and DETACH DELETEs the owned nodes by elementId in batches. Shared
concept nodes survive; only their edges into the deleted graph are
removed. Fails loud if the post-delete gate (owned-gone / concept-
survives) does not hold. Returns node/relationship delete counts.

Also renames the A-SUM-introduced 'session family' vocabulary to
'session graph' to match the approved doc ('the entire session graph,
root + all descendants'): SessionFamily->SessionGraph,
resolve_session_family->resolve_session_graph, and all internal
constants/vars/tests. One concept, one name.

Proven on real isolated Neo4j: tests/neo4j/test_delete_session_graph.py
7 passed (owned removed, shared concept survives, unrelated graph reached
only via shared concept untouched, sub-session id == root id, counts
match, unknown-id no-op); resolve+delete 16 neo4j passed; non-neo4j
suite green; pyright clean.

Part of context-intelligence session data delete (server, A2).

🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier)

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
@colombod @microsoft-amplifier
feat(queue): session queue delete + pending_count on QueueManager (A3)
84b4242 
Add two methods on QueueManager (the sole owner of queue-artifact I/O):
- pending_count(session_id): authoritative drained/pending check,
  reusing the exact _read_committed_offset / _complete_data_end /
  _count_newlines helpers recover() uses; 0 == fully drained.
- delete_session(session_id): removes .log/.offset AND .dead.jsonl
  (a data delete must drop dead letters too, unlike delete_drained).
  Computes pending inside the key file-lock and raises rather than
  delete when anything is still pending -- fail loud, no partial
  delete, no TOCTOU against a concurrent append. Idempotent.

All queue filesystem access stays inside queue_manager.py; the deletion
service will call these via the QueueManager surface, never touch files.

Proven: tests/test_queue_manager.py 101 passed (pending reflects
uncommitted, zero after drain, ignores torn line; delete removes all
three artifacts incl dead letters, refuses on pending, idempotent on
missing, dead-letter-only session deletable); non-neo4j suite 2006
passed; pyright clean.

Part of context-intelligence session data delete (server, A3).

🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier)

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
@colombod @microsoft-amplifier
feat(deletion): DeletionService composing graph/blob/queue (A4)
781befd 
New context_intelligence_server/deletion.py: DeletionService(graph_store,
blob_store, queue_manager) composing ONLY the storage Protocol APIs -- no
Neo4j, no filesystem (grep-verified). Two operations:
- preview(session_id): dry, mutates nothing; returns whole-graph facts
  (reusing resolve_session_graph) plus deletable + pending_sessions.
- apply(session_id, requested_by): captures session_ids/blob_refs first,
  refuses (raises, deletes nothing) if ANY session in the graph has
  pending queue records, then deletes graph -> blobs (every session) ->
  queue (every session), and logs the applied deletion. preview and apply
  share one drain check so they cannot disagree.

Blob-count reconciliation (blobs_deleted vs whole-graph blob_refs) is a
WARNING, not a failure: orphan/already-reclaimed blobs make an exact
match non-guaranteed, and raising after an irreversible delete would
report failure on a completed op.

Proven: tests/test_deletion.py 8 passed (dry no-mutation, pending refusal,
whole-graph delete across every session, reconcile-mismatch warns not
fails, logging); tests/neo4j/test_deletion_service.py 4 passed on real
Neo4j (whole graph gone, shared concept + unrelated graph survive,
pending refusal); non-neo4j suite green; pyright clean.

Part of context-intelligence session data delete (server, A4).

🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier)

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
@colombod @microsoft-amplifier
feat(api): add session summary and delete routes (A5)
a256a1c 
New router context_intelligence_server/routers/deletion.py, added to the
app. Two routes, both a thin layer over DeletionService:

- GET /sessions/{session_id}/summary (needs read access): reports what
  deleting this session's data would do, without deleting anything. Uses
  the read-only Neo4j connection.
- DELETE /sessions/{session_id} (needs write access): with apply=false
  (the default) it returns the same preview and deletes nothing, using the
  read-only connection; with apply=true it deletes the whole session graph,
  its blobs, and its queue data, using the admin connection.

A read always uses the read-only connection and a change always uses the
admin connection: the summary route and the delete route's dry run both
read, so both use the read-only connection; only a real delete uses the
admin connection.

Returns 404 for an unknown session and 409 when a session in the graph is
still receiving data (not yet drained). The caller's id is recorded on the
delete. The routes hold no delete or graph logic of their own.

Proven: tests/routers/test_deletion.py 9 passed (summary fields, 404,
read-only caller allowed on summary, dry run previews and deletes nothing,
apply returns the result, caller id passed through, unknown session 404,
still-receiving-data 409, write access required for delete); non-neo4j
suite 2023 passed; pyright clean.

Part of context-intelligence session data delete (server, A5).

🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier)

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
@colombod @microsoft-amplifier
fix: find a session graph's blobs through the blob store, not the graph
f223ec4 
The summary and delete worked out a graph's blobs by scanning graph node
properties for a $blob_ref marker. That was wrong in two ways: it is the
blob store's job, not the graph's, and in real use the marker is stored
nested inside a node's data field where the scan never found it, so the
blob count came back 0 for real sessions.

The blob store is keyed by session id and already lists every blob for a
session. So the graph now only supplies the set of session ids, and the
DeletionService asks the blob store for the blobs of each session and adds
them up. This is correct no matter where the graph keeps its markers.

- Remove SessionGraph.blob_refs, the extract_blob_refs helper, and the
  marker constant; resolve_session_graph no longer reads node properties
  for blobs (graph and in-memory paths).
- preview counts blobs by listing them from the blob store per session;
  apply deletes them the same way. Removed the old reconcile-and-warn step,
  since there is no separate graph blob count to compare against anymore.

Proven end to end: tests/integration/test_delete_session_endpoint.py now
posts real events that the server offloads to real blob files, then checks
the summary's blob count is greater than 0 and matches the real blobs on
disk. Fast set 1914 passed; neo4j resolve+delete 12 passed; end-to-end
1 passed; pyright clean.

Part of context-intelligence session data delete (server).

🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier)

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
@colombod @microsoft-amplifier
docs+release: document the delete endpoints and bump to 6.8.0
0a9aef1 
Add the two new routes to the API reference and a plain-language section
explaining how deleting a session's data works (whole graph, shared nodes
kept, preview then apply, 409 while still receiving data, permanent). Note
in Data Persistence that a delete clears the graph, blobs, and queue files
together.

Bump the server version 6.7.1 -> 6.8.0 for the new delete feature (single
source in pyproject.toml; the version endpoint and package version read it).

Part of context-intelligence session data delete (server, release).

🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier)

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
@colombod @microsoft-amplifier
refactor(api): resolve by session id only; drop workspace input and a…
893aaa6 
…pply flag

Review of the draft PR: a session id is the unique identifier, so the
server now resolves everything from it and neither route takes a workspace
or an apply flag.

- The graph store looks up which workspace a session id belongs to (match
  by node_id alone), then scopes the resolve and the delete by that
  discovered workspace. Blobs and the queue were already keyed by session
  id only. If a session id is found in more than one workspace the server
  refuses rather than guessing (raises, turned into HTTP 409).
- GET /sessions/{session_id}/summary is the preview; DELETE
  /sessions/{session_id} always deletes. No apply flag: preview by GET,
  then DELETE when sure. A session can be deleted once it is no longer
  receiving data; a session still receiving data is refused with 409.
- README updated: no workspace or apply parameters; states plainly that a
  session that is no longer receiving data can be deleted, and 409 is
  returned only while it is still receiving.

Proven: fast set 1917 passed; neo4j resolve+delete 13 passed (incl. a
same-id-in-two-workspaces case that now raises); real-server end-to-end
1 passed; pyright clean.

Part of context-intelligence session data delete (server, API review).

🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier)

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
@colombod @microsoft-amplifier
feat(api): add GET /whoami read endpoint for caller identity
7c34fdc 
The delete bundle's agent needs to resolve "who is the acting user" so it
can compare that against a session's created_by before acting on it. The
server already knows this identity -- it stamps every delete with a
requested_by extracted from the bearer token during auth -- but there was
no way for a client to ask the server "who am I" directly.

Add GET /whoami, gated by the same require_read dependency the summary
route uses. It returns {"contributor_id": "..."}, reading the exact same
contributor_id the auth middleware (auth.py) already writes to
request.scope["state"] for every authenticated request -- the same value
routers/deletion.py's _caller_id reads to stamp a delete's requested_by,
and routers/admin.py's _admin_who reads for audit logging. No new identity
source is introduced.

When auth is disabled (allow_unauthenticated, no credential configured),
contributor_id is null rather than a 500 -- same response shape, anonymous
value.

Bump version 6.8.0 -> 6.9.0 for the new read endpoint (matches this repo's
convention of a version bump per API addition).

Additive only: no delete/summary behavior changed.

Part of context-intelligence session data delete (server, PR #97).

🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier)

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
@colombod Diego Colombo (colombod) mentioned this pull request Sep 2, 2026
Draft
@colombod

Diego Colombo (colombod) commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator Author

Bundle-side (Phase B) counterpart: microsoft/amplifier-bundle-context-intelligence#108 (draft). That PR adds the server-data-ops delete agent that consumes this server's session-summary + delete surface.

@colombod @microsoft-amplifier
feat(delete): 409 on undrained graph carries a machine-readable retry…
ce09dce 
… hint

A delete refused because the session graph still has undrained queue records is a
transient, retryable condition -- but the 409 carried only a human message, so a
caller could not know WHEN to retry and had to poll by hand.

- New typed SessionsPendingError(root_id, pending_sessions) raised by
  DeletionService.apply() for the pending case (was a bare RuntimeError).
- Router maps it to 409 WITH a Retry-After header and a structured body
  (reason=sessions_pending, pending_sessions, retry_after_seconds). The
  ambiguous-id 409 stays non-retryable (no Retry-After); a graph-vanished race
  stays a plain 409.
- retry-after value is configurable: Settings.delete_retry_after_seconds (default 2s).
- Tests: service raises the typed error with pending_sessions; router emits
  Retry-After + structured body for pending, and no Retry-After for the
  vanished-race 409.

🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier)

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
@colombod @microsoft-amplifier
docs(readme): note the delete 409 carries a Retry-After hint for a st…
8c577d6 
…ill-draining session

Document that the retryable 'still receiving data' 409 now carries Retry-After +
retry_after_seconds + pending_sessions, and that the ambiguous-id 409 does not
(not retryable).

🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier)

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@colombod