fix(recycle): prune expired Admin/_Recycle requests with their reports (#6813)

[rbuergi]

Closes #6813

What

Admin/_Recycle/{id} requests and their {request}/{process} reports were never deleted. Once every NodeType definition recycle issues one (#6810), they grow without bound.

• Declared retention: ActivationRecycleRetention, read from configuration and registered as a mesh-scoped singleton by AddActivationRecycle:
  • ActivationRecycle:Retention:Enabled, default true
  • ActivationRecycle:Retention:MaxAge, default 30.00:00:00; a configured value below 1 day is clamped to 1 day
  • ActivationRecycle:Retention:MaxDeletionsPerRun, default 50
• Pruned by the existing write-time pass, with no new timer: right after ActivationRecycle.Request creates a new request, it calls ActivationRecycle.Prune. Prune lists the oldest requests and deletes the expired ones as System through IMeshService.DeleteNode. The delete cascades, so each request's reports go with it. This is the same owner-prunes-on-write shape CompileActivityRetention uses. The rows grow only through that write, so pruning on it is enough to bound them.
• The request just written is never pruned. The selection is fail-closed: a row that is undated, is not a request, or is not directly under Admin/_Recycle is kept. A failed listing or delete is logged and retried by the next request's prune. The caller's ticket does not depend on the prune.
• Why 30 days is the default, and the config table: Doc/Architecture/StaleStateUntilRecycle → "Recycling a type".

No new config-key allow file applies in core.

Evidence

ActivationRecycleRetentionTest (MeshWeaver.Graph.Test) has 4 new tests. The full ~ActivationRecycle filter gives total: 17, failed: 0, succeeded: 17.

• ANewRequest_PrunesAnExpiredRequestWithItsReports_AndKeepsItself: with a 1 ms window, the old request and its report are gone and the new request remains.
• ARequestInsideTheWindow_SurvivesThePrune_WithItsReport
• SelectForPruning_TakesExpiredRequestsOldestFirst_AndFailsClosed (pure)
• FromConfiguration_ClampsTheWindow_AndIgnoresMalformedValues

Negative control: with Prune disabled (if (!retention.Enabled || true) return 0), the run gives total: 4, failed: 1: ANewRequest_PrunesAnExpiredRequestWithItsReports_AndKeepsItself fails with "Expected the observable to emit a value within 36s because the old request is past the window and the new request's prune removes it, but it did not." After restoring the code, the run gives 17/17.

dotnet build -c Release -warnaserror reports 0 errors and 0 warnings for MeshWeaver.Graph, MeshWeaver.Graph.Test and MeshWeaver.Documentation.

Pairs-with: none — additive only (a new public type and a new public method; nothing is removed or changed)

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

Cleanup delays request acceptance, and nonpositive retention values bypass the documented minimum-age clamp.

2 open findings

• Pruning delays successful request acceptance · New
• Zero and negative spans incorrectly use the default age · New

What changed in this PR

Adds configurable retention for activation-recycle requests and their reports, addressing audit growth as NodeType recycling becomes more frequent.

Changes:

• Adds an age-based retention policy and deletion cap.
• Triggers cascading cleanup after new requests.
• Adds retention tests and configuration documentation.

File	Description
test/​MeshWeaver.Graph.Test/​ActivationRecycleRetentionTest.cs	Tests pruning, report retention, selection, and configuration.
src/​MeshWeaver.Graph/​Configuration/​ActivationRecycleRetention.cs	Defines retention configuration and candidate selection.
src/​MeshWeaver.Graph/​Configuration/​ActivationRecycle.cs	Registers retention and invokes cleanup after creation.
src/​MeshWeaver.Documentation/​Data/​Architecture/​StaleStateUntilRecycle.md	Documents retention settings and cleanup behavior.

🧠 Review effort: Balanced

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

Test Results

    15 files   -    11      15 suites   - 11   50m 59s ⏱️ - 8m 58s
11 128 tests  - 1 597  11 128 ✅  - 1 404  0 💤  - 193  0 ❌ ±0 
11 144 runs   - 1 597  11 144 ✅  - 1 404  0 💤  - 193  0 ❌ ±0 

Results for commit 4ddbbba. ± Comparison against base commit cab0ae9.

This pull request removes 1636 and adds 13 tests. Note that renamed tests count towards both.

   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---)
   --- End of inner exception stack trace ---, expected: True)
   --- End of inner exception stack trace ---, isDenial: True)
 ---> (Inner Exception #1) MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.<---
 ---> (Inner Exception #1) MeshWeaver.Messaging.Hub.Test.InfrastructureFaultTest+ProviderException (0x80004005): Failed to connect to 10.42.18.4:5432<---
 ---> (Inner Exception #1) System.ArgumentException: Value does not fall within the expected range.<---
 ---> (Inner Exception #1) System.InvalidOperationException: boom<---
 ---> (Inner Exception #1) System.InvalidOperationException: source B is misconfigured<---
…

Memex.Portal.Shared.Test.InstanceIdRulesMatchTheRegistryTest ‑ TheSetupHostAgreesWithTheRegistry(candidate: "1c7c64ef-020b-43b7-b6c3-7c8ebaa335a5")
Memex.Portal.Shared.Test.PublishStorageFaultIsNotAConflictTest ‑ AStorageFault_IsA500_NamingTheError(fault: System.AggregateException: One or more errors occurred. (Access to the path '/data/modules/activation.d/MeshWeaver.Markdown.Export.json' is denied.)
 ---> System.UnauthorizedAccessException: Access to the path '/data/modules/activation.d/MeshWeaver.Markdown.Export.json' is denied.
   --- End of inner exception stack trace ---)
Memex.Portal.Shared.Test.PublishStorageFaultIsNotAConflictTest ‑ AStorageFault_IsA500_NamingTheError(fault: System.InvalidOperationException: landing faulted
 ---> System.UnauthorizedAccessException: Access to the path '/data/modules/activation.d/MeshWeaver.Markdown.Export.json' is denied.
   --- End of inner exception stack trace ---)
Memex.Portal.Shared.Test.SessionDenialIsAnAnswerTest ‑ OnlyAVerdictReadsAsADenial(shape: "the same verdict nested, as a late denial dispatch"···, failure: System.InvalidOperationException: write failed
 ---> System.UnauthorizedAccessException: Access denied
   --- End of inner exception stack trace ---, isDenial: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall as an aggregate's FIRST member", fault: System.AggregateException: One or more errors occurred. (Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.)
 ---> MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
   --- End of inner exception stack trace ---, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall as an aggregate's SECOND member", fault: System.AggregateException: One or more errors occurred. (The operation has timed out.) (Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.)
 ---> System.TimeoutException: The operation has timed out.
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.<---
, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall nested in an aggregate inside a wrapper", fault: System.InvalidOperationException: outer
 ---> System.AggregateException: One or more errors occurred. (Value does not fall within the expected range.) (Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.)
 ---> System.ArgumentException: Value does not fall within the expected range.
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.<---

   --- End of inner exception stack trace ---, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall wrapped as InnerException", fault: System.InvalidOperationException: outer
 ---> MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
   --- End of inner exception stack trace ---, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "an aggregate of unrelated faults", fault: System.AggregateException: One or more errors occurred. (The operation has timed out.) (Value does not fall within the expected range.)
 ---> System.TimeoutException: The operation has timed out.
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) System.ArgumentException: Value does not fall within the expected range.<---
, expected: False)
MeshWeaver.Graph.Test.ActivationRecycleRetentionTest ‑ ANewRequest_PrunesAnExpiredRequestWithItsReports_AndKeepsItself
…

♻️ This comment has been updated with latest results.

[Copilot]

🟢 Approval recommended

Both previously reported defects are addressed, and no unresolved blocking findings remain.

0 open findings

2 resolved since last review

• Zero and negative spans incorrectly use the default age
• Pruning delays successful request acceptance

🧠 Review effort: Balanced