fix(compile): a release request answered by a delivery hold records why (#6832)

[rbuergi]

Refs #6832

What was wrong

The records in the issue (read on the company instance: Google/Account is StaleAdopted, adopted module 1.3.1 over current 1.5, requestedReleaseBy: system-security, requestedReleaseAt == lastReleaseRequestHandledAt, no compilationError, and no _GitSync among the partition's children) show what happened:

1. The release watcher found the record dirty (StaleAdopted clears CompiledSources). It stamped the request as handled and flipped the record to Pending.
2. The compile watcher's on-demand adoption declined the stale bundle (fingerprint mismatch). A forced request (the shape recycle stamps) skips this step, as Force-release is silently discarded: the compile watcher re-adopts the prebuilt bundle instead of compiling, so there is no verb that forces a build of the live source #2818 intends.
3. DispatchOrPark then reached the untracked-module gate (Bundles are baked for ONE framework identity per run, so a portal's GitSynced sources stay days behind main (or its types go unadopted) — options 1-4 undecided #3583). The type is module content and its partition tracks no source on that mesh, so the gate does not compile it. The type still had a usable build, so SettleAsHold kept that build serving as StaleAdopted.
4. BuildDeliveryHold.Settle cleared CompilationError because the held record is Ok. That field was the only place the gate's reason went. The handled request therefore produced no build, and nothing on the record said why.

Settled with code, per the brief:

• A forced release skips adoption on every path, including reactivation after a dispose and a record that is already StaleAdopted (NodeTypeCompilationHelpers, the RequestedReleaseForce branch before SeedForTypes). It then meets the delivery gates. NodeTypeCompilation.md documents that "RequestedReleaseForce does NOT bypass" the untracked-module gate.
• An unforced release over a known-stale adoption compiles the live source wherever the partition tracks a source. This is how the public instance compiled the same five types at ~09:56Z. On an untracked partition it is held, as documented.
• "Handled" is stamped with a dispatch. The Pending flip happens on the same write. The missing piece was the outcome's reason, not the attempt.

The fix

• New operational member NodeTypeDefinition.DeliveryHoldReason. It is masked by the sync seams (NodeTypeOperationalContent.MemberNames) and mirrored on NodeTypeCompileState.
• Every hold writes it: the three BuildDeliveryHold.Settle branches and the RequirePrebuilt branches of ApplyAdoptedSourceStamp. It names the refusing gate, both module versions and fingerprints, and the fix.
• ApplyCompileSuccess, a fresh seeder adoption and the adoption stamp clear it.
• The declined-bundle wording of UntrackedPartitionParkReason no longer claims the files are the stale side. In this incident the bundle was the older side.
• Doc: Doc/Architecture/NodeTypeCompilation, section "A module's content this mesh does not TRACK…", gains a "the hold says why" paragraph.

Test (deterministic, real mesh, real GitHub tracking seam)

AHeldReleaseRequestSaysWhyTest.AReleaseRequestAnsweredByAHold_RecordsWhy_AndACompileClearsIt has three arms:

• Forced request (the recycle's shape) on an untracked module type that is StaleAdopted: the type is held, and DeliveryHoldReason names the gate, the partition, the fix and 1.3.1.
• Unforced request over a dirty record (an install's shape): same result.
• Control: a forced request in a tracked partition compiles (Compiled, a new MVID) and the reason is null.

It sits beside UntrackedModuleContentNeverCompilesHereTest, which uses the same gate scaffolding, in MeshWeaver.Compiler.Pipeline.Test.

Negative control: removing only DeliveryHoldReason = ServingNotice(pending, reason) from BuildDeliveryHold.Settle makes the test fail:

failed MeshWeaver.Compiler.Pipeline.Test.AHeldReleaseRequestSaysWhyTest.AReleaseRequestAnsweredByAHold_RecordsWhy_AndACompileClearsIt
  Expected value not to be null or empty because a forced release (the recycle verb): a handled release request that produced no build must record WHY on the record (#6832).

With the line restored it passes: passed (2s 958ms) total: 1 failed: 0.

Related suites are green locally (Release, -warnaserror builds, 0 Error(s)):

• UntrackedModuleContent, BuildDeliveryHold, AdoptedBuildProvenance, StaleAdoption*, AdoptionIsNotJudged*: 97/97
• NodeTypeCompileState: 5/5
• Graph.Test NodeTypeOperationalContent, NodeTypeMemberOwnership, ShippedNodeTypeState: 18/18

What this does NOT do

It does not make those types serve their 1.5 code on the company instance. That needs one of the following, and neither is a core code change:

• OWNER ACT: publish a bundle of the current source for framework c003e001 for these packages. The second report shows Personal 1.1.1's only bake cancelled at the 45-minute cap.
• OWNER ACT: add a sync source for those partitions on that instance.
• DECISION for the maintainer: should a forced release (the operator's recycle) compile an untracked partition whose source an install of a published package version wrote? Today the documented answer is no.

Recycle post-roll

Recycle these addresses so their next handled request writes the reason (no new build is expected until a bundle lands): AppleMusic/Music, AppleWeather/Weather, Google/Account, HomeAssistant/Home, ICloud/Calendar, Personal/PersonalApp.

Pairs-with: none — only adds a member to a record; no public type or member removed.
Mirror-sync: none — no i18n catalog change.

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

Diagnostic accuracy, JSON-safe test reads, and coverage of clearing an existing hold reason need correction.

5 open findings

• Prebuilt adoption notices omit the blocking compile gate reason · New
• Declined bundle version is not tracked in stale decline metadata · New
• Raw CLR payload check rejects valid JSON-shaped definitions · New
• Completion predicate and return reject JSON-shaped node definitions · New
• Control does not verify clearing an existing delivery hold reason · New

What changed in this PR

Records why a NodeType release request produces a delivery hold rather than a new build, preserving the explanation even when the previous build remains usable.

Changes:

• Adds operational DeliveryHoldReason state.
• Writes hold explanations and clears them after compilation or adoption.
• Adds regression coverage and documents the diagnostic.

File	Description
test/​MeshWeaver.Compiler.Pipeline.Test/​UntrackedModuleContentNeverCompilesHereTest.cs	Corrects assertion explanation text.
test/​MeshWeaver.Compiler.Pipeline.Test/​AHeldReleaseRequestSaysWhyTest.cs	Tests held requests and tracked compilation.
src/​MeshWeaver.Mesh.Contract/​NodeTypeOperationalContent.cs	Masks hold reasons from authored sync content.
src/​MeshWeaver.Graph.Contract/​NodeTypeDefinition.cs	Declares the hold-reason field.
src/​MeshWeaver.Documentation/​Data/​Architecture/​NodeTypeCompilation.md	Documents hold diagnostics and remedies.
src/​MeshWeaver.Compiler.Pipeline/​PrebuiltAssemblySeeder.cs	Revises refusal wording and clears prior reasons.
src/​MeshWeaver.Compiler.Pipeline/​NodeTypeCompileState.cs	Mirrors operational hold state.
src/​MeshWeaver.Compiler.Pipeline/​NodeTypeCompilationHelpers.cs	Maintains reasons across adoption and compilation.
src/​MeshWeaver.Compiler.Pipeline/​BuildDeliveryHold.cs	Records reasons when settling holds.

🧠 Review effort: Balanced

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

Test Results

    20 files   -   6      20 suites   - 6   1h 0m 42s ⏱️ +45s
12 401 tests  - 324  12 401 ✅  - 131  0 💤  - 193  0 ❌ ±0 
12 417 runs   - 324  12 417 ✅  - 131  0 💤  - 193  0 ❌ ±0 

Results for commit cfa0c36. ± Comparison against base commit cab0ae9.

This pull request removes 369 and adds 19 tests. Note that renamed tests count towards both.

   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---)
   --- End of inner exception stack trace ---, expected: True)
   --- End of inner exception stack trace ---, isDenial: True)
 ---> (Inner Exception #1) MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.<---
 ---> (Inner Exception #1) MeshWeaver.Messaging.Hub.Test.InfrastructureFaultTest+ProviderException (0x80004005): Failed to connect to 10.42.18.4:5432<---
 ---> (Inner Exception #1) System.ArgumentException: Value does not fall within the expected range.<---
 ---> (Inner Exception #1) System.InvalidOperationException: boom<---
 ---> (Inner Exception #1) System.InvalidOperationException: source B is misconfigured<---
…

Memex.Portal.Shared.Test.InstanceIdRulesMatchTheRegistryTest ‑ TheSetupHostAgreesWithTheRegistry(candidate: "d1cce807-0e4c-4d1d-b926-aa73839c87bf")
Memex.Portal.Shared.Test.PublishStorageFaultIsNotAConflictTest ‑ AStorageFault_IsA500_NamingTheError(fault: System.AggregateException: One or more errors occurred. (Access to the path '/data/modules/activation.d/MeshWeaver.Markdown.Export.json' is denied.)
 ---> System.UnauthorizedAccessException: Access to the path '/data/modules/activation.d/MeshWeaver.Markdown.Export.json' is denied.
   --- End of inner exception stack trace ---)
Memex.Portal.Shared.Test.PublishStorageFaultIsNotAConflictTest ‑ AStorageFault_IsA500_NamingTheError(fault: System.InvalidOperationException: landing faulted
 ---> System.UnauthorizedAccessException: Access to the path '/data/modules/activation.d/MeshWeaver.Markdown.Export.json' is denied.
   --- End of inner exception stack trace ---)
Memex.Portal.Shared.Test.SessionDenialIsAnAnswerTest ‑ OnlyAVerdictReadsAsADenial(shape: "the same verdict nested, as a late denial dispatch"···, failure: System.InvalidOperationException: write failed
 ---> System.UnauthorizedAccessException: Access denied
   --- End of inner exception stack trace ---, isDenial: True)
MeshWeaver.Compiler.Pipeline.Test.AHeldReleaseRequestSaysWhyTest ‑ AReleaseRequestAnsweredByAHold_RecordsWhy_AndACompileClearsIt
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall as an aggregate's FIRST member", fault: System.AggregateException: One or more errors occurred. (Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.)
 ---> MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
   --- End of inner exception stack trace ---, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall as an aggregate's SECOND member", fault: System.AggregateException: One or more errors occurred. (The operation has timed out.) (Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.)
 ---> System.TimeoutException: The operation has timed out.
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.<---
, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall nested in an aggregate inside a wrapper", fault: System.InvalidOperationException: outer
 ---> System.AggregateException: One or more errors occurred. (Value does not fall within the expected range.) (Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.)
 ---> System.ArgumentException: Value does not fall within the expected range.
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.<---

   --- End of inner exception stack trace ---, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall wrapped as InnerException", fault: System.InvalidOperationException: outer
 ---> MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
   --- End of inner exception stack trace ---, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "an aggregate of unrelated faults", fault: System.AggregateException: One or more errors occurred. (The operation has timed out.) (Value does not fall within the expected range.)
 ---> System.TimeoutException: The operation has timed out.
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) System.ArgumentException: Value does not fall within the expected range.<---
, expected: False)
…

♻️ This comment has been updated with latest results.

[Copilot]

🟢 Approval recommended

The diagnostic-state changes have no unresolved blocking findings; tests were not executed during this review.

0 open findings

5 resolved since last review

• Control does not verify clearing an existing delivery hold reason
• Completion predicate and return reject JSON-shaped node definitions
• Raw CLR payload check rejects valid JSON-shaped definitions
• Declined bundle version is not tracked in stale decline metadata
• Prebuilt adoption notices omit the blocking compile gate reason

🧠 Review effort: Balanced