NuGet packages bind on every load path: the build records its #r closure

[rbuergi]

Problem

#6834 fixed the compile of a NodeType declaring #r "nuget:…" (host wins). On an enterprise client portal at platform 3.1.10992 (contains cab0ae9), measured 2026-10-16, a NodeType declaring #r "nuget: Microsoft.Data.SqlClient, 6.1.1" compiled Ok, but every load of its cached build failed on both replicas, dozens of times:

Failed to extract NodeTypeConfigurations from /data/assembly-cache/<type>/v33-….dll: Could not load file or assembly 'Microsoft.Data.SqlClient, Version=6.0.0.0, …'
  ReflectionTypeLoadException at RuntimeModule.GetTypes() at MeshNodeCompilationService.CompileResultFromAssembly(…)
DynamicContentTypeRegistrationHostedService: 1 baked dynamic NodeType(s) could NOT be registered … (Faulted: …)
EnrichWithNodeType: the recorded build … still does not load in this process after 1 recompile attempt(s)

Root cause

The package probing directories reached the NodeAssemblyLoadContext only through ICompilationCacheService.RegisterProbingDirectories(nodeName, …), which only the emit called, and only in memory. Any other reader of the same bytes had an empty registry: a replica that never compiled the type, the compiling replica after a restart, an adopted CI bake, the start-up registration pass, EnrichWithNodeType, and a compile attempt that hit an existing build instead of emitting. Those readers loaded the build with nothing to probe. Load fell through to the default context, which does not ship SqlClient, so GetTypes() threw. Nothing about the build recorded which packages it needed, so no reader could recover.

Fix: the build carries its package closure

1. NuGetBuildRecord (MeshWeaver.Compiler). Both producers, the runtime PrepareGeneratedSource and the bake's NodeSetCompiler.ResolveInputs, stamp one [assembly: System.Reflection.AssemblyMetadata("MeshWeaver.NuGetPackage", "Id, Version")] per reference into the generated skeleton, ahead of its own assembly attribute.

   • The record is in the bytes, so it travels wherever a build travels (store, blob, bundle).
   • Package paths are not recorded, because another replica does not share the compiling process's package folder.
   • The stamp is part of the generated text, so the content key covers it. A pre-stamp build of the same sources gets a different key and is rebuilt, not adopted.
   • A NodeType with no #r nuget gets a byte-identical source and keeps its key, so nothing in the fleet is rebuilt.

2. BuildPackageProbing (Compiler.Pipeline). Before every load of an already-built assembly, it reads the record from the file's metadata (a header read; nothing is loaded). It re-resolves the record through INuGetAssemblyResolver, which is memoized and served from a warm package folder without the network. It then registers the probing directories for that build path via the new RegisterBuildProbingDirectories, which takes precedence over the node-wide registration. It is wired into:

   • the post-compile scan in CompileAndGetConfigurations (this covers the cache-hit attempt);
   • GetConfigurationsFromExistingAssembly, the funnel for start-up registration, EnrichWithNodeType, the data-model and contract areas and adopted builds. It stays eager, as callers expect, when nothing is recorded;
   • the cell surface.

   After that the load context binds exactly as on the compiling process: platform first, then host unification, then the package's runtimes/<rid>/lib asset. The In-mesh #r nuget compiles: host wins, no stub in the default context, ref-assembly compile surface #6834 host-wins rule is unchanged.

3. If a resolution fails or times out (AssemblyLoadTimeout), a WARNING names the packages and the cause, and the load proceeds and fails with the loader's own message.

Tests

• NuGetPackagesLoadOnAFreshReplicaTest (live feed; skips when nuget.org is unreachable, like NuGetPackageHostWinsTest). A SqlClient NodeType whose type derives from a package type is built twice: once by the runtime compile and once by the bake. Each build is copied to a store-shaped path the process never loaded and loaded through a fresh CompilationCacheService + MeshNodeCompilationService, i.e. an empty probing registry and resolver memo, the state of another replica or a restart. The configuration scan must succeed, SqlConnection must be constructed, and SqlClient must be the runtimes/ asset. Both cases were red before the fix, with the production message verbatim.
• NuGetBuildRecordTest (offline): no package ⇒ byte-identical source; each reference round-trips through an emitted image; nothing recorded ⇒ empty.
• Local: MeshWeaver.Compiler.Pipeline.Test 1506/1506 passed; MeshWeaver.Graph.Test NuGet filter 24/24. Touched projects and their dependents build with -c Release -warnaserror, 0 warnings.

Deploy note

After the roll, an affected NodeType's recorded build predates the stamp, so its content key no longer matches and the next compile produces a stamped build. Recycle the affected NodeType (for example via the recycle verb) if its address still serves the assembly-unavailable diagnosis.

Pairs-with: none — no public type or member removed; ICompilationCacheService is internal.

Doc: Doc/Architecture/NuGetRuntimeAssets → "Every load path: the build carries its package closure".

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

Unresolved pool cancellation, activation dependencies, offline resolution, probing precedence, and synchronous metadata reads affect loading reliability.

5 open findings

• Feed outage prevents resolution from installed package files · New
• Timeout leaves pool slot held by replay subscription · New
• Compile pool dependency can deadlock cached-build activation · New
• Fallback probing context overrides later build-specific registration · New
• Synchronous DLL metadata reads block hub signalling thread · New

What changed in this PR

Extends the earlier NuGet compile fix so cached NodeType builds carry package references that other replicas can resolve before loading.

Changes:

• Records package references in runtime-compiled and baked assemblies.
• Resolves build-specific probing directories before assembly loading.
• Adds fresh-replica tests and documents the loading behavior.

File	Description
test/​MeshWeaver.Compiler.Pipeline.Test/​NuGetPackagesLoadOnAFreshReplicaTest.cs	Adds replica-loading and metadata round-trip tests.
test/​MeshWeaver.Compiler.Pipeline.Test/​NuGetPackageHostWinsTest.cs	Shares the NuGet availability helper.
src/​MeshWeaver.Documentation/​Data/​Architecture/​NuGetRuntimeAssets.md	Documents recorded references and loading paths.
src/​MeshWeaver.Compiler/​NuGetBuildRecord.cs	Writes and reads package metadata.
src/​MeshWeaver.Compiler/​NodeSetCompiler.cs	Stamps references into baked builds.
src/​MeshWeaver.Compiler.Pipeline/​MeshNodeCompilationService.cs	Stamps runtime builds and prepares package probing.
src/​MeshWeaver.Compiler.Pipeline/​CompilationCacheService.cs	Adds build-specific probing registration.
src/​MeshWeaver.Compiler.Pipeline/​CellSurfaceAssemblyProvider.cs	Prepares package probing for cell surfaces.
src/​MeshWeaver.Compiler.Pipeline/​BuildPackageProbing.cs	Resolves recorded packages before loading.

🧠 Review effort: Balanced

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

Test Results

    15 files   -    11      15 suites   - 11   46m 17s ⏱️ - 10m 50s
11 228 tests  - 1 608  11 228 ✅  - 1 415  0 💤  - 193  0 ❌ ±0 
11 244 runs   - 1 608  11 244 ✅  - 1 415  0 💤  - 193  0 ❌ ±0 

Results for commit 802cb9f. ± Comparison against base commit b27d874.

This pull request removes 1649 and adds 15 tests. Note that renamed tests count towards both.

   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---)
   --- End of inner exception stack trace ---, expected: True)
   --- End of inner exception stack trace ---, isDenial: True)
 ---> (Inner Exception #1) MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.<---
 ---> (Inner Exception #1) MeshWeaver.Messaging.Hub.Test.InfrastructureFaultTest+ProviderException (0x80004005): Failed to connect to 10.42.18.4:5432<---
 ---> (Inner Exception #1) System.ArgumentException: Value does not fall within the expected range.<---
 ---> (Inner Exception #1) System.InvalidOperationException: boom<---
 ---> (Inner Exception #1) System.InvalidOperationException: source B is misconfigured<---
…

Memex.Portal.Shared.Test.InstanceIdRulesMatchTheRegistryTest ‑ TheSetupHostAgreesWithTheRegistry(candidate: "e4b12105-22eb-44a9-b437-60cfc83342a2")
Memex.Portal.Shared.Test.PublishStorageFaultIsNotAConflictTest ‑ AStorageFault_IsA500_NamingTheError(fault: System.AggregateException: One or more errors occurred. (Access to the path '/data/modules/activation.d/MeshWeaver.Markdown.Export.json' is denied.)
 ---> System.UnauthorizedAccessException: Access to the path '/data/modules/activation.d/MeshWeaver.Markdown.Export.json' is denied.
   --- End of inner exception stack trace ---)
Memex.Portal.Shared.Test.PublishStorageFaultIsNotAConflictTest ‑ AStorageFault_IsA500_NamingTheError(fault: System.InvalidOperationException: landing faulted
 ---> System.UnauthorizedAccessException: Access to the path '/data/modules/activation.d/MeshWeaver.Markdown.Export.json' is denied.
   --- End of inner exception stack trace ---)
Memex.Portal.Shared.Test.SessionDenialIsAnAnswerTest ‑ OnlyAVerdictReadsAsADenial(shape: "the same verdict nested, as a late denial dispatch"···, failure: System.InvalidOperationException: write failed
 ---> System.UnauthorizedAccessException: Access denied
   --- End of inner exception stack trace ---, isDenial: True)
MeshWeaver.Compiler.Pipeline.Test.BuildProbingPrecedenceTest ‑ TheBuildsRecordedClosure_ReplacesTheNodeWideFallback
MeshWeaver.Compiler.Pipeline.Test.NuGetBuildRecordTest ‑ ASourceWithoutPackages_IsUnchanged
MeshWeaver.Compiler.Pipeline.Test.NuGetBuildRecordTest ‑ EachReference_IsRecordedInTheImage
MeshWeaver.Compiler.Pipeline.Test.NuGetBuildRecordTest ‑ NothingRecorded_ReadsEmpty
MeshWeaver.Compiler.Pipeline.Test.NuGetPackagesLoadOnAFreshReplicaTest ‑ ABakedBuild_LoadsOnAReplicaThatNeverCompiledIt
MeshWeaver.Compiler.Pipeline.Test.NuGetPackagesLoadOnAFreshReplicaTest ‑ ARuntimeBuild_LoadsOnAReplicaThatNeverCompiledIt
…

♻️ This comment has been updated with latest results.

[Copilot]

🟡 Changes recommended

The new per-build probing cache retains historical registrations indefinitely after context eviction.

1 open finding

• Prevent unbounded growth of cached build probing directories · New

5 resolved since last review

• Compile pool dependency can deadlock cached-build activation
• Timeout leaves pool slot held by replay subscription
• Feed outage prevents resolution from installed package files
• Synchronous DLL metadata reads block hub signalling thread
• Fallback probing context overrides later build-specific registration

🧠 Review effort: Balanced

[rbuergi]

Agreed. The entry outlives the context it was meant for. Fixed in the stacked follow-up #6864: the registration becomes a hand-off, not a registry. The context that takes an entry removes it, in both the factory and ApplyBuildProbing, and every load path re-arms right before it loads, so a context re-created after an eviction is handed it again. That covers aliases (ApplyBuildProbing runs on the aliased context too) and a registration racing a load (whichever context takes the entry first wins; a later load re-arms). BuildProbingPrecedenceTest there pins it. I split it out because this head is fully green, including the dependent Plugins suites. The growth is one small entry per historical build path, which bounds it to the builds this process loaded, so it should not hold back the load fix.