Untyped select: reads are projected after the merged sort

[rbuergi]

Closes #6830

Root cause

An untyped read (Query<object>) carrying select: was projected to dictionaries by each provider, before the cross-provider merge saw a row. MeshQuery.ClipMergedInitial can only read a sort key, a path tiebreak or a dedup key off a MeshNode, so its authoritative sort was skipped (hits.All(h => h.Item is MeshNode)) and the window was taken in score, then provider-ARRIVAL order — a dictionary has no path to tiebreak on. select:path,name sort:<anything> limit:N served by two providers (a store plus the seeded static nodes, or several partitions) returned one provider's top N. Teaching the merge to read dictionaries cannot work: the sort key need not be among the selected fields.

Fix

MeshQuery runs an untyped read that carries a select: through the providers and the merge as nodes (Query<MeshNode> on both the public and the IMeshQueryCore surface) and projects only the final, sorted and clipped window (and every later delta) to select: dictionaries — the issue's suggested direction. The SQL providers still narrow their columns to the select: list (plus a content sort key) when asked for nodes, so nothing extra is read. An untyped read WITHOUT select: is unchanged (the providers already hand back nodes), which keeps FilterOnlyQueryClipsNewestFirstTest.AMixedBatchKeepsEveryItemUnderTheDefaultOrdering meaningful. No provider changes, so no paired Plugins PR: the Postgres/Cosmos/Snowflake providers' projectSelect path is simply no longer taken through the merge for Query<object>.

Tests

MeshWeaver.Hosting.Test/UntypedProjectionMergedSortTest — a storage-backed provider and the static-node provider (the pairing every portal runs), names interleaved so the global window takes rows from both:

• select:path,name sort:name-desc limit:3 → F, E, D; sort:name → A, B, C; select:name sort:path-desc limit:2 (sort key not selected) → A, C
• rows are still projections carrying exactly the selected keys
• control: the untyped answer equals the typed (Query<MeshNode>, already right since Query sort on a content field: typed key, nulls last #6704) answer row for row

Negative control (MeshQuery.cs reverted to main, test unchanged): total: 5, failed: 5, succeeded: 0, e.g.
Expected collection {"F", "D", "B"} to equal {"F", "E", "D"} because 'namespace:Book nodeType:Chapter select:path,name sort:name-desc limit:3' over two providers answers the mesh's window in the requested order, not one provider's.
Fix restored: total: 5, failed: 0; full MeshWeaver.Hosting.Test total: 1340, failed: 0, skipped: 1; full MeshWeaver.Graph.Test total: 3164, failed: 0 (run on an earlier, broader cut that deferred EVERY untyped read; the final cut narrows it to select: reads).

Doc: Doc/DataMesh/QuerySyntax → select ("The untyped projection happens AFTER the merged sort").

Related: #6704 / MeshWeaver.Plugins#3587 (typed callers).
Recycle on deploy: none.
Pairs-with: none — private members only.

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

The new node-based route replaces relevance ordering with path ordering when a storage provider omits scores.

1 open finding

• Preserve provider relevance scores when merging untyped queries · New

What changed in this PR

Defers untyped select: projection until after MeshWeaver merges, sorts, and clips query results, addressing #6830.

Changes:

• Routes projected untyped queries through the node-based merge.
• Adds two-provider ordering and projection regression tests.
• Documents projection timing.

File	Description
test/​MeshWeaver.Hosting.Test/​UntypedProjectionMergedSortTest.cs	Tests merged ordering, selected fields, and typed-result parity.
src/​MeshWeaver.Hosting/​Persistence/​Query/​MeshQuery.cs	Projects results after merging on both query entry points.
src/​MeshWeaver.Documentation/​Data/​DataMesh/​QuerySyntax.md	Explains projection after sorting and clipping.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[rbuergi]

Confirmed and fixed in 6bda942. The provider ranked free-text matches by fuzzy score before its load cap, but it left Scores null, so ClipMergedInitial scored every row 0 and returned path order. That was already true for TYPED free-text reads, and routing untyped select: reads through the node merge extended it to them. The root fix is in StorageAdapterMeshQueryProvider. RunRankedQueryNodes takes the score off the matched node before any projection or content drop and carries it on the Initial frame, aligned with the rows that pass the type filter. Live deltas are unchanged.

Regressions in UntypedProjectionMergedSortTest use a fixture where relevance order is the reverse of path order. The expected order comes from QueryEvaluator.GetFuzzyScore itself, with a guard that it disagrees with path order.

• Core entry (IMeshQueryCore.Query), typed and untyped, with and without select:.
• Skip=1 Limit=1 pages the relevance order.
• The public entry (MeshQuery.Query<T>(request)) needs a hub for its serializer options. It is pinned at the frame it merges: the secured IMeshQueryProvider.Query<T> Initial carries one descending score per row, the same for typed and untyped reads.

Negative control: with Scores = null restored, 5 of the 10 cases fail (for example, expected needle, the needle somewhere, … but got the path order a long haystack…, the needle somewhere, needle). With the fix all 10 pass, and so do the adjacent query suites (PagedQueryTotalOrder, MeshQueryMergeContract, ContentFieldSort, FilterOnlyQueryClipsNewestFirst).

[github-actions]

Test Results

    15 files   -    11      15 suites   - 11   50m 24s ⏱️ - 9m 33s
11 134 tests  - 1 591  11 134 ✅  - 1 398  0 💤  - 193  0 ❌ ±0 
11 150 runs   - 1 591  11 150 ✅  - 1 398  0 💤  - 193  0 ❌ ±0 

Results for commit 6bda942. ± Comparison against base commit cab0ae9.

This pull request removes 1636 and adds 19 tests. Note that renamed tests count towards both.

   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---)
   --- End of inner exception stack trace ---, expected: True)
   --- End of inner exception stack trace ---, isDenial: True)
 ---> (Inner Exception #1) MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.<---
 ---> (Inner Exception #1) MeshWeaver.Messaging.Hub.Test.InfrastructureFaultTest+ProviderException (0x80004005): Failed to connect to 10.42.18.4:5432<---
 ---> (Inner Exception #1) System.ArgumentException: Value does not fall within the expected range.<---
 ---> (Inner Exception #1) System.InvalidOperationException: boom<---
 ---> (Inner Exception #1) System.InvalidOperationException: source B is misconfigured<---
…

Memex.Portal.Shared.Test.InstanceIdRulesMatchTheRegistryTest ‑ TheSetupHostAgreesWithTheRegistry(candidate: "979c4d68-535f-4913-ab93-c3394423695b")
Memex.Portal.Shared.Test.PublishStorageFaultIsNotAConflictTest ‑ AStorageFault_IsA500_NamingTheError(fault: System.AggregateException: One or more errors occurred. (Access to the path '/data/modules/activation.d/MeshWeaver.Markdown.Export.json' is denied.)
 ---> System.UnauthorizedAccessException: Access to the path '/data/modules/activation.d/MeshWeaver.Markdown.Export.json' is denied.
   --- End of inner exception stack trace ---)
Memex.Portal.Shared.Test.PublishStorageFaultIsNotAConflictTest ‑ AStorageFault_IsA500_NamingTheError(fault: System.InvalidOperationException: landing faulted
 ---> System.UnauthorizedAccessException: Access to the path '/data/modules/activation.d/MeshWeaver.Markdown.Export.json' is denied.
   --- End of inner exception stack trace ---)
Memex.Portal.Shared.Test.SessionDenialIsAnAnswerTest ‑ OnlyAVerdictReadsAsADenial(shape: "the same verdict nested, as a late denial dispatch"···, failure: System.InvalidOperationException: write failed
 ---> System.UnauthorizedAccessException: Access denied
   --- End of inner exception stack trace ---, isDenial: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall as an aggregate's FIRST member", fault: System.AggregateException: One or more errors occurred. (Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.)
 ---> MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
   --- End of inner exception stack trace ---, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall as an aggregate's SECOND member", fault: System.AggregateException: One or more errors occurred. (The operation has timed out.) (Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.)
 ---> System.TimeoutException: The operation has timed out.
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.<---
, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall nested in an aggregate inside a wrapper", fault: System.InvalidOperationException: outer
 ---> System.AggregateException: One or more errors occurred. (Value does not fall within the expected range.) (Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.)
 ---> System.ArgumentException: Value does not fall within the expected range.
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.<---

   --- End of inner exception stack trace ---, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall wrapped as InnerException", fault: System.InvalidOperationException: outer
 ---> MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
   --- End of inner exception stack trace ---, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "an aggregate of unrelated faults", fault: System.AggregateException: One or more errors occurred. (The operation has timed out.) (Value does not fall within the expected range.)
 ---> System.TimeoutException: The operation has timed out.
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) System.ArgumentException: Value does not fall within the expected range.<---
, expected: False)
MeshWeaver.Hosting.Test.UntypedProjectionMergedSortTest ‑ AFreeTextReadKeepsTheProvidersRelevanceOrder_Core(text: "namespace:Book needle limit:3")
…

♻️ This comment has been updated with latest results.

[Copilot]

🟢 Approval recommended

The earlier relevance regression is addressed, and only nonblocking collection-convention nits remain.

1 open finding

• Preserve provider relevance scores when merging untyped queries

🧠 Review effort: Balanced