Fix batched silhouette score reduction aliasing

[csadorf]

Stores the row-wise minimum for batched silhouette scores in a separate device vector, preventing the reduction output from overwriting matrix elements that may still be read.

Adds BatchedMatchesNonBatchedAcrossMetricsAndChunkSizes coverage for four distance metrics, three chunk sizes, and both default and stream-pool handles. The test uses seed 170, which reproduced the bug during concurrent A100 testing. It remains deterministic, single-process coverage and is not guaranteed to reproduce the scheduling-sensitive race.

Validation

Regression validation used the temporary BATCHED_SILHOUETTE_CONCURRENCY_TEST added in ef922834, originally e20bfffc before the rebase. The temporary test has since been reverted.

As a negative control, reverting the fix while retaining the test caused the A100/CUDA 12.2 CI job to fail: 12 of 512 worker seeds produced 22 batched/non-batched mismatches above the 1e-4 tolerance. This confirms that the stress workload detects the unfixed reduction-aliasing behavior in the target environment.

Closes #2432

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[csadorf]

The negative-control run behaved as expected. With the presumed fix reverted, BATCHED_SILHOUETTE_CONCURRENCY_TEST failed on A100/CUDA 12.2: 12 of 512 worker seeds produced 22 batched/non-batched mismatches above the 1e-4 tolerance. The same test passed on L4/CUDA 13.

This confirms that the stress test detects the unfixed behavior in the target environment. The next step is to restore the fix and rerun the same matrix; the A100/CUDA 12.2 test should then pass.

The separate NN-descent recall failure on L4 appears unrelated.

[cjnolet]

@csadorf, if you sign your commits, CI will start automatically.

[cjnolet]

/ok to test be4b2b6