Improve parallel writes

[HeDo88TH]

Concurrent writes to the same dataset race on the SQLite writer lock, causing database is locked failures and dropped index updates. This PR serializes index writes per dataset, maps native DdbResult failures to typed exceptions, and retries transient contention instead of failing or leaving files half-indexed.

Backend

• DatasetIndexQueue: per-dataset coalescing queue: concurrent index requests for one dataset drain through a single lane as one native batch call with per-item isolation. Lanes released on dataset delete, idle-trimmed after a configurable period (IndexQueue:IdleLaneTrimSeconds, default 15 min), retirement race/leak fixed.
• Typed result mapping: NativeDdbWrapper maps DdbResult shapes via the new DdbResultMapper; new DdbBusyException propagates for retry instead of being swallowed as InvalidOperationException. Batch add returns per-item BatchAddResult (success / unchanged / failed).
• Unified HTTP mapping: ApiExceptionClassifier + ApiExceptionFilter give all controllers one exception→HTTP mapping; transient DB contention → 503 + Retry-After.
• Hangfire: busy exceptions reach the automatic retry policy. Requeue now transitions a failed job back to Enqueued under the same id (previously archived it via DeletedState). Retry endpoint returns 409 when a pending-build sweep already owns the retry, and resets stale error/artifact fields on requeue.
• Upload recovery: AddNew stages to <org>/<ds>/.uploads/ and quarantines to quarantine/ on index failure instead of leaving orphan files. New IndexReconciliationService recurring job re-queues on-disk files missing from the index, pruning .ddb, .uploads and quarantine roots.
• Interface cleanup: IDDB split into role interfaces (IDdbIndex, IDdbBuild, IDdbMeta, IDdbRaster, IDdbAnalytics).
• Tooling: suppress Qt warning floods in the prod compose file; silence module-open log in DdbManager.

Client (ClientApp submodule → Hub)

Resilient uploads: retry with AIMD adaptive concurrency and proper backoff, session teardown on permanent failure, dropzone hardening. Plus: reworked task progress bar with indeterminate state, and point clouds render without a valid CRS.

Tests

New suites: DatasetIndexQueueTests, ApiExceptionClassifierTests, DdbBuildBusyPassthroughTests, IndexReconciliationServiceTests, NativeDdbWrapperResultMappingTests, BackgroundJobsProcessorTest, RetryEndpointTest (409 / same-id requeue contract), quarantine/AddNew cases in ObjectManagerTest. Shared Hangfire test doubles extracted. All green: Registry.Web.Test 971/971, Registry.Adapters.Ddb.Test 88/88, Registry.Adapters.Test 7/7.

Behavior changes

• Index failures on upload are no longer silent: callers get 503 and the client retries; the server quarantines the file in the meantime.
• IDdbWrapper.AddWithOptions trimmed to (ddbPath, paths, stopOnError).

[Copilot]

Pull request overview

This pull request improves parallel write reliability and fairness around DroneDB index updates by introducing a per-dataset coalescing queue, adding typed handling for transient DB contention, and adding a recurring reconciliation sweep to catch/repair unindexed-on-disk files. It also refactors the DroneDB dataset API into role-based interfaces to reduce coupling for narrow consumers.

Changes:

• Add DdbBusyException + DdbResult.Busy and propagate/handle transient contention consistently (including Hangfire retry policies).
• Introduce IDatasetIndexQueue + DatasetIndexQueue to coalesce concurrent index writes into native batch calls with per-item isolation, plus reconciliation/quarantine flows.
• Split IDDB into role interfaces (IDdbIndex, IDdbBuild, IDdbMeta, IDdbRaster, IDdbAnalytics) and add batch-add result types for the new native completeness contract.

Reviewed changes

Copilot reviewed 32 out of 32 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
Registry.Web/Utilities/ProcessingNodeServiceCollectionExtensions.cs	Registers IDatasetIndexQueue and IndexReconciliationService for processing nodes.
Registry.Web/Utilities/HangfireUtils.cs	Expands Hangfire retries with backoff for transient DDB contention/build-lock scenarios.
Registry.Web/Utilities/ApiExceptionFilter.cs	Adds global exception-to-HTTP mapping, including transient 503 + Retry-After handling.
Registry.Web/Startup.cs	Registers ApiExceptionFilter and wires IDatasetIndexQueue/IndexReconciliationService in DI.
Registry.Web/Services/Ports/IDatasetIndexQueue.cs	Defines the per-dataset coalescing index-write queue contract.
Registry.Web/Services/Managers/ObjectsManager.cs	Routes indexing through IDatasetIndexQueue when available and quarantines files on indexing failures.
Registry.Web/Services/Initialization/HangfireJobsInitializer.cs	Schedules the recurring “index-reconciliation” Hangfire job.
Registry.Web/Services/HeavyTasks/Adapters/HeavyTaskJobWrapper.cs	Applies transient-only Hangfire backoff retry policy to heavy tasks.
Registry.Web/Services/Adapters/IndexReconciliationService.cs	Adds recurring sweep to re-enqueue unindexed files, report missing files, and age out quarantine.
Registry.Web/Services/Adapters/DatasetIndexQueue.cs	Implements per-dataset FIFO lanes with batching, retries, and per-item completion isolation.
Registry.Web/Models/Configuration/ReconciliationSettings.cs	Adds config tuning for reconciliation sweep behavior.
Registry.Web/Models/Configuration/IndexQueueSettings.cs	Adds config tuning for batching/coalescing behavior and deadlines.
Registry.Web/Models/Configuration/AppSettings.cs	Adds app settings hooks for index queue + reconciliation configuration and cron.
Registry.Web/Exceptions/TransientException.cs	Adds a generic retryable exception mapped to 503 + Retry-After.
Registry.Web/appsettings-default.json	Provides defaults for index queue/reconciliation settings and cron schedule.
Registry.Web.Test/ProcessingNodeDiCompletenessTests.cs	Verifies DI completeness for the new queue and reconciliation service on processing nodes.
Registry.Web.Test/ObjectManagerTest.cs	Adds coverage for quarantining behavior when indexing fails after file placement.
Registry.Web.Test/IndexReconciliationServiceTests.cs	Adds unit tests for reconciliation re-enqueue, reserved path exclusion, report-only behavior, and quarantine aging.
Registry.Web.Test/DatasetIndexQueueTests.cs	Adds unit tests for batching/coalescing, per-item errors, unchanged resolution, retry behavior, and completeness gaps.
Registry.Ports/IDdbWrapper.cs	Adds native wrapper surface for “add with options” returning completeness results.
Registry.Ports/DroneDB/IDdbRaster.cs	Introduces role interface for raster operations.
Registry.Ports/DroneDB/IDdbMeta.cs	Introduces role interface for metadata operations.
Registry.Ports/DroneDB/IDdbIndex.cs	Introduces role interface for index operations including AddRawBatchWithOptions.
Registry.Ports/DroneDB/IDdbBuild.cs	Introduces role interface for build operations.
Registry.Ports/DroneDB/IDdbAnalytics.cs	Introduces role interface for analytics operations.
Registry.Ports/DroneDB/IDDB.cs	Converts IDDB into an aggregate over the new role interfaces while retaining constants.
Registry.Ports/DroneDB/BatchAddResult.cs	Adds DTOs representing batch-add completeness results (entries/unchanged/errors).
Registry.Adapters/DroneDB/NativeDdbWrapper.cs	Implements native P/Invoke entrypoint for batch add with options and throws DdbBusyException on contention.
Registry.Adapters/DroneDB/DdbResult.cs	Adds Busy result code for transient contention.
Registry.Adapters/DroneDB/DdbBusyException.cs	Adds typed exception to represent retryable DB contention.
Registry.Adapters/DroneDB/Ddb.cs	Preserves DdbBusyException through adapter layers and adds AddRawBatchWithOptions.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 32 out of 32 changed files in this pull request and generated 4 comments.

Suppressed comments (4)

Registry.Web/Services/Adapters/DatasetIndexQueue.cs:90

• The timeout and caller token only govern channel writes. Once all requests are queued, Task.WhenAll ignores linked, so a blocked native commit can wait forever and the advertised EnqueueTimeoutSeconds/cancellation contract never fires. Apply the linked token while awaiting completion so the existing timeout catch can return TransientException.

            return await Task.WhenAll(tasks);

Registry.Web/Services/Adapters/DatasetIndexQueue.cs:104

• This implementation returns immediately even when the dataset lane has queued or in-flight requests, contradicting the public contract that FlushAsync waits for pending work to commit. A caller that starts enqueues and then flushes can therefore proceed before the index write. Add a per-lane barrier request (or track a drain generation/task) and complete it only after all preceding requests have committed.

    public Task FlushAsync(DatasetKey dataset, CancellationToken ct = default)
    {
        // The drain loop commits everything currently queued on every iteration; enqueueing
        // paths is what waits for a commit, so flushing is naturally expressed as enqueueing
        // zero work and waiting for the lane to have drained at least once. Since there is

Registry.Web/Utilities/ApiExceptionFilter.cs:20

• The global filter does not handle the main upload failure path: ObjectsController.Post catches every exception at ObjectsController.cs:603-609, and ControllerBaseEx.ExceptionResult maps DdbBusyException/TransientException to its default 400 response. Consequently, contention from the new index queue never reaches this filter and clients receive 400 instead of the intended retryable 503. Update the existing exception helper (or let these typed exceptions escape controller catches) so the mapping is effective for existing actions.

/// Additive: existing controller-level try/catch + <see cref="ControllerBaseEx.ExceptionResult(Exception)"/>
/// still work as-is for actions that already handle their own errors; this filter only fires for
/// exceptions that escape the action unhandled.

Registry.Web/Services/Adapters/DatasetIndexQueue.cs:251

• After the first busy result, the second attempt catches every exception and labels it transient. If that attempt fails with disk-full, corruption, an incompatible native library, or another permanent error, clients are incorrectly told to retry and the original classification is lost. Only a second DdbBusyException should be wrapped as TransientException; other exceptions should be propagated to the batch requests unchanged.

            catch (Exception retryEx)
            {
                var transient = new TransientException(
                    $"Dataset '{key.OrgSlug}/{key.InternalRef}' index is busy; retry the request", retryEx,
                    retryAfterSeconds: 2);

[Copilot]

Pull request overview

Copilot reviewed 65 out of 65 changed files in this pull request and generated 2 comments.

Suppressed comments (8)

Registry.Web/Services/Adapters/DatasetIndexQueue.cs:145

• The configured deadline stops only WriteAsync; once all requests fit in the channel, Task.WhenAll does not observe linked. A dead/stuck drain loop therefore leaves the HTTP caller waiting indefinitely instead of producing the documented TransientException. Apply the linked token while awaiting completion.
  Registry.Web/Services/Adapters/DatasetIndexQueue.cs:341
• This catch classifies every second-attempt failure as transient. If the first attempt is busy but the retry reveals disk-full, corruption, or another permanent error, callers receive 503/retry guidance forever and the real failure is hidden. Only wrap a second DdbBusyException; propagate other retry failures unchanged.
  Registry.Web/Services/Adapters/IndexReconciliationService.cs:130
• This prunes any directory named .ddb or .uploads, even below a normal user directory. Only root-level prefixes are reserved (IsReservedPath and ObjectsManager allow paths such as project/.ddb/file.tif), so those valid files are omitted from onDisk and may be falsely reported missing. Restrict pruning to direct children of the dataset root.
  Registry.Web/Utilities/ApiExceptionClassifier.cs:77
• The default 500 arm also catches many existing client-validation failures. For example, ObjectsManager throws InvalidOperationException for reserved paths, missing source entries, same source/destination, and overwrite conflicts; these previously became 400 responses through ExceptionResult, but now become generic 500s. Migrate those validation sites to BadRequestException/ConflictException (rather than broadly mapping every InvalidOperationException to 400) before removing the legacy helper.
  Registry.Adapters/DroneDB/DdbResultMapper.cs:26
• DDBGetLastError can return an empty string when no native call set an error (the new mapping test explicitly documents this), so the null-only fallback produces typed exceptions and API responses with no useful message. Treat empty/whitespace text as missing as well.

        return GetNativeLastError() ?? (operation != null ? "Unknown error in " + operation : "Unknown error");

Registry.Web/Utilities/ApiExceptionFilter.cs:35

• Most migrated controller actions still catch, log at Error, and rethrow. This new global log therefore records the same exception a second time, and expected 4xx exceptions are still emitted once at Error despite the classifier selecting Information. Remove the redundant catch/log/rethrow blocks (or otherwise ensure a single logging owner) so the centralized level policy actually takes effect.
  Registry.Web/Utilities/ApiExceptionClassifier.cs:58
• Correct “contentment” to “contention.”
  Registry.Web/Services/Adapters/DatasetIndexQueue.cs:160
• Correct “retairs” to “retires.”

[Copilot]

Pull request overview

Copilot reviewed 121 out of 121 changed files in this pull request and generated 13 comments.

Suppressed comments (2)

docker/production/docker-compose.yml:114

• *.warning=false disables every Qt warning category, not only the noisy default-category message described above. This can hide unrelated production diagnostics. Scope the rule to the default category instead.
  docker/production/docker-compose.yml:89
• *.warning=false disables every Qt warning category, not only the noisy default-category message described above. This can hide unrelated production diagnostics. Scope the rule to the default category instead.