Skip to content

ClusterFacesJob breaks with numeric user IDs, and its OOM takes down the whole cron run #1558

Description

@alex-yatsenko

Two problems reached through the same job. The second is not limited to face clustering: it stops all background job processing for the instance.

Environment

  • Nextcloud 34.0.2
  • recognize 12.0.0
  • PHP 8.5.9, CLI memory_limit = 512M (the Docker image default)
  • MariaDB, ~34 000 face detections across 4 users
  • User IDs are numeric strings1, 4, 27, 28 — created by an external provisioning system. This is what exposes the first bug.

1. TypeError: numeric user IDs are turned into integers

OCA\Recognize\Service\FaceClusterAnalyzer::calculateClusters():
Argument #1 ($userId) must be of type string, int given,
called in .../recognize/lib/BackgroundJobs/ClusterFacesJob.php on line 39

Root cause

FsActionService collects user IDs as array keys:

// lib/Service/FsActionService.php
$userIdsToScheduleClustering[$userId] = true;      // key "4" becomes int(4)
// ...
foreach (array_keys($userIdsToScheduleClustering) as $userId) {
    $this->jobList->add(ClusterFacesJob::class, ['userId' => $userId]);   // int
}

PHP silently casts numeric-string array keys to integers, so array_keys() hands back int(4). It is stored as {"userId":4} and comes back as an int from json_decode($row['argument'], true) in core's JobList, where it meets the string type declaration on calculateClusters().

Why the other call sites are fine

  • ClusteringFaceClassifier::classify() passes the value straight through — its IDs come from getUID() and stay strings.
  • occ recognize:cluster-faces reads IDs from the database, also strings.

So the failure only appears on the file-action path, which made it look intermittent.

Suggested fix

Cast at the job boundary, which fixes it regardless of how the argument was produced:

// lib/BackgroundJobs/ClusterFacesJob.php
$this->clusterAnalyzer->calculateClusters((string)$argument['userId'], self::BATCH_SIZE);

Avoiding the array-key round-trip in FsActionService (e.g. array_unique() over a plain list) would address it at the source as well.

Not the same as #1058, in case it comes up: that one was argument #2 on recognize 6.0.1. This is argument #1, on a different call path.


2. Out of memory in clustering kills the entire cron.php run

Jobs queued from the classifier path get past the type declaration and reach the clustering itself, where they die here instead:

PHP Fatal error: Allowed memory size of 536870912 bytes exhausted
  (tried to allocate 163840 bytes)
  in .../recognize/lib/Clustering/MrdBallTree.php:456
#0 MrdBallTree->cachedComputeNative(...)
#1 ...->updateNearestNeighbors(DualTreeClique, DualTreeClique, 4, INF, Array)

Why this is worse than it looks

ClusterFacesJob::run() wraps the call in catch (\Throwable $e), but a memory-exhaustion fatal is not catchable. The whole cron.php process dies, and every other background job scheduled for that run is skipped.

On this instance 9 of 50 cron runs died this way in one night. While face classification was still producing detections, each classification batch re-queued ClusterFacesJob, so the crash kept repeating.

Why the default is hard to satisfy

ClusterFacesJob::BATCH_SIZE is 10000, which is effectively "no limit" for most instances. The documented cost — from the CLI command's own description — is:

faces memory
2 000 450 MB
4 000 700 MB
5 000 1 200 MB

Extrapolating that table along the documented O(n²) — our own arithmetic, not a measurement — the largest user here (9 715 faces) would need on the order of 4.5 GB, against a default memory_limit of 512M. What we did measure is that the job dies at 512M. The CLI command can be kept under a limit via --batch-size; the background job has no equivalent.

Suggestions

  • Default the job's batch size to something that fits a default memory_limit, rather than 10000; or derive it from ini_get('memory_limit').
  • Check memory_get_usage() against the limit inside the clustering loop and stop cleanly, re-queueing the remainder — so a large library converges over several runs instead of failing forever.

Happy to test a patch on this instance. The data set is 34 000 detections across 4 users with numeric IDs; both failures above are from its logs.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions