Repository Version deletion is slow after creating a new repository version

[dkliban]

Version
3.107.1

Describe the bug
Repository version creation seems to happen relatively fast, but then the task continues to run as it trims older repository versions.

This is only an issue when the repository has retain-repo-versions set.

2026-04-14 18:32:24.559 pulp [ab6c3d30bf654ca5ae0b98d2aa0fc6cc]: pulpcore.app.models.repository:INFO: Deleting repository version <Repository: metadata-old; Version: 1409> due to version retention limit.
2026-04-14 18:26:45.853 pulp [ab6c3d30bf654ca5ae0b98d2aa0fc6cc]: pulpcore.app.models.repository:INFO: Deleting repository version <Repository: metadata-old; Version: 1413> due to version retention limit.
2026-04-14 18:21:02.897 pulp [ab6c3d30bf654ca5ae0b98d2aa0fc6cc]: pulpcore.app.models.repository:INFO: Deleting repository version <Repository: metadata-old; Version: 1421> due to version retention limit
2026-04-14 18:15:19.273 pulp [ab6c3d30bf654ca5ae0b98d2aa0fc6cc]: pulpcore.app.models.repository:INFO: Deleting repository version <Repository: metadata-old; Version: 1422> due to version retention limit.

[dkliban]

Here is what Claude pointed out:

The log message is at pulpcore/app/models/repository.py:421, in the cleanup_old_versions() method of Repository:

  @hook(AFTER_UPDATE, when="retain_repo_versions", has_changed=True)                                                                                                                                               
  def _cleanup_old_versions_hook(self):                                                                                                                                                                            
      transaction.on_commit(self.cleanup_old_versions)                                                                                                                                                             
                                                                                                                                                                                                                   
  def cleanup_old_versions(self):                                                                                                                                                                                  
      if self.retain_repo_versions:                                                                                                                                                                                
          versions = self.versions.complete().exclude(pk__in=self.protected_versions())
          for version in versions.order_by("-number")[self.retain_repo_versions:]:                                                                                                                                 
              _logger.info("Deleting repository version {} due to version retention limit.".format(version))
              version.delete()   # <-- the slow part                                                                                                                                                               

Triggered via django-lifecycle when retain_repo_versions changes on a repository.

Why it's slow

Each version.delete() (line 1288) runs inside transaction.atomic() and:

1. Calls select_for_update() on all RepositoryContent rows for that repository
2. Runs _squash() (line 1236), which does multiple UPDATE/DELETE queries over those rows + calls _compute_counts() on the successor version

With a repo at version 1300, there are likely thousands of RepositoryContent rows, and this entire process runs once per version being deleted, serially.

Recent changes to this code path

Two commits are directly relevant:

a6c1a8e (Feb 10, 2026) — "Harden RepositoryVersion against memoization bugs"

• Added assert len(self.content_ids) == self._content_relationships().count() inside _compute_counts(), which is called by _squash() on every version delete
• Added assert issubclass(content.model, Content) guards to add_content/remove_content/set_content

28cd1f8 (Mar 2, 2026) — "Fix a regression in repository version cleanup / squash"

• This commit fixed a regression introduced by the above: old repository versions don't have self.content_ids populated, so the new assert was crashing during cleanup. The fix changed it to if
  self.content_ids: assert ....
• This was the most recent change directly on this deletion/squash path.

[dralley]

Do you have the raw profiles to post? Or the HTML pyinstrument report

[dralley]

In any case I suppose if the asserts are a problem, you can remove them.

[dkliban]

Here is the pyinstrument profile 019d91f5-8704-77e5-ae70-17a652e1c29c.html

[dralley]

It's not clear to me that the assertion is the issue. The assertion was added to _compute_counts, which is 0.1% of the overall time according to that profile.

Also the problematic query is a downstream of RepositoryVersionQuerySet.__iter__(), which I don't think my commit added or touched?

The profile suggests the issue is in here, none of which has changed recently

https://github.com/pulp/pulpcore/blame/main/pulpcore/app/models/repository.py#L412-L423

So it maybe it has something to do with an index not being used?

[dkliban]

the working theory is that selecting all the repository versions with the new content_ids field is causing the slowdown. So we are going to try out a patch that will defer fetching content_ids

for version in versions.defer("content_ids").order_by("-number")[self.retain_repo_versions:]:

If this works improves the performance, we will open a PR later today.