Bug: `retryAttempts` is a package-level variable — shared across all Cluster objects and survives controller restarts

[mdryaan]

📜 Description

pkg/hub/controllers/cluster/reconciler.go declares var retryAttempts = 0 at the package level (line 88). The handleClusterDeletion function increments it to enforce MAX_CLUSTER_DEREGISTRATION_ATTEMPTS = 3. Because it is a package-level var:

• If multiple Cluster objects are being deleted simultaneously, their attempt counts are merged.
• After 3 failures on any single cluster, all subsequent clusters also stop retrying immediately.
• Restarting the operator pod resets the counter, breaking the intended "stop after 3 attempts" semantics.

👟 Reproduction steps

1. Create two Cluster CRs.
2. Make createDeregisterJob fail (mock or break the job).
3. Delete both clusters simultaneously.
4. Observe that the first cluster's failures exhaust retryAttempts for the second.

👍 Expected behavior

Each Cluster object should track its own deregistration attempt count — store it in the Cluster CR's status or annotations.

👎 Actual Behavior

Shared global counter causes incorrect retry limiting across unrelated cluster objects.

🐚 Relevant log output

Version

No response

🖥️ What operating system are you seeing the problem on?

No response

✅ Proposed Solution

Move attempt tracking into the Cluster CR's status field or use an annotation. Alternatively, use r.Status().Update to persist a retry count per object.

👀 Have you spent some time to check if this issue has been raised before?

• I checked and didn't find any similar issue

Code of Conduct

• I agree to follow this project's Code of Conduct