PVC increase not working, error in Operator Log: "the object has been modified; please apply your changes to the latest version and try again"

[wosteven]

Hi Team,

we are facing issues with Cassandra Operator not increasing PVCs on one of our environments. We have multiple Cassandra Clusters and they work fine (also PVC increases), but on one of our envs it stopped working.

We see error below in Operator Logs and we assume this may be reason why, but we could not figure out yet how to resolve this:

Operation cannot be fulfilled on cassandradatacenters.cassandra.datastax.com \"cassandra\": the object has been modified; please apply your changes to the latest version and try again

Looks like Operator tries to update on a revision of the CR that is not up to date.

Our Workflow is:

• Change PVC size in our CassandraDatacenter CR

Works on other envs, but on failing env following happens:

• PVC size change is detected by operator
• PVC Template in STS is updated by operator to new size
• but PVCs resources are not updated by operator to new size
• instead we see ERROR in operator log and nothing happens
• Cassandra CR status claims everything is fine

We have Cassandra Operator 1.26.0_v0.32.0 and Cassandra 4.1.8.

Any idea how we can bring back our Operator on this Env back to work ?

Here's error in Cassandra Operator Log:

...
2026-01-13T09:36:55.837Z	INFO	reconcile_racks::CheckPVCResizing	{"controller": "cassandradatacenter_controller", "controllerGroup": "cassandra.datastax.com", "controllerKind": "CassandraDatacenter", "CassandraDatacenter": {"name":"cassandra","namespace":"instana-cassandra"}, "namespace": "instana-cassandra", "name": "cassandra", "reconcileID": "f59fe89f-d075-4d12-a1e0-52b4d6e3daf9", "namespace": "instana-cassandra", "datacenterName": "cassandra", "clusterName": "instana"}
2026-01-13T09:36:55.837Z	INFO	reconciler::listPVCs	{"controller": "cassandradatacenter_controller", "controllerGroup": "cassandra.datastax.com", "controllerKind": "CassandraDatacenter", "CassandraDatacenter": {"name":"cassandra","namespace":"instana-cassandra"}, "namespace": "instana-cassandra", "name": "cassandra", "reconcileID": "f59fe89f-d075-4d12-a1e0-52b4d6e3daf9", "namespace": "instana-cassandra", "datacenterName": "cassandra", "clusterName": "instana"}
2026-01-13T09:36:55.845Z	INFO	unknown field "spec.podTemplateSpec.metadata.creationTimestamp"	{"controller": "cassandradatacenter_controller", "controllerGroup": "cassandra.datastax.com", "controllerKind": "CassandraDatacenter", "CassandraDatacenter": {"name":"cassandra","namespace":"instana-cassandra"}, "namespace": "instana-cassandra", "name": "cassandra", "reconcileID": "f59fe89f-d075-4d12-a1e0-52b4d6e3daf9"}
2026-01-13T09:36:55.845Z	ERROR	controllers.CassandraDatacenter	calculateReconciliationActions returned an error	{"cassandradatacenter": {"name":"cassandra","namespace":"instana-cassandra"}, "requestNamespace": "instana-cassandra", "requestName": "cassandra", "loopID": "f303085c-e4a9-47dd-9e69-b22b74287d06", "error": "Operation cannot be fulfilled on cassandradatacenters.cassandra.datastax.com \"cassandra\": the object has been modified; please apply your changes to the latest version and try again"}
github.com/k8ssandra/cass-operator/internal/controllers/cassandra.(*CassandraDatacenterReconciler).Reconcile
	/tmp/cass-operator-1.26.0/internal/controllers/cassandra/cassandradatacenter_controller.go:149
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:340
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202
2026-01-13T09:36:55.845Z	INFO	Operation cannot be fulfilled on cassandradatacenters.cassandra.datastax.com "cassandra": the object has been modified; please apply your changes to the latest version and try again	{"controller": "cassandradatacenter_controller", "controllerGroup": "cassandra.datastax.com", "controllerKind": "CassandraDatacenter", "CassandraDatacenter": {"name":"cassandra","namespace":"instana-cassandra"}, "namespace": "instana-cassandra", "name": "cassandra", "reconcileID": "f59fe89f-d075-4d12-a1e0-52b4d6e3daf9", "reason": "ReconcileFailed", "eventType": "Warning"}
2026-01-13T09:36:55.845Z	INFO	controllers.CassandraDatacenter	Reconcile loop completed	{"cassandradatacenter": {"name":"cassandra","namespace":"instana-cassandra"}, "requestNamespace": "instana-cassandra", "requestName": "cassandra", "loopID": "f303085c-e4a9-47dd-9e69-b22b74287d06", "duration": 0.399267678}
2026-01-13T09:36:55.845Z	ERROR	Reconciler error	{"controller": "cassandradatacenter_controller", "controllerGroup": "cassandra.datastax.com", "controllerKind": "CassandraDatacenter", "CassandraDatacenter": {"name":"cassandra","namespace":"instana-cassandra"}, "namespace": "instana-cassandra", "name": "cassandra", "reconcileID": "f59fe89f-d075-4d12-a1e0-52b4d6e3daf9", "error": "Operation cannot be fulfilled on cassandradatacenters.cassandra.datastax.com \"cassandra\": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202
2026-01-13T09:36:55.845Z	DEBUG	events	Operation cannot be fulfilled on cassandradatacenters.cassandra.datastax.com "cassandra": the object has been modified; please apply your changes to the latest version and try again	{"type": "Warning", "object": {"kind":"CassandraDatacenter","namespace":"instana-cassandra","name":"cassandra","uid":"6478601d-f01a-4466-8285-ee077a32ab06","apiVersion":"cassandra.datastax.com/v1beta1","resourceVersion":"676430786"}, "reason": "ReconcileFailed"}
2026-01-13T09:36:55.850Z	INFO	controllers.CassandraDatacenter	======== handler::Reconcile has been called	{"cassandradatacenter": {"name":"cassandra","namespace":"instana-cassandra"}, "requestNamespace": "instana-cassandra", "requestName": "cassandra", "loopID": "f4193f02-249e-4187-b3ae-a6444818d88f"}
...

Here's the CassandraDataCenter CR (shortened version, we redacted IPs also):

apiVersion: cassandra.datastax.com/v1beta1
kind: CassandraDatacenter
metadata:
  annotations:
    cassandra.datastax.com/allow-storage-changes: 'true'
  resourceVersion: '676431244'
  name: cassandra
  uid: 6478601d-f01a-4466-8285-ee077a32ab06
  creationTimestamp: '2025-10-14T06:44:06Z'
  generation: 24
  namespace: cassandra
  finalizers:
    - finalizer.cassandra.datastax.com
spec:
  size: 4
  serverVersion: 4.1.8
  k8ssandraClientImage: 'artifact-public.instana.io/self-hosted-images/3rd-party/datastore/k8ssandra-client:0.8.3_v0.25.0'
  serverImage: 'artifact-public.instana.io/self-hosted-images/3rd-party/datastore/cassandra:4.1.8_v0.32.0'
  systemLoggerImage: 'artifact-public.instana.io/self-hosted-images/3rd-party/datastore/system-logger:1.26.0_v0.18.0'
  serverType: cassandra
  storageConfig:
    cassandraDataVolumeClaimSpec:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 3000Gi
      storageClassName: ibm-block-storage-svc-sc
   .....
status:
  lastRollingRestart: '1970-01-01T00:00:01Z'
  observedGeneration: 24
  nodeStatuses:
    instana-cassandra-default-sts-0:
      hostID: b721a505-40cf-44f9-b6e6-d69a6109d04a
      ip: x.x.x.x
      rack: default
    instana-cassandra-default-sts-1:
      hostID: 61620096-b20b-4f68-8ee3-3b900fbd74d8
      ip: x.x.x.x
      rack: default
    instana-cassandra-default-sts-2:
      hostID: 6163f7d2-2f97-4d28-916d-7e9a22bd563c
      ip: x.x.x.x
      rack: default
    instana-cassandra-default-sts-3:
      hostID: 79851b7a-28b4-41e9-8374-231c4f19ad07
      ip: x.x.x.x
      rack: default
  lastServerNodeStarted: '2026-01-12T07:26:15Z'
  metadataVersion: 1
  datacenterName: ''
  superUserUpserted: '2026-01-13T09:36:56Z'
  quietPeriod: '2026-01-13T09:37:01Z'
  usersUpserted: '2026-01-13T09:36:56Z'
  conditions:
    - lastTransitionTime: '2025-10-14T06:45:47Z'
      message: ''
      reason: ''
      status: 'True'
      type: Healthy
    - lastTransitionTime: '2026-01-13T09:36:55Z'
      message: ''
      reason: ''
      status: 'False'
      type: ResizingVolumes
    - lastTransitionTime: '2025-10-14T06:49:45Z'
      message: ''
      reason: ''
      status: 'False'
      type: Stopped
    - lastTransitionTime: '2025-10-14T06:49:45Z'
      message: ''
      reason: ''
      status: 'False'
      type: ReplacingNodes
    - lastTransitionTime: '2026-01-13T09:36:56Z'
      message: ''
      reason: ''
      status: 'False'
      type: Updating
    - lastTransitionTime: '2025-10-14T06:49:45Z'
      message: ''
      reason: ''
      status: 'False'
      type: RollingRestart
    - lastTransitionTime: '2025-10-14T06:49:45Z'
      message: ''
      reason: ''
      status: 'False'
      type: Resuming
    - lastTransitionTime: '2025-10-14T06:49:45Z'
      message: ''
      reason: ''
      status: 'False'
      type: ScalingDown
    - lastTransitionTime: '2026-01-09T14:37:30Z'
      message: ''
      reason: ''
      status: 'True'
      type: Valid
    - lastTransitionTime: '2025-10-14T06:49:45Z'
      message: ''
      reason: ''
      status: 'True'
      type: Initialized
    - lastTransitionTime: '2025-10-14T06:49:45Z'
      message: ''
      reason: ''
      status: 'True'
      type: Ready
    - lastTransitionTime: '2025-11-25T08:42:59Z'
      message: ''
      reason: ''
      status: 'False'
      type: RequiresUpdate
  cassandraOperatorProgress: Ready

Any help greatly appreciated !!

[burmanm]

The cass-operator is trying to update the status field ResizingVolumes (or the PVC resizing has failed and it's trying to tell that). Now, for some reason something else tried to update CassandraDatacenter simultaneously or there's been stale watcher I guess as alternative.

Multiple reasons I can think of, such as external controllers monitoring/modifying the CassandraDatacenter, another instance of cass-operator etc. But if you're sure nothing else will touch it, then one option to try to solve it could be just restart the operator with kubectl rollout restart deployment/cass-operator (adjust the name of the deployment to your Kustomize/Helm/ArgoCD/etc generated name) and see if that helps with cache refresh.

[wosteven]

Ty @burmanm for trying to help !

We already tried restarting Operator multiple times - that did not help. We are also not aware of external controllers in this Clusters, so only thing left would be the your hint to a "stale watcher".

Do you have any advice on how we can check/identify such ?

[burmanm]

Well, not really. The watchers get recreated when restarting the operator so if it repeats then that shouldn't be the reason. Does the cluster have some webhooks that do something? Like gatekeeper or others that process any update?

Something is changing the resourceVersion (you can track this through the kubectl to see if this is true) and that causes the issue.