Skip to content

Fix 154: adding a finalizer to the tunnel's associated secret#158

Open
cyclingwithelephants wants to merge 11 commits intoadyanth:mainfrom
cyclingwithelephants:154-secret-finaliser
Open

Fix 154: adding a finalizer to the tunnel's associated secret#158
cyclingwithelephants wants to merge 11 commits intoadyanth:mainfrom
cyclingwithelephants:154-secret-finaliser

Conversation

@cyclingwithelephants
Copy link
Contributor

fixes #154

I just wanted to confirm that r.GetTunnel().GetNamespace() for the ClusterTunnel resource is expected to return the cloudflare-operator's namespace? I've tested this working, just wanted to make sure this was built on expected behaviour and wasn't a fluke

@cyclingwithelephants cyclingwithelephants marked this pull request as ready for review May 7, 2025 22:28
@cyclingwithelephants
Copy link
Contributor Author

Looks like coverage reports are not available from forks

Copy link
Owner

@adyanth adyanth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for picking this up!

@adyanth
Copy link
Owner

adyanth commented May 8, 2025

To answer your question, yes the namespace for the cluster tunnel is the operator namespace.

@cyclingwithelephants
Copy link
Contributor Author

Haven't tested yet, but please let me know if you're otherwise happy with the changes

Copy link
Owner

@adyanth adyanth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couple of suggestions, looks good otherwise!

@cyclingwithelephants
Copy link
Contributor Author

this is functionally working, but I've noticed that the controller is producing errors on deletion. This wasn't happening when using the concrete implementation for the secret. What's strange is that this Reconciler error (the last error in the logs) loops maybe 15 times and it looks like it's happening after the tunnel is actually deleted from etcd. Any idea what's going on here?

2025-05-08T23:35:54+01:00	INFO	starting deletion cycle	{"controller": "tunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "Tunnel", "Tunnel": {"name":"adamrummer-com","namespace":"default"}, "namespace": "default", "name": "adamrummer-com", "reconcileID": "2a666e68-bfb5-43e7-bdb2-d4bcc8da318d"}
2025-05-08T23:35:54+01:00	INFO	Scaling down cloudflared	{"controller": "tunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "Tunnel", "Tunnel": {"name":"adamrummer-com","namespace":"default"}, "namespace": "default", "name": "adamrummer-com", "reconcileID": "2a666e68-bfb5-43e7-bdb2-d4bcc8da318d"}
2025-05-08T23:35:54+01:00	DEBUG	events	Starting Tunnel Deletion	{"type": "Normal", "object": {"kind":"Tunnel","namespace":"default","name":"adamrummer-com","uid":"c20c36ca-7fa3-408f-9e60-b679c7085b02","apiVersion":"networking.cfargotunnel.com/v1alpha2","resourceVersion":"2001"}, "reason": "Deleting"}
2025-05-08T23:35:54+01:00	DEBUG	events	Scaling down cloudflared	{"type": "Normal", "object": {"kind":"Tunnel","namespace":"default","name":"adamrummer-com","uid":"c20c36ca-7fa3-408f-9e60-b679c7085b02","apiVersion":"networking.cfargotunnel.com/v1alpha2","resourceVersion":"2001"}, "reason": "Scaling"}
2025-05-08T23:35:54+01:00	INFO	Scaling down successful	{"controller": "tunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "Tunnel", "Tunnel": {"name":"adamrummer-com","namespace":"default"}, "namespace": "default", "name": "adamrummer-com", "reconcileID": "2a666e68-bfb5-43e7-bdb2-d4bcc8da318d"}
2025-05-08T23:35:54+01:00	DEBUG	events	Scaling down cloudflared successful	{"type": "Normal", "object": {"kind":"Tunnel","namespace":"default","name":"adamrummer-com","uid":"c20c36ca-7fa3-408f-9e60-b679c7085b02","apiVersion":"networking.cfargotunnel.com/v1alpha2","resourceVersion":"2001"}, "reason": "Scaled"}
2025-05-08T23:35:54+01:00	INFO	starting deletion cycle	{"controller": "tunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "Tunnel", "Tunnel": {"name":"adamrummer-com","namespace":"default"}, "namespace": "default", "name": "adamrummer-com", "reconcileID": "e7e01bed-c04d-48d9-bde0-53744e6ce628"}
2025-05-08T23:35:54+01:00	INFO	In validation	{"controller": "tunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "Tunnel", "Tunnel": {"name":"adamrummer-com","namespace":"default"}, "namespace": "default", "name": "adamrummer-com", "reconcileID": "e7e01bed-c04d-48d9-bde0-53744e6ce628"}
2025-05-08T23:35:54+01:00	DEBUG	events	Starting Tunnel Deletion	{"type": "Normal", "object": {"kind":"Tunnel","namespace":"default","name":"adamrummer-com","uid":"c20c36ca-7fa3-408f-9e60-b679c7085b02","apiVersion":"networking.cfargotunnel.com/v1alpha2","resourceVersion":"2001"}, "reason": "Deleting"}
2025-05-08T23:35:54+01:00	INFO	no TXT records returned for fqdn	{"controller": "tunnelbinding", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "TunnelBinding", "TunnelBinding": {"name":"ingress-nginx","namespace":"default"}, "namespace": "default", "name": "ingress-nginx", "reconcileID": "72ca0583-cfc1-4a8c-aec0-c39310e4d300", "fqdn": "*.adamrummer.com"}
2025-05-08T23:35:55+01:00	INFO	no records returned for fqdn	{"controller": "tunnelbinding", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "TunnelBinding", "TunnelBinding": {"name":"ingress-nginx","namespace":"default"}, "namespace": "default", "name": "ingress-nginx", "reconcileID": "72ca0583-cfc1-4a8c-aec0-c39310e4d300", "fqdn": "*.adamrummer.com"}
2025-05-08T23:35:55+01:00	ERROR	Error fetching DNS record	{"controller": "tunnelbinding", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "TunnelBinding", "TunnelBinding": {"name":"ingress-nginx","namespace":"default"}, "namespace": "default", "name": "ingress-nginx", "reconcileID": "72ca0583-cfc1-4a8c-aec0-c39310e4d300", "Hostname": "*.adamrummer.com", "error": "no records returned"}
github.com/adyanth/cloudflare-operator/internal/controller.(*TunnelBindingReconciler).deleteDNSLogic
	/Users/adam/code/github.com/adyanth/cloudflare-operator/internal/controller/tunnelbinding_controller.go:370
github.com/adyanth/cloudflare-operator/internal/controller.(*TunnelBindingReconciler).deletionLogic
	/Users/adam/code/github.com/adyanth/cloudflare-operator/internal/controller/tunnelbinding_controller.go:230
github.com/adyanth/cloudflare-operator/internal/controller.(*TunnelBindingReconciler).Reconcile
	/Users/adam/code/github.com/adyanth/cloudflare-operator/internal/controller/tunnelbinding_controller.go:174
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
	/Users/adam/.cache/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/Users/adam/.cache/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:334
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/Users/adam/.cache/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:294
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
	/Users/adam/.cache/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:255
2025-05-08T23:35:55+01:00	DEBUG	events	Error fetching DNS record	{"type": "Warning", "object": {"kind":"TunnelBinding","namespace":"default","name":"ingress-nginx","uid":"512c1f11-02dc-427c-a94e-20d2384bc8ad","apiVersion":"networking.cfargotunnel.com/v1alpha1","resourceVersion":"2002"}, "reason": "FailedDeletingDns"}
2025-05-08T23:35:55+01:00	DEBUG	events	Finalizer removed	{"type": "Normal", "object": {"kind":"TunnelBinding","namespace":"default","name":"ingress-nginx","uid":"512c1f11-02dc-427c-a94e-20d2384bc8ad","apiVersion":"networking.cfargotunnel.com/v1alpha1","resourceVersion":"2002"}, "reason": "FinalizerUnset"}
2025-05-08T23:35:55+01:00	INFO	TunnelBinding deleted, nothing to do	{"controller": "tunnelbinding", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "TunnelBinding", "TunnelBinding": {"name":"ingress-nginx","namespace":"default"}, "namespace": "default", "name": "ingress-nginx", "reconcileID": "ffaf97d1-676c-4680-8371-8fe4fc58d0bd"}
2025-05-08T23:35:56+01:00	INFO	Tunnel deleted	{"controller": "tunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "Tunnel", "Tunnel": {"name":"adamrummer-com","namespace":"default"}, "namespace": "default", "name": "adamrummer-com", "reconcileID": "e7e01bed-c04d-48d9-bde0-53744e6ce628", "tunnelID": "8447f5e8-161d-407e-a721-5eda624d3600"}
2025-05-08T23:35:56+01:00	DEBUG	events	Tunnel deletion successful	{"type": "Normal", "object": {"kind":"Tunnel","namespace":"default","name":"adamrummer-com","uid":"c20c36ca-7fa3-408f-9e60-b679c7085b02","apiVersion":"networking.cfargotunnel.com/v1alpha2","resourceVersion":"2001"}, "reason": "Deleted"}
2025-05-08T23:35:56+01:00	DEBUG	events	Tunnel Finalizer removed	{"type": "Normal", "object": {"kind":"Tunnel","namespace":"default","name":"adamrummer-com","uid":"c20c36ca-7fa3-408f-9e60-b679c7085b02","apiVersion":"networking.cfargotunnel.com/v1alpha2","resourceVersion":"2001"}, "reason": "FinalizerUnset"}
2025-05-08T23:35:56+01:00	ERROR	Reconciler error	{"controller": "tunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "Tunnel", "Tunnel": {"name":"adamrummer-com","namespace":"default"}, "namespace": "default", "name": "adamrummer-com", "reconcileID": "e7e01bed-c04d-48d9-bde0-53744e6ce628", "error": "Operation cannot be fulfilled on tunnels.networking.cfargotunnel.com \"adamrummer-com\": StorageError: invalid object, Code: 4, Key: /registry/networking.cfargotunnel.com/tunnels/default/adamrummer-com, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: c20c36ca-7fa3-408f-9e60-b679c7085b02, UID in object meta: "}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/Users/adam/.cache/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:347
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/Users/adam/.cache/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:294
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
	/Users/adam/.cache/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:255
2025-05-08T23:35:56+01:00	INFO	removing finalizer	{"controller": "tunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "Tunnel", "Tunnel": {"name":"adamrummer-com","namespace":"default"}, "namespace": "default", "name": "adamrummer-com", "reconcileID": "3ab2e802-ae85-479d-ace1-43a7a949458d", "finalizer": "cfargotunnel.com/finalizer", "object": "default/cloudflare-secrets"}
2025-05-08T23:35:56+01:00	ERROR	Reconciler error	{"controller": "tunnel", "controllerGroup": "networking.cfargotunnel.com", "controllerKind": "Tunnel", "Tunnel": {"name":"adamrummer-com","namespace":"default"}, "namespace": "default", "name": "adamrummer-com", "reconcileID": "7e29d1a4-44ed-4ed7-9c05-6def336a799f", "error": "Secret \"cloudflare-secrets\" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/Users/adam/.cache/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:347
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/Users/adam/.cache/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:294
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
	/Users/adam/.cache/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:255

Comment on lines 148 to 153
if apierrors.IsNotFound(err) {
// Tunnel object not found, could have been deleted after reconcile request.
// Owned objects are automatically garbage collected. For additional cleanup logic use finalizers.
// Return and don't requeue
r.log.Info("Tunnel deleted, nothing to do")
// Owned objects are automatically garbage collected.
objectClient, err := k8s.NewObjectClient(r.Client, &r.log)
if err != nil {
return ctrl.Result{}, err
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This logic is in the wrong place (ref: the error you saw). When this condition is hit, the tunnel is already removed from etcd.

What you need is, instead of the logic in both here and tunnel_controller.go, move this to

func cleanupTunnel(r GenericTunnelReconciler) (ctrl.Result, bool, error) {
if controllerutil.ContainsFinalizer(r.GetTunnel().GetObject(), tunnelFinalizer) {
// Run finalization logic. If the finalization logic fails,
// don't remove the finalizer so that we can retry during the next reconciliation.
r.GetLog().Info("starting deletion cycle")

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, derp. Thanks will fix this

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you get around to this? Would love to merge this in

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't sorry, life has been a lot this past month! Hoping to get to this next weekend

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All good. Life comes first!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just wanted to give you another heads up, might be another week

StringKe referenced this pull request in StringKe/cloudflare-operator Jan 7, 2026
Implement comprehensive Cloudflare Zero Trust Kubernetes operator with 18 CRDs:

**New CRDs:**
- AccessApplication: Zero Trust application definitions
- AccessGroup: Reusable access policy groups
- AccessIdentityProvider: IdP configurations (OIDC, SAML, GitHub, Azure AD)
- AccessServiceToken: Machine-to-machine authentication tokens
- VirtualNetwork: Cloudflare virtual networks for traffic isolation
- NetworkRoute: IP routes through tunnels to private networks
- PrivateService: Expose K8s Services via WARP private IPs
- GatewayRule: Gateway DNS/HTTP/network policies
- GatewayList: Lists for gateway policy rules
- GatewayConfiguration: Global gateway settings
- DeviceSettingsPolicy: WARP client settings and split tunnels
- DevicePostureRule: Device posture checks for Zero Trust
- DNSRecord: DNS record management
- WARPConnector: WARP connector deployments

**Enhancements:**
- Add EnableWarpRouting to Tunnel/ClusterTunnel for private network access
- Add cluster-resource-namespace flag with Downward API injection (PR #178)
- Store previous-hostnames in TunnelBinding annotation (PR #166)
- Fix Secret finalizer order in cleanupTunnel (PR #158)

**API Clients:**
- Access API: Applications, Groups, Identity Providers, Service Tokens
- Gateway API: Rules, Lists, Configurations
- Device API: Split Tunnel, Fallback Domains, Posture Rules
- Network API: Virtual Networks, Routes
- DNS API: Record management

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
StringKe referenced this pull request in StringKe/cloudflare-operator Jan 7, 2026
Release v0.14.0 with Zero Trust CRDs:
- 14 new CRDs for Access, Gateway, Device, and Network management
- WARP routing support for private network access
- Upstream PR fixes (#178, #166, #158)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Discussion on docs/deletion.md

2 participants