[BUG] Kubernetes label provider access crashes instead of being disabled

Describe the Bug

Using Tiny in a bare metal kubernetes cluster (stock v1.34.3), and just did an in-place version update with renovate and encountered the following error:

Failed to access Ingress API, Kubernetes label provider will be disabled error="ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:production:default" cannot list resource "ingresses" in API group "networking.k8s.io" at the cluster scope" api=networking.k8s.io/v1 stream=app

Followed by:
Failed to execute command: command tinyauth error: failed to bootstrap app: failed to initialize services: failed to get label provider: failed to invoke kubernetes service: could not build arguments for function

Instead of being disabled, it feels like a crash.

Solved with the following manifest that was not previously needed, so my problem is resolved but wanted to report it as a consideration for others who might be upgrading

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: tinyauth-ingress-reader
rules:

apiGroups: ["networking.k8s.io"]
resources: ["ingresses"]
verbs: ["get", "list", "watch"]

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: tinyauth-ingress-reader-binding
subjects:

kind: ServiceAccount
name: default
namespace: production
roleRef:
kind: ClusterRole
name: tinyauth-ingress-reader
apiGroup: rbac.authorization.k8s.io

How to Reproduce

Update in place from 5.0.3 to 5.1.0 in k8s

image: ghcr.io/tinyauthapp/tinyauth:v5.1.0
imagePullPolicy: IfNotPresent

Expected Behavior

In place upgrade runs instead of crashing.

Additional Context

I only use Tiny with NGINX Gateway Fabric and Gateway API, using the following snippet:

apiVersion: gateway.nginx.org/v1alpha1
kind: SnippetsFilter
metadata:
name: tinyauth-auth-filter
namespace: production
spec:
snippets:
- context: http.server.location
value: |
# Send auth subrequest
auth_request /tinyauth;

    # Intercept 401 and redirect directly via an inline error page target
    error_page 401 =302 https://tinyauth.[redacted]/login?redirect_uri=$scheme://$http_host$request_uri;

    # The internal validation location endpoint
    location = /tinyauth {
        internal;
        proxy_pass http://tinyauth.production.svc.cluster.local:3000/api/auth/nginx;
        proxy_pass_request_body off;
        proxy_set_header Content-Length "";
        proxy_set_header x-forwarded-proto $scheme;
        proxy_set_header x-forwarded-host $http_host;
        proxy_set_header x-forwarded-uri $request_uri;
    }

And then in my httproutes:

rules:

matches:
- path:
  type: PathPrefix
  value: /
  filters:
  - type: ExtensionRef
    extensionRef:
    group: gateway.nginx.org
    kind: SnippetsFilter
    name: tinyauth-auth-filter

Logs

2026-07-16T04:16:04Z INF Starting Tinyauth version: v5.1.0 stream=app
2026-07-16T04:16:04Z WRN Failed to access Ingress API, Kubernetes label provider will be disabled error="ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:production:default" cannot list resource "ingresses" in API group "networking.k8s.io" at the cluster scope" api=networking.k8s.io/v1 stream=app
Failed to execute command: command tinyauth error: failed to bootstrap app: failed to initialize services: failed to get label provider: failed to invoke kubernetes service: could not build arguments for function "github.com/tinyauthapp/tinyauth/internal/bootstrap".(*BootstrapApp).getLabelProvider.func1 (/tinyauth/internal/bootstrap/service_bootstrap.go:91): failed to build *service.KubernetesService: received non-nil error from function "github.com/tinyauthapp/tinyauth/internal/service".NewKubernetesService (/tinyauth/internal/service/kubernetes_service.go:60): failed to access ingress api: ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:production:default" cannot list resource "ingresses" in API group "networking.k8s.io" at the cluster scope

Operating System

No response

Browser

No response

Tinyauth Version

No response

Docker Version (if applicable)

No response

Human Written Confirmation

I confirm this issue was written by me and not generated by an LLM or AI assistant.

@tagdara

Hi @tagdara! I'm Dosu and I'm helping the Tinyauth team.

This does appear to be a bug introduced in v5.1.0. The log message says "Kubernetes label provider will be disabled" but the code actually returns an error that propagates up and crashes the bootstrap process instead of gracefully continuing without the provider.

The root cause is PR #936, which refactored the service bootstrap to use dependency injection and changed the error handling philosophy to fail-fast. In v5.0.3, the system used a more tolerant approach — logging warnings and continuing with degraded functionality. In v5.1.0, the NewKubernetesService function logs the warning but then immediately returns an error [1], which getLabelProvider propagates upward [2], crashing the app during bootstrap.

So the warning message is misleading — it promises graceful degradation but the code doesn't deliver it. Your workaround (adding the ClusterRole/ClusterRoleBinding for ingress read access) is the correct fix for now if you're running in-cluster, but the behavior should ideally match the log message and actually disable the provider instead of crashing.

Thanks for reporting this — it's a valid bug where the error handling doesn't match the stated intent.

To reply, just mention @dosu.

Docs are dead. Just use Dosu.

@tagdara

@tagdara can you please test with v5.1.1-rc.1? It should allow Tinyauth to start up even without any special Kubernetes configuration (disabling the Kubernetes access controls provider if it fails to initialize).

came here to report same on v5.0.7 -> v5.1.0 (traefik within k3s kubernetes)
will try v5.1.1-rc.1 and report back

v5.1.1-rc.1 deployment in kubernetes now successfully starts. still fails to access Ingress API.

include debug log below

Click to expand inline

$ kubectl  logs tinyauth-6f9d76fdc6-n9zpt
2026-07-16T17:58:31Z INF internal/bootstrap/app_bootstrap.go:88 > Starting Tinyauth version: v5.1.1-rc.1 stream=app
2026-07-16T17:58:31Z DBG internal/service/policy_engine.go:60 > Using 'allow' ACL policy: access to apps will be allowed by default unless explicitly blocked stream=app
2026-07-16T17:58:31Z DBG internal/service/policy_engine.go:73 > Registering ACL rule in policy engine rule=rule-user-allowed stream=app
2026-07-16T17:58:31Z DBG internal/service/policy_engine.go:73 > Registering ACL rule in policy engine rule=rule-oauth-group stream=app
2026-07-16T17:58:31Z DBG internal/service/policy_engine.go:73 > Registering ACL rule in policy engine rule=rule-ldap-group stream=app
2026-07-16T17:58:31Z DBG internal/service/policy_engine.go:73 > Registering ACL rule in policy engine rule=rule-auth-enabled stream=app
2026-07-16T17:58:31Z DBG internal/service/policy_engine.go:73 > Registering ACL rule in policy engine rule=rule-ip-allowed stream=app
2026-07-16T17:58:31Z DBG internal/service/policy_engine.go:73 > Registering ACL rule in policy engine rule=rule-ip-bypassed stream=app
2026-07-16T17:58:31Z DBG internal/bootstrap/service_bootstrap.go:83 > Using Kubernetes label provider stream=app
2026-07-16T17:58:31Z WRN internal/service/kubernetes_service.go:82 > Failed to access Ingress API, Kubernetes label provider will be disabled error="ingresses.networking.k8s.io is forbidden: User \"system:serviceaccount:default:default\" cannot list resource \"ingresses\" in API group \"networking.k8s.io\" at the cluster scope" api=networking.k8s.io/v1 stream=app
2026-07-16T17:58:31Z WRN internal/bootstrap/service_bootstrap.go:98 > Failed to invoke kubernetes service error="could not build arguments for function \"github.com/tinyauthapp/tinyauth/internal/bootstrap\".(*BootstrapApp).getLabelProvider.func1 (/tinyauth/internal/bootstrap/service_bootstrap.go:92): failed to build *service.KubernetesService: received non-nil error from function \"github.com/tinyauthapp/tinyauth/internal/service\".NewKubernetesService (/tinyauth/internal/service/kubernetes_service.go:60): failed to access ingress api: ingresses.networking.k8s.io is forbidden: User \"system:serviceaccount:default:default\" cannot list resource \"ingresses\" in API group \"networking.k8s.io\" at the cluster scope" stream=app
2026-07-16T17:58:31Z DBG internal/bootstrap/app_bootstrap.go:276 > Configured authentication provider provider=Local stream=app
2026-07-16T17:58:31Z DBG internal/bootstrap/app_bootstrap.go:307 > Starting database cleanup routine stream=app
2026-07-16T17:58:31Z DBG internal/bootstrap/app_bootstrap.go:312 > Starting heartbeat routine stream=app
2026-07-16T17:58:31Z INF internal/bootstrap/router_bootstrap.go:152 > Starting server on http://0.0.0.0:3000 stream=app

@idcrook

@idcrook can you try adding the cluster role and cluster role binding @tagdara mentioned above?

as it stands I don't have a setup for ACLs using tinyauth and traefik -- in thoses cases I other traefik middleware (IP whitelists, etc).
so I do not currently have a use case to do any kind of related testing.
Also the docs have no traefik + kubernetes examples for tinyauth, so not even sure how the kubernetes annotations(?) would look...

@idcrook

@idcrook since Tinyauth can start with Kubernetes failed, this issue is fixed. However, you are correct that there is not any documentation on how to setup ACLs. But, here is a small example on how it would work:

You firstly need to create a service account for Tinyauth:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: tinyauth-sa
  namespace: tinyauth

Make sure it's in the same namespace as your deployment and that it's included in the Tinyauth deployment. Then, you need to create a cluster role defining what Tinyauth should be able to access:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: tinyauth-ingress
rules:
  - apiGroups: ["networking.k8s.io"]
    resources: ["ingresses"]
    verbs: ["get", "list", "watch"]

Finally, create a cluster role binding so that the Tinyauth service account has access to the ingresses:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tinyauth-ingress
subjects:
  - kind: ServiceAccount
    name: tinyauth-sa
    namespace: tinyauth
roleRef:
  kind: ClusterRole
  name: tinyauth-ingress
  apiGroup: rbac.authorization.k8s.io

After restarting the Tinyauth deployment, the Kubernetes service should start up just fine and you should be able to define ACLs in your ingresses like in the example below:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    tinyauth.apps.foo.config.domain: foo.example.com
  name: foo-ingress
  namespace: foo
...

Keep in mind that the domain you set in the annotations needs to match with at least one of the domains you have configured in the rules of the ingress.

@steveiliop56

@idcrook since Tinyauth can start with Kubernetes failed, this issue is fixed. However, you are correct that there is not any documentation on how to setup ACLs. But, here is a small example on how it would work:

@steveiliop56 tried your example on this fix release. This is not working for me with added clusterrole, etc.. didn't get to try annotations part yet.

perhaps I should file a new issue under the topic of "Document tinyauth ACLs using kubernetes and traefik" ?

Config additions

---
apiVersion: v1
kind: Namespace
metadata:
  name: tinyauth

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: tinyauth-sa
  namespace: tinyauth

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: tinyauth-ingress
rules:
  - apiGroups: ["networking.k8s.io"]
    resources: ["ingresses"]
    verbs: ["get", "list", "watch"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tinyauth-ingress-crb
subjects:
  - kind: ServiceAccount
    name: tinyauth-sa
    namespace: tinyauth
roleRef:
  kind: ClusterRole
  name: tinyauth-ingress
  apiGroup: rbac.authorization.k8s.io

Debug log output:

Click to expand inline

$ kubectl  --namespace tinyauth logs pod/tinyauth-6659f4b7b9-xfv9l
2026-07-17T16:11:47Z INF internal/bootstrap/app_bootstrap.go:88 > Starting Tinyauth version: v5.1.1-rc.1 stream=app
2026-07-17T16:11:47Z DBG internal/service/policy_engine.go:60 > Using 'allow' ACL policy: access to apps will be allowed by default unless explicitly blocked stream=app
2026-07-17T16:11:47Z DBG internal/service/policy_engine.go:73 > Registering ACL rule in policy engine rule=rule-user-allowed stream=app
2026-07-17T16:11:47Z DBG internal/service/policy_engine.go:73 > Registering ACL rule in policy engine rule=rule-oauth-group stream=app
2026-07-17T16:11:47Z DBG internal/service/policy_engine.go:73 > Registering ACL rule in policy engine rule=rule-ldap-group stream=app
2026-07-17T16:11:47Z DBG internal/service/policy_engine.go:73 > Registering ACL rule in policy engine rule=rule-auth-enabled stream=app
2026-07-17T16:11:47Z DBG internal/service/policy_engine.go:73 > Registering ACL rule in policy engine rule=rule-ip-allowed stream=app
2026-07-17T16:11:47Z DBG internal/service/policy_engine.go:73 > Registering ACL rule in policy engine rule=rule-ip-bypassed stream=app
2026-07-17T16:11:47Z DBG internal/bootstrap/service_bootstrap.go:83 > Using Kubernetes label provider stream=app
2026-07-17T16:11:47Z WRN internal/service/kubernetes_service.go:82 > Failed to access Ingress API, Kubernetes label provider will be disabled error="ingresses.networking.k8s.io is forbidden: User \"system:serviceaccount:tinyauth:default\" cannot list resource \"ingresses\" in API group \"networking.k8s.io\" at the cluster scope" api=networking.k8s.io/v1 stream=app
2026-07-17T16:11:47Z WRN internal/bootstrap/service_bootstrap.go:98 > Failed to invoke kubernetes service error="could not build arguments for function \"github.com/tinyauthapp/tinyauth/internal/bootstrap\".(*BootstrapApp).getLabelProvider.func1 (/tinyauth/internal/bootstrap/service_bootstrap.go:92): failed to build *service.KubernetesService: received non-nil error from function \"github.com/tinyauthapp/tinyauth/internal/service\".NewKubernetesService (/tinyauth/internal/service/kubernetes_service.go:60): failed to access ingress api: ingresses.networking.k8s.io is forbidden: User \"system:serviceaccount:tinyauth:default\" cannot list resource \"ingresses\" in API group \"networking.k8s.io\" at the cluster scope" stream=app
2026-07-17T16:11:47Z DBG internal/bootstrap/app_bootstrap.go:276 > Configured authentication provider provider=Local stream=app
2026-07-17T16:11:47Z WRN internal/bootstrap/router_bootstrap.go:42 > Trusted proxies are not configured, IP access controls will NOT work stream=app
2026-07-17T16:11:47Z DBG internal/bootstrap/app_bootstrap.go:307 > Starting database cleanup routine stream=app
2026-07-17T16:11:47Z DBG internal/bootstrap/app_bootstrap.go:312 > Starting heartbeat routine stream=app
2026-07-17T16:11:47Z INF internal/bootstrap/router_bootstrap.go:152 > Starting server on http://0.0.0.0:3000 stream=app

$ kubectl  --namespace tinyauth get all
NAME                            READY   STATUS    RESTARTS   AGE
pod/tinyauth-6659f4b7b9-pdtqs   1/1     Running   0          7m49s

NAME                   TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/tinyauth-svc   ClusterIP   10.43.149.96   <none>        3000/TCP   20m

NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/tinyauth   1/1     1            1           20m

NAME                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/tinyauth-6659f4b7b9   1         1         1       20m

$ kubectl  --namespace tinyauth describe clusterrolebindings.rbac.authorization.k8s.io   tinyauth-ingress
Name:         tinyauth-ingress
Labels:       <none>
Annotations:  <none>
Role:
  Kind:  ClusterRole
  Name:  tinyauth-ingress
Subjects:
  Kind            Name         Namespace
  ----            ----         ---------
  ServiceAccount  tinyauth-sa  tinyauth

$ kubectl  --namespace tinyauth describe clusterrole  tinyauth-ingress
Name:         tinyauth-ingress
Labels:       <none>
Annotations:  <none>
PolicyRule:
  Resources                    Non-Resource URLs  Resource Names  Verbs
  ---------                    -----------------  --------------  -----
  ingresses.networking.k8s.io  []                 []              [get list watch]

@steveiliop56

@steveiliop56 I think I spotted the problem. In error in debug log

User \"system:serviceaccount:tinyauth:default\" is denied permission

However, checking permissions on my cluster, the service account is another name that we created:

$ kubectl auth can-i list ingresses --as=system:serviceaccount:tinyauth:tinyauth-sa -n tinyauth
yes
$ kubectl auth can-i list ingresses --as=system:serviceaccount:tinyauth:default -n tinyauth
no
$ kubectl auth can-i list ingresses --as=system:serviceaccount:tinyauth:tinyauth-sa -n default
yes

So it is the "system:serviceaccount:tinyauth:default" service account that is surfacing the issue.

@steveiliop56

@steveiliop56 I updated example with example that matches original above and test release now starts up without error

---
apiVersion: v1
kind: Namespace
metadata:
  name: tinyauth

# ---
# apiVersion: v1
# kind: ServiceAccount
# metadata:
#   name: tinyauth-sa
#   namespace: tinyauth

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: tinyauth-ingress
rules:
  - apiGroups: ["networking.k8s.io"]
    resources: ["ingresses"]
    verbs: ["get", "list", "watch"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tinyauth-ingress
  namespace: tinyauth

subjects:
  - kind: ServiceAccount
    # name: tinyauth-sa
    name: default
    namespace: tinyauth
roleRef:
  kind: ClusterRole
  name: tinyauth-ingress
  apiGroup: rbac.authorization.k8s.io

debug log output

$ kubectl  --namespace tinyauth logs pod/tinyauth-6659f4b7b9-x7q6m
2026-07-17T16:52:27Z INF internal/bootstrap/app_bootstrap.go:88 > Starting Tinyauth version: v5.1.1-rc.1 stream=app
2026-07-17T16:52:27Z DBG internal/service/policy_engine.go:60 > Using 'allow' ACL policy: access to apps will be allowed by default unless explicitly blocked stream=app
2026-07-17T16:52:27Z DBG internal/service/policy_engine.go:73 > Registering ACL rule in policy engine rule=rule-user-allowed stream=app
2026-07-17T16:52:27Z DBG internal/service/policy_engine.go:73 > Registering ACL rule in policy engine rule=rule-oauth-group stream=app
2026-07-17T16:52:27Z DBG internal/service/policy_engine.go:73 > Registering ACL rule in policy engine rule=rule-ldap-group stream=app
2026-07-17T16:52:27Z DBG internal/service/policy_engine.go:73 > Registering ACL rule in policy engine rule=rule-auth-enabled stream=app
2026-07-17T16:52:27Z DBG internal/service/policy_engine.go:73 > Registering ACL rule in policy engine rule=rule-ip-allowed stream=app
2026-07-17T16:52:27Z DBG internal/service/policy_engine.go:73 > Registering ACL rule in policy engine rule=rule-ip-bypassed stream=app
2026-07-17T16:52:27Z DBG internal/bootstrap/service_bootstrap.go:83 > Using Kubernetes label provider stream=app
2026-07-17T16:52:27Z DBG internal/service/kubernetes_service.go:86 > Successfully accessed Ingress API, starting watcher api=networking.k8s.io/v1 stream=app
2026-07-17T16:52:27Z DBG internal/service/kubernetes_service.go:101 > Kubernetes label provider started successfully stream=app
2026-07-17T16:52:27Z DBG internal/bootstrap/app_bootstrap.go:276 > Configured authentication provider provider=Local stream=app
2026-07-17T16:52:27Z WRN internal/bootstrap/router_bootstrap.go:42 > Trusted proxies are not configured, IP access controls will NOT work stream=app
2026-07-17T16:52:27Z DBG internal/bootstrap/app_bootstrap.go:307 > Starting database cleanup routine stream=app
2026-07-17T16:52:27Z DBG internal/bootstrap/app_bootstrap.go:312 > Starting heartbeat routine stream=app
2026-07-17T16:52:27Z INF internal/bootstrap/router_bootstrap.go:152 > Starting server on http://0.0.0.0:3000 stream=app
2026-07-17T16:52:27Z DBG internal/service/kubernetes_service.go:290 > Resync complete api=networking.k8s.io/v1 count=0 stream=app
2026-07-17T16:52:27Z DBG internal/service/kubernetes_service.go:355 > Watcher started successfully api=networking.k8s.io/v1 stream=app
2026-07-17T16:52:28Z DBG internal/middleware/zerolog_middleware.go:77 > Request address=10.42.1.1:48662 client_ip=10.42.1.1 latency="148.611µs" method=GET path=/api/healthz status=200 stream=http

move tinyauth back into own namespace

The reason it didn't work is because you probably didn't add the service account to your deployment.

The reason it didn't work is because you probably didn't add the service account to your deployment.

Sure enough. adding service account into deployment also allows kubernetes API ingress API to load.

idcrook/kubernetes-homespun@f0a81eb#diff-cd7de2a74245aefe5f4fee1f96f37447fcb0bfdafac101b8ed6906cdde2d4c86R59

Navigation Menu

Repository navigation

[BUG] Kubernetes label provider access crashes instead of being disabled #1011

Describe the Bug

How to Reproduce

Expected Behavior

Additional Context

Logs

Operating System

Browser

Tinyauth Version

Docker Version (if applicable)

Human Written Confirmation

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[BUG] Kubernetes label provider access crashes instead of being disabled #1011

Description

Describe the Bug

How to Reproduce

Expected Behavior

Additional Context

Logs

Operating System

Browser

Tinyauth Version

Docker Version (if applicable)

Human Written Confirmation

Activity

dosubot commented on Jul 16, 2026

steveiliop56 commented on Jul 16, 2026

idcrook commented on Jul 16, 2026

idcrook commented on Jul 16, 2026

steveiliop56 commented on Jul 16, 2026

idcrook commented on Jul 16, 2026

steveiliop56 commented on Jul 16, 2026

idcrook commented on Jul 17, 2026

idcrook commented on Jul 17, 2026

idcrook commented on Jul 17, 2026

steveiliop56 commented on Jul 17, 2026

idcrook commented on Jul 17, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions