Istio/opa external CUSTOM Authorization policy causes knative activator request time out

[tonychue]

Expected Behavior
Trying to apply opa external authorizer as separate pod using CUSTOM authorization policy
Activator request keeps timing out
see link - https://istio.io/latest/blog/2021/better-external-authz/

Want to be able to authorize request using opa and rego policy defined in a bundle server running in Azure storage account

Actual Behavior
deploy a knative service, which runs fine.
deploy opa as a seperate pod option
inject istio to namespace using label istio-injection=enabled
inject opa to namespace as a namespace label - opa-istio-injection=enabled
Opa pods starts and runs okay as side car

Steps to Reproduce the Problem

1. deploy a knative service, which runs fine.
2. apply below envoy filter using target/label selector
   3.wait for pod to scale to zero, then the service is no longer recheable.

Steps to repro the issue

1. Create a namespace called opa-knative
2. Label namespace as istio-injection-=enabled
   3.deployment bundle configmap
3. deploy knative service application
4. deploy opa external auth
5. set namespace label to opa-istio-injection=enabled
6. deploy and configure istio configmap to inject opa ext endpoint
7. expose the endpoint through virtual service or ingress gateway
8. deploy exter auth with following config (CUSTOM)

Authorization Policy
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: httpbinary
namespace: opa-knative
spec:
selector:
matchLabels:
app: httpbinary
action: CUSTOM
provider:
name: "opa.opa-knative"
rules:

• to:
  • operation:
    notPaths: ["/ip"]

Configmap

apiVersion: v1
data:
mesh: |-
defaultConfig:
extensionProviders:
- name: "opa.opa-knative"
envoyExtAuthzGrpc:
service: "opa.opa-knative.svc.cluster.local"
port: "9191"

Additional Info
The request endpoint is as following
http://10.111.1000.100/headers

Error
Activator time out

Logs:
Activator log error
kubectl logs activator-dd544f9cc-rbqkf -n knative-serving | grep error

{"severity":"WARNING","timestamp":"2022-11-17T13:51:46.558659653Z","logger":"activator","caller":"net/revision_backends.go:342","message":"Failed probing pods","commit":"e82287d","knative.dev/controller":"activator","knative.dev/pod":"activator-dd544f9cc-rbqkf","knative.dev/key":"opa-knative/httpbinary-00001","curDests":{"ready":"10.244.4.128:8012","notReady":""},"error":"unexpected status code: want [200], got 403"}

[tonychue]

Found a workaround.
"activator","knative.dev/pod":"activator-dd544f9cc-rbqkf","knative.dev/key":"opa-knative/httpbinary-00001","curDests":
{"ready":"10.244.4.128:8012","notReady":""},"error":"unexpected status code: want [200], got 403"}
The activator probe keeps failing due to the opa policy constantly try to complete a health check as such it needs to confirm that the opa pods are healthy before the activator can send traffic.

so disabling the /healthz on the opa CUSTOM policy fixed the issue.