Wasm-shim

A Proxy-Wasm module written in Rust, acting as a shim between Envoy and both Rate-limiting and External Auth services.

Sample configuration

Following is a sample configuration used by the shim.

services:
  auth-service:
    type: dynamic
    endpoint: auth-cluster
    failureMode: deny
    timeout: 10ms
    grpcService: envoy.service.auth.v3.Authorization
    grpcMethod: Check
  ratelimit-service:
    type: dynamic
    endpoint: ratelimit-cluster
    failureMode: allow
    grpcService: envoy.service.ratelimit.v3.RateLimitService
    grpcMethod: ShouldRateLimit
  tracing-service:
    type: tracing
    endpoint: tracing-cluster
    failureMode: allow
observability:
  httpHeaderIdentifier: x-request-id
  defaultLevel: INFO
  tracing:
    service: tracing-service
actionSets:
  - name: rlp-ns-A/rlp-name-A
    routeRuleConditions:
      hostnames: [ "*.toystore.com" ]
      predicates:
      - request.url_path.startsWith("/get")
      - request.host == "test.toystore.com"
      - request.method == "GET"
    actions:
    - type: grpc
      var: auth_check
      service: auth-service
      predicate: "true"
      terminal: false
      messageBuilder: "envoy.service.auth.v3.CheckRequest { ... }"
      onReply:
      - type: deny
        predicate: auth_check.status.code != 0
        terminal: true
        denyWith: "DenyResponse{status: 403u}"
    - type: grpc
      var: rl_check
      service: ratelimit-service
      predicate: "true"
      terminal: false
      messageBuilder: "envoy.service.ratelimit.v3.RateLimitRequest { domain: 'my-domain', hits_addend: 1u, descriptors: [] }"
      onReply:
      - type: deny
        predicate: rl_check.overall_code == 2
        terminal: true
        denyWith: "DenyResponse{status: 429u}"

Top level fields:

Field	Required	Description
services	yes	Map of service name to service configuration, see Services
actionSets	yes	List of ActionSets evaluated, in order, against every request
observability	no	httpHeaderIdentifier, defaultLevel and tracing.service (name of a tracing-typed service) used for tracing
descriptorService	no	Name of a dynamic-typed service used to resolve rate-limit descriptor definitions. Defaults to kuadrant-operator-grpc

Services

Each entry under services configures an external service that actions can call:

type	Description
dynamic	Any gRPC service/method, set explicitly via grpcService/grpcMethod, for use with grpc typed actions
tracing	An OpenTelemetry (OTLP) collector, referenced from observability.tracing.service

Every service also accepts endpoint (the Envoy cluster name), failureMode (deny or allow, default deny) and timeout (a duration string, e.g. 10ms, default 20ms). dynamic services additionally require grpcService and grpcMethod fields specifying the gRPC service name and method.

Features

Actions

Each ActionSet's actions is a list of typed actions describing the request/response pipeline: issuing an arbitrary gRPC call, branching on its response, modifying headers, denying a request, or storing data for later CEL expressions:

actions:
- type: grpc
  predicate: request.method == 'GET'
  terminal: false
  var: rl_check
  service: ratelimit-service
  messageBuilder: "envoy.service.ratelimit.v3.RateLimitRequest { domain: 'my-domain' }"
  onReply:
  - type: deny
    predicate: rl_check.overall_code == 2
    terminal: true
    denyWith: "DenyResponse{status: 429u}"
  - type: headers
    predicate: "true"
    terminal: false
    target: response
    headers: rl_check.response_headers_to_add

Fields common to every typed action:

Field	Description
type	Selects the operation: grpc, deny, headers, store or fail (see below)
predicate	CEL predicate. The action only runs when this evaluates to true
terminal	When true, no further actions in the ActionSet are evaluated after this one
isGuard	Defaults to true. When true, later filter phases wait for this action to complete before continuing
execution	parallel (default) or sequential. A sequential action waits for all prior actions to complete and blocks subsequent actions until it finishes
sources	Names (var) of other grpc actions whose response this action's expressions may reference

Operation-specific fields:

type	Fields	Description
grpc	var, service, messageBuilder, onReply, label	Calls service with a message built from the messageBuilder CEL expression, storing the response under var. onReply is a list of typed actions evaluated once the response arrives
deny	denyWith	Ends request processing with the direct response built from the denyWith CEL expression, evaluating to a DenyResponse{status, headers, body} value
headers	target (request or response), headers	Adds/modifies target headers with the list produced by evaluating headers
store	path, value, exportToHost	Stores the evaluated value under path for later CEL expressions; when exportToHost is true, it is also exported as dynamic metadata to Envoy
fail	logMessage	Logs logMessage and fails the action

CEL Predicates and Expression

routeRuleConditions's predicates are expressed in Common Expression Language (CEL). Predicates evaluating to a bool value, while Expression, used for passing data to a service, evaluate to some Value.

These expression can operate on the data made available to them through the Well Known Attributes, see below

Custom CEL Functions

requestBodyJSON(json_pointer)

Parses request body as json and looks up a value by a JSON Pointer. JSON Pointer defines a string syntax for identifying a specific value within a JavaScript Object Notation (JSON) document. A Pointer is a Unicode string with the reference tokens separated by /. For more information read RFC6901.

If the request body is not a valid JSON, the function returns evaluation error. If there is no such value, the function returns evaluation error. If the value is found, it returns the value as a CEL Value.

Example:

when the request body is:

{
  "my": {
    "value": "hello",
    "list": ["a", "b", "c"]
  }
}

and the expression is:

data:
- expression:
    key: my_value
    value: requestBodyJSON('/my/value')

it evaluates to: "hello" CEL value. Similarly,

requestBodyJSON('/my/list/1') evaluates to "b" CEL value.

requestBodyJSON('/a/b/c') evaluates to Null CEL value.

It can also be used in predicates:

predicates:
- requestBodyJSON('/my/value') == 'hello'

responseBodyJSON(json_pointer)

Parses response body as json and looks up a value by a JSON Pointer. JSON Pointer defines a string syntax for identifying a specific value within a JavaScript Object Notation (JSON) document. A Pointer is a Unicode string with the reference tokens separated by /. For more information read RFC6901.

If the response body is not a valid JSON, the function returns evaluation error. If there is no such value, the function returns evaluation error. If the value is found, it returns the value as a CEL Value.

Example:

when the response body is:

{
  "my": {
    "value": "hello",
    "list": ["a", "b", "c"]
  }
}

and the expression is:

data:
- expression:
    key: my_value
    value: responseBodyJSON('/my/value')

it evaluates to: "hello" CEL value. Similarly,

responseBodyJSON('/my/list/1') evaluates to "b" CEL value.

responseBodyJSON('/a/b/c') evaluates to Null CEL value.

It can also be used in predicates:

predicates:
- responseBodyJSON('/my/value') == 'hello'

Well Known Attributes

Attribute	Description
Envoy Attributes	Contextual properties provided by Envoy during request and connection processing
source.remote_address	This attribute evaluates to the trusted client address (IP address without port) as it is being defined by Envoy Doc
auth.*	Data made available by the authentication service to the ActionSet's pipeline

Metrics

The WASM module exposes the following Prometheus-compatible metrics via Envoy:

Metric Name	Type	Description
kuadrant.configs	Counter	Number of times the plugin configuration has been loaded
kuadrant.hits	Counter	Number of requests that matched an action set
kuadrant.misses	Counter	Number of requests that did not match any action set
kuadrant.allowed	Counter	Number of requests allowed after evaluation
kuadrant.denied	Counter	Number of requests denied as a result of actions
kuadrant.errors	Counter	Number of errors encountered during request processing

These metrics are automatically exposed through Envoy's stats endpoint and can be scraped by Prometheus or other monitoring systems. To view metrics, access Envoy's admin interface (typically at :8001/stats/prometheus).

Building

Prerequisites:

• Install wasm32-wasip1 build target

rustup target add wasm32-wasip1

Build the WASM module

make build

Build the WASM module in release mode

make build BUILD=release

Build the WASM module with features

make build FEATURES=debug-host-behaviour

Testing

cargo test

Running local development environment (kind)

docker is required.

Run local development environment

make local-setup

This deploys a local kubernetes cluster using kind, with the local build of wasm-shim mapped to the envoy container. An echo API as well as limitador, authorino, and some test policies are configured.

To expose the envoy endpoint run the following:

kubectl port-forward --namespace kuadrant-system deployment/envoy 8000:8000

There is then a single auth action set defined for e2e testing:

• auth-a which defines auth is required for requests to /get for the AuthConfig with effective-route-1

curl -H "Host: test.a.auth.com" http://127.0.0.1:8000/get -i
# HTTP/1.1 401 Unauthorized

curl -H "Host: test.a.auth.com" -H "Authorization: APIKEY IAMALICE" http://127.0.0.1:8000/get -i
# HTTP/1.1 200 OK

And some rate limit action sets defined for e2e testing:

• rlp-a: Invalid expression looking up unknown host property. As failure mode is deny, expect a 500 Internal Server Error.

curl -H "Host: test.a.rlp.com" http://127.0.0.1:8000/get -i
# HTTP/1.1 500 Internal Server Error

• rlp-b: Conditions do not match. Hence, rate limiting service should not be called.

curl -H "Host: test.b.rlp.com" http://127.0.0.1:8000/get -i
# HTTP/1.1 200 OK

• rlp-c: Descriptor entries from multiple data items should be generated. Hence, rate limiting service should be called.

curl -H "Host: test.c.rlp.com" -H "x-forwarded-for: 50.0.0.1" -H "my-custom-header-01: my-custom-header-value-01" -H "x-dyn-user-id: bob" http://127.0.0.1:8000/get -i
# HTTP/1.1 200 OK

Check limitador logs for received descriptor entries.

kubectl logs -f deployment/limitador-limitador -n kuadrant-system

The expected descriptor entries:

Entry { key: "limit_to_be_activated", value: "1" }

Entry { key: "source.address", value: "50.0.0.1:0" }

Entry { key: "request.headers.my-custom-header-01", value: "my-custom-header-value-01" }

• rlp-d: source.address is rate limited appropriately.

Alice (IP: 40.0.0.1) has 2 requests per 10 seconds:

while :; do curl --write-out '%{http_code}\n' --silent --output /dev/null  -H "X-Forwarded-For: 40.0.0.1" -H "Host: test.d.rlp.com" http://127.0.0.1:8000/get | grep -E --color "\b(429)\b|$"; sleep 1; done

Bob (IP: 50.0.0.1) with privileged IP 50.0.0.1 does not get rate limited:

while :; do curl --write-out '%{http_code}\n' --silent --output /dev/null  -H "X-Forwarded-For: 50.0.0.1" -H "Host: test.d.rlp.com" http://127.0.0.1:8000/get | grep -E --color "\b(429)\b|$"; sleep 1; done

• multi-a which defines two actions for authenticated ratelimiting.

curl -H "Host: test.a.multi.com" http://127.0.0.1:8000/get -i
# HTTP/1.1 401 Unauthorized

Alice has 5 requests per 10 seconds:

while :; do curl --write-out '%{http_code}\n' --silent --output /dev/null -H "Authorization: APIKEY IAMALICE" -H "Host: test.a.multi.com" http://127.0.0.1:8000/get | grep -E --color "\b(429)\b|$"; sleep 1; done

Bob has 2 requests per 10 seconds:

while :; do curl --write-out '%{http_code}\n' --silent --output /dev/null -H "Authorization: APIKEY IAMBOB" -H "Host: test.a.multi.com" http://127.0.0.1:8000/get | grep -E --color "\b(429)\b|$"; sleep 1; done

To rebuild and deploy to the cluster:

make build local-rollout

Stop and clean up resources:

make local-cleanup

License

Apache 2.0 License