Skip to content

feat(upstream): verify the upstream certificate against configurable CAs #13863

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
nic-6443 merged 4 commits into from
Aug 27, 2026
Merged

feat(upstream): verify the upstream certificate against configurable CAs #13863

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
5794995
feat(upstream): verify the upstream certificate against configurable CAs
nic-6443 Aug 21, 2026
6f5d40d
chore: bump APISIX-Runtime to 1.3.17
nic-6443 Aug 21, 2026
40e4742
test(upstream): cover ca_certs without a client certificate
nic-6443 Aug 21, 2026
39fd13d
fix(balancer): key the upstream keepalive pool by verification policy
nic-6443 Aug 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .requirements
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,5 +17,5 @@

APISIX_PACKAGE_NAME=apisix

APISIX_RUNTIME=1.3.16
APISIX_RUNTIME=1.3.17
APISIX_DASHBOARD_COMMIT=045e3142867e3b7d5d1b8ec40bf8f66a7ce24a64
35 changes: 31 additions & 4 deletions apisix/balancer.lua
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,8 @@ local set_more_tries = balancer.set_more_tries
local get_last_failure = balancer.get_last_failure
local set_timeouts = balancer.set_timeouts
local ngx_now = ngx.now
local ngx_md5 = ngx.md5
local tostring = tostring

local module_name = "balancer"
local pickers = {}
Expand Down Expand Up @@ -369,6 +371,18 @@ end
_M.pick_server = pick_server


-- Keyed by the `ca_certs` array itself: a config update always rebuilds that
-- table, so a stale digest can never outlive the certificates it was made from.
local ca_certs_digest_cache = core.lrucache.new({
ttl = 300, count = 256,
})


local function ca_certs_digest(ca_certs)
return ngx_md5(core.table.concat(ca_certs, "\n"))
end


local set_current_peer
do
local pool_opt = {}
Expand Down Expand Up @@ -409,12 +423,25 @@ do
local sni = ctx.var.upstream_host
pool = pool .. "#" .. sni

local tls = up_conf.tls
-- separate the pool by client cert so referenced SSL objects
-- don't share a connection
if up_conf.tls and up_conf.tls.client_cert then
pool = pool .. "#" .. up_conf.tls.client_cert
elseif up_conf.tls and up_conf.tls.client_cert_id then
pool = pool .. "#" .. up_conf.tls.client_cert_id
if tls and tls.client_cert then
pool = pool .. "#" .. tls.client_cert
elseif tls and tls.client_cert_id then
pool = pool .. "#" .. tls.client_cert_id
end

-- and by the verification policy, which is applied while the
-- connection is being established: a pooled connection keeps
-- whatever policy it was handshaked under, so reusing it across
-- policies would skip the verification the config asks for
if tls and (tls.verify ~= nil or tls.ca_certs) then
pool = pool .. "#" .. tostring(tls.verify)
if tls.ca_certs then
pool = pool .. "#" .. ca_certs_digest_cache(tls.ca_certs, nil,
ca_certs_digest, tls.ca_certs)
end
end
end
pool_opt.pool = pool
Expand Down
3 changes: 3 additions & 0 deletions apisix/cli/ngx_tpl.lua
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1002,6 +1002,9 @@ http {
grpc_set_header Content-Type application/grpc;
grpc_set_header TE trailers;
grpc_socket_keepalive on;
# only consulted once upstream.tls.verify turns verification on;
# without it the certificate would be checked against "apisix_backend"
grpc_ssl_name $upstream_host;
grpc_pass $upstream_scheme://apisix_backend;

{% if enabled_plugins["proxy-mirror"] then %}
Expand Down
11 changes: 8 additions & 3 deletions apisix/schema_def.lua
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -444,9 +444,14 @@ local upstream_schema = {
client_key = private_key_schema,
verify = {
type = "boolean",
description = "Turn on server certificate verification, "..
"currently only kafka upstream is supported",
default = false,
description = "enable or disable upstream certificate verification, " ..
"fall back to the nginx configuration when not set",
},
ca_certs = {
type = "array",
description = "CA certificates used to verify the upstream certificate",
minItems = 1,
items = certificate_scheme,
},
},
dependencies = {
Expand Down
110 changes: 109 additions & 1 deletion apisix/upstream.lua
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ local discovery = require("apisix.discovery.init").discovery
local upstream_util = require("apisix.utils.upstream")
local apisix_ssl = require("apisix.ssl")
local resource = require("apisix.resource")
local openssl_x509 = require("resty.openssl.x509")
local openssl_x509_store = require("resty.openssl.x509.store")
local error = error
local tostring = tostring
local ipairs = ipairs
Expand All @@ -32,14 +34,95 @@ local upstreams
local healthcheck_manager

local set_upstream_tls_client_param
local set_upstream_ssl_verify
local set_upstream_ssl_trusted_store
local ok, apisix_ngx_upstream = pcall(require, "resty.apisix.upstream")
if ok then
set_upstream_tls_client_param = apisix_ngx_upstream.set_cert_and_key
else
set_upstream_ssl_verify = apisix_ngx_upstream.set_ssl_verify
set_upstream_ssl_trusted_store = apisix_ngx_upstream.set_ssl_trusted_store
end
-- guard each function independently: an older runtime may expose the module
-- (set_cert_and_key) without the newer upstream TLS C-APIs
if not set_upstream_tls_client_param then
set_upstream_tls_client_param = function ()
return nil, "need to build APISIX-Runtime to support upstream mTLS"
end
end
if not set_upstream_ssl_verify then
set_upstream_ssl_verify = function ()
return nil, "need to build APISIX-Runtime to support upstream certificate verification"
end
end
if not set_upstream_ssl_trusted_store then
set_upstream_ssl_trusted_store = function ()
return nil, "need to build APISIX-Runtime to support upstream CA certificates"
end
end


-- Keyed by the `ca_certs` array itself: a config update always rebuilds that
-- table, so a stale store can never outlive the certificates it was built from.
local trusted_store_cache = core.lrucache.new({
ttl = 300, count = 256,
})


local function create_trusted_store(ca_certs)
local store, err = openssl_x509_store.new()
if not store then
return nil, err
end

for _, ca_cert in ipairs(ca_certs) do
local x509, err = openssl_x509.new(ca_cert, "PEM")
if not x509 then
return nil, err
end

local ok, err = store:add(x509)
if not ok then
return nil, err
end
end

return store
end


-- Apply upstream.tls.verify / upstream.tls.ca_certs to the current request.
-- Both settings live in the apisix-nginx-module request context, which is wiped
-- by the internal redirect to @grpc_pass, so grpcs has to apply them again from
-- `grpc_access_phase` just like the client certificate does.
local function set_upstream_ssl_opts(up_conf)
local tls = up_conf.tls
if not tls then
return true
end

if tls.verify ~= nil then
local ok, err = set_upstream_ssl_verify(tls.verify)
if not ok then
return nil, err
end
end

if tls.ca_certs then
local store, err = trusted_store_cache(tls.ca_certs, up_conf.resource_version,
create_trusted_store, tls.ca_certs)
if not store then
return nil, err
end

local ok, err = set_upstream_ssl_trusted_store(store)
if not ok then
return nil, err
end
end

return true
end


local set_stream_upstream_tls
local set_stream_upstream_cert_and_key
Expand Down Expand Up @@ -356,6 +439,15 @@ function _M.set_by_route(route, api_ctx)
local checker = healthcheck_manager.fetch_checker(up_conf.resource_key, resource_version)
api_ctx.up_checker = checker
local scheme = up_conf.scheme
if scheme == "https" then
-- grpcs applies these in `set_grpcs_upstream_param` instead, after the
-- internal redirect that drops the settings made here
local ok, err = set_upstream_ssl_opts(up_conf)
if not ok then
return 503, err
end
end

local tls_has_cert = up_conf.tls and (up_conf.tls.client_cert or up_conf.tls.client_cert_id)
if (scheme == "https" or scheme == "grpcs") and tls_has_cert then
local client_cert, client_key
Expand Down Expand Up @@ -395,6 +487,13 @@ end


function _M.set_grpcs_upstream_param(ctx)
if ctx.upstream_conf and ctx.upstream_conf.scheme == "grpcs" then
local ok, err = set_upstream_ssl_opts(ctx.upstream_conf)
if not ok then
return 503, err
end
end

if ctx.upstream_grpcs_cert then
local cert = ctx.upstream_grpcs_cert
local key = ctx.upstream_grpcs_key
Expand Down Expand Up @@ -517,6 +616,15 @@ local function check_upstream_conf(in_dp, conf)
end
end

if conf.tls and conf.tls.ca_certs then
for _, ca_cert in ipairs(conf.tls.ca_certs) do
local ok, err = apisix_ssl.validate(ca_cert)
if not ok then
return false, err
end
end
end

if conf.type ~= "chash" then
return true
end
Expand Down
6 changes: 3 additions & 3 deletions ci/linux-install-openresty.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,19 +61,19 @@ else
sudo apt-get -y update --fix-missing
sudo apt-get install -y build-essential gcc g++ cpanminus libxml2-dev libxslt-dev

if [ "$APISIX_RUNTIME" != "1.3.16" ]; then
if [ "$APISIX_RUNTIME" != "1.3.17" ]; then
echo "Please update the apisix-runtime-debug checksum for APISIX_RUNTIME=$APISIX_RUNTIME" >&2
exit 1
fi

case "$ARCH" in
x86_64|amd64)
DEB_ARCH="amd64"
EXPECTED_SHA256="a56f0adc9bf6f6a491f7548df4f8e45fa3df3dd5e209d4a2ba5341b66eb7e060"
EXPECTED_SHA256="d60067ba7a89cab6fca8e70994e4158fa8c414a569406e1692998be2567832a6"
;;
arm64|aarch64)
DEB_ARCH="arm64"
EXPECTED_SHA256="b645ee4f5ea36d26aaacb1b1c8278d89756b0b8448701ec561ab982b4768204a"
EXPECTED_SHA256="2db6619c6fa31128e7ea45b2cdcc56dd26d2b11d7b4a68600f4f04746cb34766"
;;
*)
echo "Unsupported architecture: $ARCH" >&2
Expand Down
2 changes: 1 addition & 1 deletion docs/en/latest/FAQ.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -728,7 +728,7 @@ The `ssls` is managed through the `/apisix/admin/ssls` API. It's used for managi

The `tls.client_cert`, `tls.client_key`, and `tls.client_cert_id` in upstream are used for mTLS communication with the upstream.

The `ssl_trusted_certificate` in `config.yaml` configures a trusted CA certificate. It is used for verifying some certificates signed by private authorities within APISIX, to avoid APISIX rejects the certificate. Note that it is not used to trust the certificates of APISIX upstream, because APISIX does not verify the legality of the upstream certificates. Therefore, even if the upstream uses an invalid TLS certificate, it can still be accessed without configuring a root certificate.
The `ssl_trusted_certificate` in `config.yaml` configures a trusted CA certificate. It is used for verifying some certificates signed by private authorities within APISIX, to avoid APISIX rejects the certificate. Note that APISIX does not verify the certificate of an upstream unless that upstream sets `tls.verify` to `true`, so by default an upstream using an invalid TLS certificate can still be accessed. Once `tls.verify` is enabled, the certificate is checked against the upstream's own `tls.ca_certs` if set, and against `ssl_trusted_certificate` otherwise.

## Where can I find more answers?

Expand Down
19 changes: 18 additions & 1 deletion docs/en/latest/admin-api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1019,7 +1019,8 @@ In addition to the equalization algorithm selections, Upstream also supports pas
| tls.client_cert | False, can't be used with `tls.client_cert_id` | HTTPS certificate | Sets the client certificate while connecting to a TLS Upstream. | |
| tls.client_key | False, can't be used with `tls.client_cert_id` | HTTPS certificate private key | Sets the client private key while connecting to a TLS Upstream. | |
| tls.client_cert_id | False, can't be used with `tls.client_cert` and `tls.client_key` | SSL | Set the referenced [SSL](#ssl) id. | |
| tls.verify | False, currently only kafka upstream is supported | Boolean | Turn on server certificate verification, currently only kafka upstream is supported. | |
| tls.verify | False | Boolean | Enables or disables verification of the Upstream certificate. Falls back to the nginx configuration when unset. Also used by the `kafka` scheme. | |
| tls.ca_certs | False | Array of HTTPS certificates | CA certificates used to verify the Upstream certificate, replacing the ones loaded from `ssl_trusted_certificate`. | |
| keepalive_pool.size | False | Auxiliary | Sets `keepalive` directive dynamically. | |
| keepalive_pool.idle_timeout | False | Auxiliary | Sets `keepalive_timeout` directive dynamically. | |
| keepalive_pool.requests | False | Auxiliary | Sets `keepalive_requests` directive dynamically. | |
Expand Down Expand Up @@ -1048,6 +1049,22 @@ To use mTLS to communicate with Upstream, you can use the `tls.client_cert/key`

Or you can reference SSL object by `tls.client_cert_id` to set SSL cert and key. The SSL object can be referenced only if the `type` field is `client`, otherwise the request will be rejected by APISIX. In addition, only `cert` and `key` will be used in the SSL object.

To verify the certificate presented by the Upstream, set `tls.verify` to `true`. Leaving it unset keeps the behaviour configured in nginx, which is off unless `proxy_ssl_verify` is turned on. The certificate is checked against the CA certificates in `tls.ca_certs`, or against `ssl_trusted_certificate` from `config.yaml` when `tls.ca_certs` is not set:

```json
{
"scheme": "https",
"type": "roundrobin",
"nodes": {
"127.0.0.1:8443": 1
},
"tls": {
"verify": true,
"ca_certs": ["<content of ca.crt>"]
}
}
```

To allow Upstream to have a separate connection pool, use `keepalive_pool`. It can be configured by modifying its child fields.

Example Configuration:
Expand Down
Loading
Loading