Skip to content

fix(consul): do not read the response body in the watcher threads - #14014

Open
nic-6443 wants to merge 1 commit into
apache:masterfrom
nic-6443:fix/consul-watch-headers-only
Open

nic-6443 wants to merge 1 commit into
apache:masterfrom
nic-6443:fix/consul-watch-headers-only

Conversation

@nic-6443

@nic-6443 nic-6443 commented Oct 8, 2026

Copy link
Copy Markdown
Member

Description

Consul discovery's connect() spawns a catalog watcher and a health watcher, waits for the first one to return and then kills both. The watchers used resty.consul, which reads the response body through resty.http, and resty.http reads a body in a child coroutine. If the watcher that gets killed is still reading its body (typically the health watch streaming a large /v1/health/state/any response while the catalog watch returns first), ngx.thread.kill does not cancel the read pending in that child coroutine. When the read completes later, it resumes connect() at whatever it is waiting on at that moment, usually the status-line receive of the following catalog fetch. That fetch then fails with:

fetch_services_from_server(): connect consul: ... by sub url: /catalog/services, got catalog result: ".../resty/http.lua:479: bad argument #1 to 'str_sub' (string expected, got boolean)"

and the round is retried with backoff. It is easy to hit when both indexes change together and the health response is large.

The watchers only need the status and the X-Consul-Index header. This PR makes them send the blocking query with resty.http request(), which does all of its I/O (connect, send, status line, headers) on the calling coroutine, and close the connection without reading the body. A killed watcher therefore never leaves a pending operation behind in a child coroutine. The connect() loop itself (spawn, wait, kill, re-arm, retry/backoff, non-keepalive mode) is unchanged, and the query parameters, token header and timeouts are the same as before.

Behavior notes:

  • watcher connections are closed after each query instead of going back to the keepalive pool, so each watch round opens two new connections to Consul;
  • when a watch fails, the error log now carries the status and headers of the response but no longer its body.

The new test t/discovery/consul-watch-race.t runs a mock Consul that bumps both indexes together, answers the catalog watch shortly after the health watch has started streaming a slow chunked body, and delays the non-blocking catalog read so that the health body finishes while the fetch is waiting. Before this change the test fails with the got boolean error and the route returns 503; after it, the services are fetched and the route returns 200.

Which issue(s) this PR fixes:

N/A

Checklist

  • I have explained the need for this PR and the problem it solves
  • I have explained the changes or the new features added to this PR
  • I have added tests corresponding to this change
  • I have updated the documentation to reflect this change
  • I have verified that this change is backward compatible (If not, please discuss on the APISIX mailing list first)

connect() spawns the catalog and health watchers, waits for the first one and
kills both. resty.consul reads the response body through resty.http, which
does that in a child coroutine. When the killed watcher was reading a body,
ngx.thread.kill does not cancel that child's pending socket read; its
completion later resumes connect() at an unrelated yield point, e.g. the
receive of the following catalog fetch, which then fails with
"bad argument #1 to 'str_sub' (string expected, got boolean)".

The watchers only need the status and the X-Consul-Index header. Read them
with resty.http request(), which does all its I/O on the calling coroutine,
and close the connection without reading the body.
Copilot AI balanced review requested due to automatic review settings October 8, 2026 07:01

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The focused implementation addresses the coroutine race while preserving watcher behavior and includes targeted regression coverage.

0 open findings

What changed in this PR

Prevents killed Consul watcher threads from leaving pending body reads that corrupt subsequent discovery requests.

Changes:

  • Reworks watchers to read only response status and headers.
  • Adds a regression test reproducing the watcher race.
File Description
apisix/​discovery/​consul/​client.lua Uses direct header-only HTTP requests for watchers.
t/​discovery/​consul-watch-race.t Tests concurrent index changes and slow response streaming.

🧠 Review effort: Balanced


Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants