From ed53664f2532ecf34848003de44f9d9e18c45e5d Mon Sep 17 00:00:00 2001 From: Aslak Knutsen Date: Thu, 27 Aug 2026 16:10:14 +0200 Subject: [PATCH 1/3] docs: propose pipeline-scoped filter task supervisor Filters have no lifecycle for work that must die with the pipeline, so overlay reload invented a private thread and azure_ad is copying it. Track the What/Why against praxis#1042. Signed-off-by: Aslak Knutsen --- proposals/01042_filter-task-supervisor.md | 207 ++++++++++++++++++++++ 1 file changed, 207 insertions(+) create mode 100644 proposals/01042_filter-task-supervisor.md diff --git a/proposals/01042_filter-task-supervisor.md b/proposals/01042_filter-task-supervisor.md new file mode 100644 index 0000000..190ccce --- /dev/null +++ b/proposals/01042_filter-task-supervisor.md @@ -0,0 +1,207 @@ +--- +discussion: https://github.com/praxis-proxy/ai/pull/815/changes#r3854446287 +issue: https://github.com/praxis-proxy/praxis/issues/1042 +status: proposed +repos: + - praxis + - ai + - extproc +authors: + - aslakknutsen +graduation_criteria: + - How? section with requirements and design + - Lifecycle (construct / start / cancel-on-pipeline-drop) agreed by stakeholders + - Fail-construction vs fail-closed spawn modes agreed + - Decision on whether intelligent_route overlay reload migrates in the same change +stakeholders: + - shaneutt + - alexsnaps + - szedan-rh +related: + - https://github.com/praxis-proxy/praxis/issues/1043 +--- + +# Pipeline-Scoped Filter Task Supervisor + +## What? + +`HttpFilter` / `TcpFilter` have no lifecycle for work +that must outlive a single request and die with the +pipeline. Filter construction is synchronous, often +runs with no Tokio runtime, and pipeline hot-reload +drops the old chain. + +Filters that need background work today mint a +private OS thread and a private current-thread +runtime, then cancel it in `Drop`. That is an +accident of missing API, not a design. + +This proposal adds a **pipeline-owned task +supervisor**: filters may start named background +tasks during pipeline build; those tasks are +cancelled when that pipeline is dropped (reload or +shutdown). Filters do not spawn threads or create +runtimes. + +Pingora `BackgroundService` stays for +**process-lifetime** work registered before +`server.run()` (listeners, admin HTTP). This +supervisor is **pipeline-lifetime**. Those are +different scopes. Config watchers, health-check +loops, and TLS cert watchers stay server-owned. + + +### Goals + +- Give filters a supported way to run background + work whose lifetime is the pipeline, not the + process and not the request. +- Keep `from_config` as YAML parse + validation. + Background work starts in a later pipeline-build + hook, so most filters stay unaware of the + supervisor. +- Cancel all tasks for a pipeline when that + pipeline is dropped, so reload cannot leak + refreshers or watchers. +- Distinguish **fail construction** (background + work never became ready; overlay watcher) from + **fail closed** (requests reject until work + succeeds; token refresh). +- Make spawn/runtime failure a pipeline-build + error, not a live filter that 503s forever. +- Stop new filters from copying the overlay-watcher + thread pattern. + +### Non-Goals + +- Replacing Pingora `BackgroundService` or + `server.add_service`. +- Moving server-owned loops (config watcher, health + checks, TLS cert watcher) onto this API. Those + are process/listener lifetime, not filter + lifetime. +- Making `from_config` async or requiring a Tokio + runtime at config parse time. +- Changing request-path `HttpFilter` methods. +- Specifying thread count, shared vs per-pipeline + runtime, or the exact trait shape. That is How? +- Shipping Azure AD / Entra token refresh. That + feature may *use* this API; it is not this API. + +## Why? + +### Motivation + +Praxis swaps filter pipelines at runtime. Anything +a filter starts in `from_config` must stop when +that pipeline is dropped, or reload leaks work and +credentials. + +The filter trait does not say that. So authors copy +the one working example: `intelligent_route` +overlay reload, which spawns +`routing-overlay-watcher` (OS thread + private +Tokio runtime + `CancellationToken` + bounded +join). That shape exists because overlay reload is +an inotify loop that must not block pipeline build. +It is also the *only* filter on `main` (core, ai, +extproc) that owns a thread. + +The next consumer does not look like inotify. An +Entra client-credentials filter is a timer plus one +HTTP POST, then inject `Authorization`. It still +cannot `tokio::spawn` (`from_config` is sync, often +off-runtime) and cannot register a Pingora +background service (process-lifetime, not torn down +on pipeline swap). Without a pipeline-scoped API, +it copies the overlay thread anyway — including the +weaker variant that logs spawn failure and still +constructs a filter that fail-closes forever. That +copy is already in flight as +[praxis-proxy/ai#815](https://github.com/praxis-proxy/ai/pull/815). + +Inventory today: + +| Owner | Lifetime | Mechanism | +| --- | --- | --- | +| `intelligent_route` (overlay + reload) | pipeline | private thread + runtime | +| TLS cert watcher | listener / process | server-owned thread | +| Config watcher, health checks | process | server-owned thread | +| Pingora services | process | `BackgroundService` | +| Every other filter | request | none | + +extproc does not add a fourth pattern. It builds an +AI filter pipeline; overlay `intelligent_route` in +that pipeline is the same thread. + +Filter-owned threads are the wrong lifetime, not +just an ugly implementation: + +- **Drop is not a supervisor.** The only stop + signal is `Drop` of the filter, which is + synchronous. Overlay and the Entra draft both + cancel, wait a couple of seconds, then detach. + The thread can outlive the pipeline with + whatever it captured (client secret, token, + HTTP client). There is no process-level + shutdown join. +- **Reload multiplies them.** Each listener + pipeline that constructs the filter gets a + thread. A config reload constructs the new + pipeline before the old `Drop` runs, so two + copies overlap. Repeated reloads plus a timed-out + join accumulate detached threads. +- **A private runtime is a second scheduler.** + Pingora already runs worker threads. Each filter + thread brings its own current-thread Tokio + runtime (timer wheel, I/O driver) for work that + is usually a sleep and one HTTP POST. That cost + is paid per filter instance, not per process. +- **Spawn failure is invisible.** `from_config` + can return `Ok` after the thread or runtime + fails to start. The filter is live; the cache + stays empty; every request 503s (or worse, if + someone fail-opens). Overlay at least fails + construction when the watcher never comes up. + Copies do not have to. +- **There is no shared budget or signal.** Each + filter invents its own `CancellationToken`, join + timeout, and `warn!`. Operators cannot list, + cap, or drain “background work for this + pipeline.” Tests have to stub a fake handle so + construction does not spawn a real thread. + +A thread is justified for an inotify loop that +must not sit on a Pingora worker. It is not +justified as the default way a filter stays alive +between requests. Once that is the copy-paste +contract, credential refresh, IMDS, and OIDC each +get another thread the pipeline cannot reliably +kill. + +One filter does not by itself justify new core API. +A second copy of a thread-per-filter runtime +*does*: token refresh, IMDS, OIDC federation, and +any other watch-or-refresh filter will otherwise +each own a thread that Drop may fail to join, +leaking secrets. The missing lifecycle is the +cause. The overlay thread is a workaround that +should not become the contract. + +### User Stories + +- As a filter author, I want to start + pipeline-scoped background work without creating + a thread or runtime so that credential refresh + and file watch stay in-tree without copying + overlay internals. +- As a filter author, I want spawn failure to fail + pipeline build so that a dead refresher cannot + ship as a live 503-forever filter. +- As an operator, I want hot reload to stop the old + pipeline's background work so that refreshers and + watchers do not accumulate across config swaps. +- As a maintainer, I want one place that owns + filter background lifetime so that AI, core, and + extproc do not grow independent thread+runtime + copies. From d70571ab95e19f02dbb6b1ab3a349018abce5c19 Mon Sep 17 00:00:00 2001 From: Aslak Knutsen Date: Tue, 1 Sep 2026 15:13:59 +0200 Subject: [PATCH 2/3] Update proposals/01042_filter-task-supervisor.md Co-authored-by: Shane Utt Signed-off-by: Aslak Knutsen --- proposals/01042_filter-task-supervisor.md | 8 -------- 1 file changed, 8 deletions(-) diff --git a/proposals/01042_filter-task-supervisor.md b/proposals/01042_filter-task-supervisor.md index 190ccce..fd2099c 100644 --- a/proposals/01042_filter-task-supervisor.md +++ b/proposals/01042_filter-task-supervisor.md @@ -179,14 +179,6 @@ contract, credential refresh, IMDS, and OIDC each get another thread the pipeline cannot reliably kill. -One filter does not by itself justify new core API. -A second copy of a thread-per-filter runtime -*does*: token refresh, IMDS, OIDC federation, and -any other watch-or-refresh filter will otherwise -each own a thread that Drop may fail to join, -leaking secrets. The missing lifecycle is the -cause. The overlay thread is a workaround that -should not become the contract. ### User Stories From 0999cea6842fbbf679525ad78e890b35925be623 Mon Sep 17 00:00:00 2001 From: Aslak Knutsen Date: Tue, 1 Sep 2026 15:14:17 +0200 Subject: [PATCH 3/3] Update proposals/01042_filter-task-supervisor.md Co-authored-by: Shane Utt Signed-off-by: Aslak Knutsen --- proposals/01042_filter-task-supervisor.md | 8 -------- 1 file changed, 8 deletions(-) diff --git a/proposals/01042_filter-task-supervisor.md b/proposals/01042_filter-task-supervisor.md index fd2099c..b8918de 100644 --- a/proposals/01042_filter-task-supervisor.md +++ b/proposals/01042_filter-task-supervisor.md @@ -171,14 +171,6 @@ just an ugly implementation: pipeline.” Tests have to stub a fake handle so construction does not spawn a real thread. -A thread is justified for an inotify loop that -must not sit on a Pingora worker. It is not -justified as the default way a filter stays alive -between requests. Once that is the copy-paste -contract, credential refresh, IMDS, and OIDC each -get another thread the pipeline cannot reliably -kill. - ### User Stories