Proxy escalation #570

Activity

1. us commented on Sep 22, 2026

   

   us

   on Sep 22, 2026

   Last edited by us

   Member

   More actions

   Your read of proxy_list is right: it is an always-on pool, not an escalation tier. The behaviour you want is already in the engine, split across two knobs that were never documented. That part is on us.

   Leave proxy_list / proxy unset, then:

   HTTP path

   CRW_HTTP_RATELIMIT_PROXY_URL="http://user:pass@gateway:port"

   Direct first. Retried once through the proxy on a 429, on a vendor challenge header (cf-mitigated, x-amzn-waf-action), or on a body that classifies as a block wall rather than a page, which covers the Cloudflare interstitial served as a 200. A per-host latch means later URLs on a host that already blocked you start on the proxy instead of repaying the discovery cost each time, with a direct rescue still available if the exit turns out worse than your own IP.

   JS path

   [renderer]
   auto_egress_escalation = true
   
   [renderer.chrome_proxy]
   ws_url = "ws://chrome-proxy:9222"

   Holds the residential tier out of the normal ladder and fires it only on a hard block (403/429/503/401/520-530 or a bot-wall interstitial). Suppressed on fingerprint-vendor walls (managed CF challenge, DataDome, PerimeterX, Kasada) since a different exit does not clear those.

   On your retry-count idea: each layer makes one proxy attempt today. We benchmarked always-on first and it was net-negative (success down ~2pp, p90 up ~69%), so recovery is bounded to a single attempt behind a block signal. A tunable count on top of that gate is a fair ask and it is on the list.

   Docs for both knobs are going up at https://docs.fastcrw.com/proxies. Thanks for the writeup, the bandwidth framing is the useful part.
2. rlex commented on Sep 22, 2026

   

   rlex

   on Sep 22, 2026

   Author

   More actions

   does it cover every path (markdown, lightpanda, chrome and chrome-stealth?). Searx i assume will need separate configuration.
3. us commented on Sep 23, 2026

   

   us

   on Sep 23, 2026

   Member

   More actions

   yes, with one clarification and one thing i owe you.

   markdown is an output format, not a path. what picks the path is renderJs.

   http path (renderJs: false) is CRW_HTTP_RATELIMIT_PROXY_URL. direct first, one retry through the proxy on a 429, on a cf-mitigated or x-amzn-waf-action challenge header, or on a body that classifies as a block wall rather than a page. a per-host latch means later urls on a host that already blocked you start on the proxy instead of repaying the discovery cost.

   js ladder (renderJs: true) is auto_egress_escalation. the gate is not per renderer, it is "did any attempt in the ladder end in a hard block" (401/403/429/503/520-530, or a bot-wall interstitial served as a 200). so a block from lightpanda and a block from chrome arm the same recovery, and chrome_proxy runs once after the ladder rather than inside it.

   chrome-stealth is not a separate tier. it is [renderer.chrome] pointed at a browserless endpoint, so it is covered as chrome, and the residential arm still fires on top of it.

   two caveats worth knowing before you turn it on:

   • the arm is suppressed on fingerprint-vendor walls: managed cloudflare challenge, datadome, perimeterx, kasada. a different exit ip does not clear those, it only burns the deadline.
   • if the request asks for a screenshot, chrome_proxy stays in the ladder instead of being held back, so a capture never fails for lack of a capable tier.

   search runs on its own stack and does not share these knobs, so yes, configured separately.

   one correction to my last message: i pointed you at https://docs.fastcrw.com/proxies, and that page is live but neither of these two knobs is on it yet. that is on me, writing it up now.
4. us commented on Sep 23, 2026

   

   us

   on Sep 23, 2026

   Member

   More actions

   correction to my last message: i was reading a stale local checkout. the proxies page does cover both knobs, and the two sections you want are "escalate only on a block" and "coverage":

   https://docs.fastcrw.com/proxies/

   sorry for the misdirect. the rest of what i wrote stands.