Using a proxy for the crawler

[tommaso-castellani]

As per Crawl4AI documentation I tried to add the proxy inside backend/app/config.py as parameter of

BrowserConfig(
    proxy="http://123.123.123.123:1234", 
    headless=True...
)

Unfortunately it didn't work, as I can see CrawlConfigManager it's not being used.

So I added it as parameter of the post requests in both crawler.discover_pages and crawler.crawl_pages:

browser_config_payload = {
    "type": "BrowserConfig",
    "params": {"headless": True, "proxy": "http://123.123.123.123:1234"}
}
response = requests.post(
     f"{CRAWL4AI_URL}/crawl",
     headers=headers,
     json={"urls": url, "browser_config": browser_config_payload},
     timeout=30
)

But it didn't work either.

What's the issue?