feat(enrichers): add option to Maigret enricher to scan all sites, bypass Cloudflare, and improve logging

[PrinceBunBun981]

This adds 4 parameters to the Maigret enricher:

• MAX_CONNECTIONS: to set the number of current connections to use for a scan, higher numbers will sometimes cause issues with Flaresolverr/Trawl.
• SCAN_ALL_SITES: to perform a username scan on all sites instead of just the top 500.
• CLOUDFLARE_BYPASS: to enable bypassing Cloudflare protections using Flaresolverr or Trawl.
• CLOUDFLARE_BYPASS_URL: for the Flaresolverr/Trawl URL to use (this must be something like http://localhost/v1)

This also improves the logging for Maigret, instead of waiting for the enricher to finish running, it now logs found social accounts as they are found:

Before	After

Resolves #211

[dextmorgn]

Hey @PrinceBunBun981,

Thanks for this great work. I managed to make the settings pass from front to back; that great timing!

There is a failing job, Lint & Typecheck / Python typecheck (mypy, changed files only) (pull_request)

Would you mind checking what's wrong ? probably a mypy error.

[PrinceBunBun981]

Hey @PrinceBunBun981,

Thanks for this great work. I managed to make the settings pass from front to back; that great timing!

There is a failing job, Lint & Typecheck / Python typecheck (mypy, changed files only) (pull_request)

Would you mind checking what's wrong ? probably a mypy error.

Yep, seemed to be mypy. Been a while since I opened the project; didn't realize linting and type checks were added haha, but all passes now. Will do the checks locally before opening PRs now lmao

[dextmorgn]

Looks like the timeout=100 we had on the old subprocess.run got lost when this moved to Popen. Nothing bounds the read loop now, and scan() is async with no asyncio.wait_for around it, so if maigret hangs we freeze the whole event loop, not just this one scan. Can we get a timeout back in there?

[PrinceBunBun981]

Still working on a fix for this one. subprocess.Popen doesn't have a timeout option like subprocess.run, so I'm trying to figure out another good way to keep the new logging.

[dextmorgn]

We never .wait() or kill the Popen. If reading stdout throws before the child exits on its own, we'd leak an orphaned maigret process. Maybe a finally: process.terminate()?

[PrinceBunBun981]

Should be fixed in f90ad87, added a check for the return_code and then a finally that checks for process.poll() and terminates if None. Can swap to just a basic finally: process.terminate() if preferred, though.

[dextmorgn]

This does a DB write per matched line inside the loop, with SCAN_ALL_SITES on and a high-hit username that could be a lot of sequential writes. Not urgent, but might be worth batching at some point.