Other IP-Lists

[superstes]

Greetings.

Thank you for gathering this information 👍

We also have a project where we list some of the IP-List API's: https://github.com/O-X-L/open-bot-list
You could copy some to your workflow.

It might make sense to separate the AI-training-crawlers from the other ones as they might not be seen as 'good-ones' but the AI-user-crawlers might be important for user-search (?).

Note - these could be simplified:

• DuckDuckGo has some good API-endpoints: Duckduckbot, Duckduckassist (AI) - instead of parsing the MD-file
• Bunny-CDN seems to also have a plain-IPv6 endpoint => https://bunnycdn.com/api/system/edgeserverlist/IPv6/plain