271 bots / 15 categories | March 2026
Test a User-Agent string
Autonomous browser-controlling AI that completes tasks on behalf of users
Fetch website content to include in AI-generated answers
Download content for training large language models (LLMs)
Index content for AI-powered search engines
AI bots with unknown or undocumented purpose
Traditional search engine indexing
Competitive analysis and SEO monitoring tools
Generate link previews for social sharing
Check site availability and performance
Vulnerability scanning and security assessment tools
RSS and content feed aggregators
CMS-related bots and plugins
Web archive and content preservation services
Headless browsers and automation frameworks
Programming libraries used for HTTP requests