See which AI crawlers reach your pages
Log Analyzer turns access logs into evidence of crawler activity: which bots requested your pages, when they visited and how the server responded. A request is not proof of a citation.
Find the pages crawlers cannot reach
Inspect requested paths, response codes and crawl frequency together. A frequently requested page that returns an error needs a different fix from a useful page that is never fetched. Compare log evidence with your sitemap and internal links before changing crawler access rules.
What logs can and cannot prove
Access logs record requests, not the answers an AI assistant later produces. User-agent labels may also be spoofed; treat a matched name as a classification signal rather than proof of identity. Use observed citations and referral reports alongside logs when evaluating business impact.
Frequently Asked Questions
What log formats does Log Analyzer support?
Apache, Nginx, Cloudflare Enterprise Logs and AWS CloudFront JSON are auto-detected on upload. No schema mapping, no SQL and no manual configuration required.
Which AI crawlers does it identify?
Log Analyzer recognises 200+ named crawlers including GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended, Googlebot, Bytespider, CCBot and many others. The full list is maintained in the open 3LA AI Bot Database.
What is the difference between Log Analyzer and Bot Tracker?
Log Analyzer parses uploaded server access logs and produces a historical report. Bot Tracker classifies crawlers in near-real-time as cite-worthy, neutral or hostile, and links crawl activity to actual citation events and referral traffic.
How far back does the historical archive go?
Log Analyzer stores up to 90 days of parsed log data. You can filter by bot, URL path and date range, and export everything as CSV.
What is threat detection?
Threat detection auto-flags crawlers that make an unusually high number of requests, violate robots.txt directives or exhibit scraping patterns. Each threat is assigned a severity level with a recommended robots.txt entry.
Can I generate a robots.txt from the results?
Yes. Log Analyzer includes a one-click AI-aware robots.txt generator that produces directives based on the crawlers observed in your logs — allowing cite-worthy bots and blocking hostile scrapers.