CRAWLER DOCUMENTATION

ThreeLayerBot

If you found this URL in your access log, one of your own site’s owners asked us to audit a page. The exact user-agent we send, from every part of the platform, is: ThreeLayerBot/1.0 (+https://threelayerapproach.com/bot). We never claim to be Chrome, Safari or any other browser, we send no browser fingerprint headers, and we do not switch identity if a request is refused.

To allow us in robots.txt: "User-agent: ThreeLayerBot" followed by "Allow: /". To refuse us entirely: "User-agent: ThreeLayerBot" followed by "Disallow: /" — we stop, and the audit reports that it was disallowed.

A WAF rule refuses the request before robots.txt is ever read, so a 403 or a CAPTCHA challenge is fixed in the firewall rather than in robots.txt. SiteGround: allow the user-agent ThreeLayerBot/1.0 in Site Protection. Cloudflare: a Skip rule on User Agent contains ThreeLayerBot. Wordfence: allowlist the request in Live Traffic, and do not rate-limit it.

For other crawler operators: we do not use the Mozilla/5.0 (compatible; NAME; +url) form. Measured against a SiteGround-hosted site from a non-allow-listed address, that form returned 403 for every name except allow-listed search engines, while the bare NAME/VERSION (+url) form returned 200 — apparently fake-Googlebot mitigation.

  • One GET request per page, on demand — triggered by a person, not a schedule.
  • robots.txt fetched first and respected.
  • We read HTML, robots.txt, sitemap.xml and llms.txt. Nothing else.
  • No POST, no logins, no cookies kept between requests.
  • Sensitive paths (.env, .git, wp-config.php, phpinfo.php) are blocked in our own code before a request is made.
  • No user data from your site is stored.

Identify a ThreeLayerBot request

The documented user-agent lets site operators understand why a request was made and locate the relevant access-log entry. A user-agent string alone is not authentication: other clients can copy it. Review the request, your firewall rules and the audit that triggered it together.

Resolve access at the layer that refuses it

If robots.txt disallows crawling, update the intended crawler policy. If a firewall returns a CAPTCHA or a 403 before the page is fetched, changing page copy or schema will not solve the access problem. Adjust only the necessary rule, then request a new audit to verify that the public HTML can be read.