ThreeLayerBot
If you found this URL in your access log, one of your own site's owners asked us to audit a page. This is the whole story: what we request, what we never request, and how to allow or refuse us.
ThreeLayerBot/1.0 (+https://threelayerapproach.com/bot)
Every request from every part of our platform sends this one string. We never claim to be Chrome, Safari or any other browser, we send no browser fingerprint headers, and we do not switch identity if a request is refused.
What the crawler does
- One GET request per page, on demand — triggered by a person who entered that URL in our tool, not by a schedule.
- robots.txt is fetched first and respected. A Disallow that matches us means we stop.
- We read HTML, robots.txt, sitemap.xml and llms.txt. Nothing else.
- No form submissions, no POST, no logins, no cookies kept between requests.
- We never request hidden or sensitive paths (.env, .git, wp-config.php, phpinfo.php and similar) — those are blocked in our own code before a request is made, because touching them gets an IP banned, and rightly so.
- No user data is stored from your site. The audit keeps the page score and the text it analysed, for the person who requested it.
Allow or refuse us in robots.txt
To allow the audit explicitly:
User-agent: ThreeLayerBot Allow: /
To refuse us entirely — we will stop, and the audit will report that it was disallowed:
User-agent: ThreeLayerBot Disallow: /
Allowing us in the common setups
A WAF rule refuses the request before robots.txt is ever read, so if the audit reports a 403 or a CAPTCHA challenge, the fix is here rather than in robots.txt.
Site Tools → Security → Site Protection. Ask support to allow the user-agent ThreeLayerBot/1.0. Allowing an IP address works too but ages badly — our egress addresses rotate, and the allow-list is capped at five entries.
Security → WAF → Custom rules. Add a Skip rule where User Agent contains ThreeLayerBot, skipping Bot Fight Mode and managed challenges for it.
Firewall → Blocking → Allowlisted services, or Tools → Live Traffic to find the blocked request and allowlist it. Do not rate-limit the user-agent: one audit is a handful of requests, not a crawl.
Why we do not use the (compatible; …) form
We first identified as Mozilla/5.0 (compatible; ThreeLayerBot/1.0; +url) — the Googlebot convention, and the form most crawler libraries default to. Measured against a SiteGround-hosted site from an address that was not allow-listed, that form returned 403, while the bare ThreeLayerBot/1.0 (+url) returned 200. The same 403 applied to every name we tried in that form except allow-listed search engines, which looks like fake-Googlebot mitigation: the (compatible; shape is what spoofers use. If you operate a crawler and cannot work out why one host refuses you, this may be why.
It is also why we do not send a browser user-agent with a bot token appended, which is what we used to do. One address presenting several browser identities is the signature of a cloaking bot, and a firewall is right to treat it as one.
Still seeing something you did not expect?
Write to hello@threelayerapproach.com with the date, path and status code from your log. If our crawler did something this page does not describe, we want the log line — that is a bug on our side, not a configuration question on yours.
