A strategic decision guide for anyone preparing a company website for the new agent and citation economy. Written for marketers, web developers and digital decision-makers. All content is open, with no sign-up required.
Quick checklist: is your website AI-ready?
Before you choose a platform or a strategy, check whether AI agents can read your content at all:
4-point AI checklist
- Create an llms.txt fileA machine-readable menu in the root directory that tells AI agents exactly what to read, and in what order of priority.
- Implement JSON-LD schemaFAQ, Article and Organization schemas woven into the source code, validated and consistent across pages.
- Stop blanket blocking in robots.txtIf you block every crawler to protect your content, you also remove yourself from answers in ChatGPT, Perplexity and Gemini.
- Write in Atomic Content Units (ACUs)Direct, verifiable answers with a source at the top of articles, instead of long, impenetrable walls of text.
The problem: from SEO to the citation economy
The traditional search landscape is being replaced by Generative Engine Optimization (GEO). Instead of users clicking through links, AI models compose direct answers from content they retrieve from the web. That creates a new logic: either you are the source the AI cites, or you do not exist in the answer.
Many organisations react defensively: block everything and protect the content. The research shows this backfires:
ACU — The cost of blocking everything
A study from Rutgers and Wharton (December 2025) found that publishers who blocked all AI crawlers saw a 23.1% drop in total web traffic, without any meaningful reduction in how often they were cited in model answers. Defensive blocking costs traffic without delivering the protection it promises. Source: Rutgers/Wharton study, December 2025.
ACU — The bottleneck is the format, not the model
The NEXT-EVAL study (2025) showed that language models reach an accuracy (F1 score) above 0.95 when extracting data from the web, provided the source code is structured and delivered in a clean format. The bottleneck for AI visibility is no longer the model's capability but the site's raw formatting. Source: NEXT-EVAL, 2025.
ACU — The shift is already measurable
In February 2026, SemiAnalysis estimated that 4% of all public GitHub commits were created by Claude Code, double the previous month, with a forecast of 20%+ by the end of 2026. AI agents are not a vision of the future; they already produce, read and cite at scale. Source: SemiAnalysis, February 2026.
Two routes to AI-ready architecture
There are two fundamentally different routes, and the choice depends less on technology than on who will run the website:
Static (e.g. Claude Code + Astro)
- 100% static files, zero JavaScript overhead
- 66% of Astro sites pass Core Web Vitals (vs. 48% for WordPress)
- Type-safe JSON-LD at build time
- Serverless deployment via edge networks
Choose this if: you are building documentation portals, SaaS pages or landing pages, and your team is comfortable with Git and the terminal.
WordPress + Three-Layer Approach™
- A familiar CMS for non-technical writers
- Automated llms.txt generation and maintenance
- 3LA optimisation: L1 Human, L2 Search and L3 AI at the same time
- Crawler management and log analysis at application level
Choose this if: you have ongoing content production, a marketing team or an online shop, and your writers should keep the interface they know.
Note: both routes can achieve full AI citability. The NEXT-EVAL finding applies to both. What matters is clean, structured output, not which engine generates it.
Crawler strategy: the most common mistake, and the right model
Many guides recommend blocking "training crawlers" (GPTBot, ClaudeBot) while allowing "citation crawlers". The distinction sounds logical, but it does not hold up in practice, for two reasons:
1. The same bot does both jobs. GPTBot feeds both model training and the ChatGPT Search index. If you block it to stop training, you also disappear from ChatGPT Search. The same applies to several of the others.
2. robots.txt is a request, not enforcement. In August 2025, Cloudflare documented operators using undeclared crawlers that ignored robots.txt directives. Perplexity has argued that its user agent is not bound by robots.txt because it acts on behalf of an active user. Real protection requires a firewall (WAF) or CDN layer, not a text file.
The right model
Allow citing and searching AI crawlers by default in robots.txt. Enforce protection against documented abuse at WAF/CDN level, or with application tools that separate legitimate agents from harmful ones. Remember the RFC 9309 rule: named user-agent groups do not inherit wildcard rules, so each group must be complete on its own.
llms.txt — a three-layer implementation
| File | Content | Purpose |
|---|---|---|
| llms.txt | Concise Markdown index: H1, blockquote summary, H2 sections with links | A quick overview for AI agents at request time |
| llms-full.txt | Complete text content from priority pages in plain Markdown | The whole knowledge base in a single request, which saves token cost |
| llms-ctx.txt (optional) | Interpretation guidelines: tone, definitions, editorial rules | Steers how the model should understand and reproduce the content |
Google has stated that llms.txt does not affect traditional organic rankings, but Chrome Lighthouse includes the file in its Agentic Browsing category as a direct signal of AI readiness. The file is about the agent economy, not Google rankings.
Want to know how AI-ready your website is today? The 3LA audit checks all four points on the checklist (llms.txt, schema, crawler access and ACU structure) in under 45 seconds.
Run a free AI visibility audit →Strategic conclusion
| Situation | Recommended route |
|---|---|
| Documentation portal, SaaS marketing, static landing pages, technical team | Code route: static generation with type-safe schema and edge deployment |
| Ongoing content production, marketing team, online shop, editorial team | WordPress with the 3LA framework: writers keep their CMS while the tooling packages and hardens the content for AI |
| Unsure, or both | Start with an AI visibility audit of your current site. The decision should rest on a measured baseline, not assumptions |
Frequently asked questions
Should I block AI crawlers such as GPTBot and ClaudeBot?
No, not by default. Rutgers/Wharton (Dec. 2025): blocking everything caused a 23.1% traffic drop with no meaningful reduction in use in model answers. Allow by default and enforce against abuse via WAF/CDN.
What is llms.txt, and do I need it?
A machine-readable Markdown index in the root directory for AI agents. Full implementation: llms.txt + llms-full.txt + optional llms-ctx.txt. Chrome Lighthouse uses it as an Agentic Browsing signal.
A static site or WordPress for AI visibility?
Both work. Go static if the team is technical and the content is stable; choose WordPress with 3LA if writers publish continuously. The bottleneck is structured output, not the platform.
Want updates when the landscape changes?
AI search changes every quarter. Leave your email and we will let you know when we update the guide with new figures and recommendations. No spam, only substance.
[EMAIL FORM GOES HERE — Flatsome form or WPForms shortcode]
Related (in Norwegian): SEO guide · AI-driven SEO · Technical SEO · Visible in AI · Denne guiden på norsk