L2
Crawlability
The ability of search engine and AI crawlers to access and index a website's content. JavaScript-heavy pages often have poor crawlability.
Good crawlability is a prerequisite for visibility – both in search and for AI. JavaScript-heavy pages, missing sitemaps, and errors in robots.txt are common causes of poor crawlability. For AI crawlers, this is even more critical than for traditional search engines.
This term is part of the 3LA EntityMap → /entitymap.html
Related terms
AI Crawlability
L3The ability of LLM agents and AI crawlers to access, parse, and understand a website's content for use in training or real-time retrieval.
Technical SEO
L2The practice of optimizing website infrastructure – crawlability, indexability, site speed, and mobile-friendliness – to improve search engine rankings.
llms.txt
L3A proposed standard file (similar to robots.txt) that communicates to AI crawlers which content should or should not be indexed by large language models.
