Question 1

Do you actually test whether AI crawlers get through?

Accepted Answer

Yes, and most checkers don’t. Reading robots.txt only shows the policy; the enforcement lives in the CDN or firewall, which can 403 a bot the robots file "allows" (Cloudflare began blocking AI crawlers by default for new sites on 1 July 2025). So this scan also fetches your page live, identifying as GPTBot and PerplexityBot, and reports what the network layer actually serves them. One honest caveat: if a site allowlists only verified bots by IP, the vendors’ real crawlers may pass where our test is walled: which is why a wall here is reported with that caveat, and why we tell you to confirm in your server logs.

Question 2

Why is server-side rendering weighted heaviest?

Accepted Answer

Most AI crawlers fetch your raw HTML and do not execute JavaScript, Vercel measured this across its edge network: OpenAI’s and Anthropic’s crawlers do not render JS. If your content only exists after JavaScript runs, ChatGPT, Claude and Perplexity see an empty shell, so rendering and indexability carry 27 of 100 points, the largest single block in our rubric. The measurement is not small: Vercel analysed over 500 million GPTBot fetches and found zero evidence of JavaScript execution. Google’s Gemini is the one exception, because it rides Googlebot’s rendering service.

Question 3

Does Google use llms.txt?

Accepted Answer

No. Google’s own search documentation says: “You don’t need to create new machine readable files, AI text files, or markup to appear in these features”, and its Search team confirmed verbally (July 2025) that llms.txt is not used and not planned. Anthropic and Perplexity, by contrast, publish their own docs in the format. Ahrefs measured 137,210 domains in May 2026: 97% of llms.txt files received zero requests. That is why we score it just 4 of 100 points, worth shipping because it costs minutes and two major AI surfaces use the format, but never the headline mechanism some agencies sell it as. (We have cut this weight twice now as the evidence came in. The rubric moves when the evidence does, which is the whole reason we publish it.)

Question 4

Why two files: what is llms-full.txt for?

Accepted Answer

They serve two different machines. llms.txt is a menu: a few kilobytes an AI agent fetches to decide which of your pages answers a question, before fetching just that page. llms-full.txt is the whole book: your entire content in one file, for tools that ingest everything upfront. AI coding assistants, retrieval pipelines, and assistants that load full context in a single request. Where these files get fetched at all, measured traffic shows llms-full.txt is fetched more, with ChatGPT the most frequent visitor. One caveat we verified ourselves (11 Jul 2026): Anthropic’s own llms-full.txt is 90.1 MB and Perplexity’s is 3.8 MB, far too large for any model to ingest in one shot. The file only helps if it fits, so the ones we build stay small enough to be swallowed whole.

Question 5

Why is structured data worth so little here when every SEO guide pushes it?

Accepted Answer

Because the best available evidence does not support scoring it high, and we would rather lose the upsell than tell you something we cannot back. Ahrefs tracked 1,885 pages that added JSON-LD against 4,000 matched controls: the differences for ChatGPT and Google AI Mode were within noise, and AI Overviews citations actually fell 4.6%. A separate controlled test found ChatGPT and Perplexity tokenize JSON-LD as raw text rather than parsing it as data, and that they ignore markup describing anything not visible on the page. Some studies do find a lift (AirOps measured 38.5% vs 32.0% citation rates across 16,851 ChatGPT queries), so it is not nothing. We landed on 14 of 100 points across four checks: worth having, never a substitute for the answer being visible in the HTML. We used to score this at 21.

Question 6

Which AI crawler should I actually be worried about blocking?

Accepted Answer

The search one, and the names do not make it obvious. Every vendor runs at least two bots: GPTBot, ClaudeBot and PerplexityBot gather training data, while OAI-SearchBot, Claude-SearchBot, ChatGPT-User and Perplexity-User build and serve the answers people actually read. Opting out of training is a defensible business decision plenty of firms make deliberately. Blocking the search bots is how a firm disappears from ChatGPT without ever deciding to, and it usually happens because one line in robots.txt was written to keep AI out and nobody separated the two jobs. This scan names any blocked search crawler separately for exactly that reason. Bingbot belongs in the same list, because Bing’s index feeds Microsoft Copilot.

Question 7

Why do some scores get capped?

Accepted Answer

Two failures make everything else moot. A noindex directive tells every engine to forget the page exists, so the score caps at 25. Blocking five or more AI crawlers in robots.txt means the engines that answer your customers cannot read the site at all, so the score caps at 55 regardless of how polished the rest is.

Can AI actually read your site?

The evidence behind the rubric