The web is readable to AI. It mostly doesn't say who wrote it.

Method, and what it can and cannot say

Sample. 500 hosts drawn at random from the Tranco daily top 10,000, list of 5 October 2026. Tranco aggregates several popularity rankings and resists manipulation better than any single list.

Three categories, not two. Tranco ranks domains, not websites. 97 of our 500 are infrastructure: CDN hostnames, ad servers, API endpoints, names that don't resolve to a public address. They are not pages anyone reads, and counting them as failures would inflate our numbers.

Our denominator is 403 websites. Every percentage below is against that, or against the 286 we could score — each one says which.

One request per host, no JavaScript. This is a real limitation for single-page applications, and we would expect it to understate schema detection somewhat.

Evidence stored. Unlike our September run, we kept the served HTML, the robots.txt body, and every HTTP status. Any check here can be replayed without crawling again. One host (yoomoney.ru) contained a null byte that Postgres rejects; it was stripped and the row flagged.

Raw counts are at the bottom. Tell us if we got something wrong.

1. Readable, but anonymous

Split the score into its five dimensions and the gap is not where we expected:

Access is not the problem. Crawlers are allowed, sitemaps resolve, pages parse. The two weak dimensions are both about provenance.

The individual checks make it concrete:

• 81.8% give no author signal
• 66.8% carry no publication date
• 64.3% declare no Organization schema
• 67.5% declare no WebSite schema

A page with no author, no date and no publisher entity is readable but unattributable. A model can summarise it. It has much less reason to name it.

2. The door is now identity-based

117 of 403 websites — 29% — would not let our crawler read them.

And within those HTTP errors, the distribution is lopsided:

Forty-six outright refusals. This is bot protection declining an agent it does not recognise.

Be careful what this means. It does not mean 29% of the web blocks AI. GPTBot announces itself and is widely allowlisted; our scanner is not. What it shows is that access is granted by identity, not by behaviour. A named, known crawler gets in. An unknown one does not — even when it reads one page, slowly, and obeys robots.txt.

That is a structural fact about the 2026 web, and it has a consequence: a model can only cite what its own crawler is permitted to fetch, and permission now depends on being on a list.

3. Being popular buys you nothing

Correlation between score and rank in the top 10,000: −0.066. There is no relationship.

Four bands, four medians within one point of each other. The most-visited sites on the internet score the same as those ranked ten thousandth.

Nobody has a head start here. That window will not stay open.

4. The distribution

One site reached 90. None reached 100.

5. llms.txt: 19%

Of the 403 completed websites, 76 served a /llms.txt — 18.9%. Restricting to the 337 that returned any status at all, it is 22.6%.

llms.txt is a young convention and we are not claiming every site needs one. But it remains the clearest single marker of whether anyone at an organisation has thought about this at all.

6. Which AI crawlers get blocked

Across the 298 websites whose robots.txt we read:

CCBot — the Common Crawl collector, the oldest of the six and the one most associated with training data — is blocked most. PerplexityBot, the newest and the one most associated with answering live questions, is blocked least.

The ordering tracks how much press each crawler received during the 2023–2024 scraping backlash more closely than it tracks what each one actually does today. These rules were written as a reaction, and most have not been revisited.

7. The eighteen checks

Failure rate across the 286 scored sites:

* Software schema is reported but excluded from the score. It asks a homepage to declare itself SoftwareApplication, which is reasonable for a SaaS and meaningless for a newspaper or a shop. Keeping it would have penalised almost every site for not being software.

Note the bottom of the table. Brand in title, paragraph length, heading hierarchy — the classic on-page items two decades of SEO advice drilled in — are where sites do well.

The top of the table is everything a model needs to attribute a claim to a source.

What we take from this

The problem moved. A year ago the question was whether models could reach your pages. On this sample they can: AI accessibility scores 66.9, sitemaps resolve, crawlers are allowed. The question now is whether a model can tell your page apart from anyone else's, and name you for it.

Attribution is cheap to fix and nobody is doing it. An author field, a publication date, an Organization block. These are a morning's work, not a quarter's. 82% of the top 10,000 have not done the first one.

Access is becoming a membership question. 46 outright refusals to an unknown crawler, against 41 named blocks of GPTBot. More sites refuse an agent they don't recognise than deliberately block the one everybody has heard of. If being fetchable depends on being on an allowlist, the set of sources a model can cite narrows to those already known.

Raw data

• Sample: 500 hosts, random draw from Tranco daily top 10,000, list of 5 October 2026
• Scanned: 5–7 October 2026
• Scored 286 · unreadable websites 117 · not websites 97
• Denominator for site-level rates: 403 websites
• Score: min 18, p25 49, median 59, p75 70.75, max 95, mean 58.72
• Dimensions (mean): brand clarity 68.03, AI accessibility 66.87, content structure 61.28, authority signals 42.94, schema coverage 41.30
• /llms.txt: 76 of 403 returned 200 (18.9%); 76 of 337 measured (22.6%)
• robots.txt parsed on 298 websites
• Pearson correlation score × rank: −0.0655 (n = 286)
• Served HTML, robots.txt bodies and HTTP statuses retained for replay

Correction history

5 October 2026. An earlier version of this study, published 30 September, reported that the web was "built for humans and badly built for models", on the strength of an AI accessibility score of 30.3 against brand clarity of 66.6.

That finding came from a bug in our scorer. Two checks asked whether robots.txt named GPTBot and ClaudeBot, and counted an unnamed crawler as blocked. An unnamed crawler inherits User-agent: *; not being named means allowed. We were penalising roughly 85% of sites for something that was not happening, and the same page reported both 16.4% blocking GPTBot and 84.8% failing the "GPTBot allowed" check without us noticing the contradiction.

Two further checks were stricter than the specification they implemented. organization_schema required the literal type Organization and rejected NewsMediaOrganization, Corporation and LocalBusiness, all valid subtypes. software_schema required a homepage to be a software listing.

All three are fixed. This page is a fresh sample measured with the corrected scorer, not a patch of the old numbers.