An analysis of 70 major publishers’ robots.txt files reveals how news organisations are managing AI crawlers, balancing discoverability with content protection and strengthening their position in licensing negotiations.

Robots.txt is often compared to a ‘do not walk on the lawn’ sign: a polite request that visitors are free to ignore. As a description of the technology, that comparison is fair. But because robots.txt is a public, machine-readable file with a clear timestamp, it also puts a publisher’s position on the record. That position could be used as evidence in regulatory proceedings, strengthening publishers’ leverage in licensing negotiations.

In an especially opaque market, where deals are protected by NDAs and “frenemy” relationships abound, robots.txt files are among the few relevant documents in the public domain. So we analysed the files of major publishers to see if they reveal something about the deals and strategies behind them and collate best practices from leaders in the market.

Playing Defence:

Action Controls
Declare robots.txt, the site-wide public statement (focus of this piece)
Signal at page or response level Meta robots tags, X-Robots-Tag headers and machine-readable rights reservations
Enforce CDN and edge controls for verification, rate limiting and blocking
Apply commercial charges Tolling, pay-per-crawl and HTTP 402
Restrict at source Paywalls, registration walls and truncated feeds
Grant rights Licensing, terms of service and TDM opt-outs

 

What we found

  • 60 of 70 publishers explicitly restrict at least one configured AI crawler. Fifty-seven fully block at least one.

  • 39 vs 16 publishers fully block GPTBot and OAI-SearchBot respectively.
  • 48 relationships appear in the working commercial dataset. It contains 30 AI or AI-search relationships, 14 Google News Showcase licences and four Apple News distribution relationships, so the data must be read in this context.

Robots.txt files do not specify commercial agreements, though we have taken into consideration publicly announced LLM agreements for our analysis.

 

Training Vs. Retrieval 

Robots.txt is a public instruction sheet for automated crawlers. Some models have different crawlers for specific functions (e.g. training, search and retrieval/agents), while others concentrate multiple functions in one bot.

While training is a one-off event for each model, retrieval crawlers return to sites on-demand, whenever someone asks a question, capturing real-time information. Retrieval bots checking recent news often use content to validate and verify their results with trusted sources, therefore presenting a higher risk of cannibalising a publisher’s content if the e.g. chatbot answer is enough to satisfy the reader. However, readers that still click through AI citations tend to be more engaged than the average.

relationship-dense-1

The table shows the crawler “families” of the most popular AI services, and illustrates a pattern: publishers welcome crawlers that index content for search, close the door on those that train AI models, and take a more selective approach to those that feed AI-generated answers. GPTBot, OpenAI’s training crawler, is blocked by 39 of the 70 publishers assessed; OAI-SearchBot, its search and answer crawler, by 16. This distinction between learning from an archive and drawing on live reporting reflects publishers’ concern that AI products could cannibalise their traffic, subscriptions and emerging B2B licensing opportunities.

For example, OpenAI's publisher guidance says publishers should not block OAI-SearchBot if they want content included in ChatGPT summaries and snippets, but advises publishers to disallow GPTBot from pages they want excluded from potential training. As a result, twenty-three publishers block GPTBot while leaving OAI-SearchBot either Partial or Allowed. A Partial result means that the rule excludes selected paths, or that a root restriction has an exception. In these policies, the training crawler hits a firm boundary while the AI-search crawler keeps some access.

Meanwhile, Google offers broader control. Google's documentation says Google-Extended covers training future Gemini models and grounding Gemini responses. The rule has no effect on inclusion or ranking in Google Search. In our sample, 33 publishers fully block Google-Extended. All 70 leave at least some access open to Googlebot.

Some controls combine categories, making it difficult to isolate publishers’ specific concerns when they block some bots, while a default path exclusion may simply reflect routine site administration. We therefore report two figures: 66 publishers have a Blocked or Partial result for any AI crawler, and 60 have a crawler-specific AI restriction.Al-use crawlers face more full blocks than search crawlers

The different strategies for robots.txt

In summary, we can see four different strategies adopted by publishers regarding their robots.txt:

  • Open: no or very few limitations, targeting maximum visibility in AI answers and search. This is more often adopted by publishers relying on reach, but it has clear risks of cannibalisation for premium and exclusive content or future negotiation leverage.

  • Closed: All AI crawlers blocked. This is more indicated to publishers with premium content and independent paths to discovery, as it jeopardises referral traffic from AI citations - as robots.txt does not guarantee enforcement, that should be combined with a bot blocking solution.

  • Split by function: Block training, admit search and answer crawlers. This is the market's emerging default, with 23 of 70 publishers blocking GPTBot while leaving OAI-SearchBot some access. It preserves discoverability while protecting archives. However, as licensing value migrates to retrieval (as does cannibalisation risk) this strategy still leaves publishers exposed.

  • Split by content: Following a logic similar to paywall rules, publishers can create different permissions for different paths: commodity coverage open, premium verticals and archives restricted. This helps publishers protect their most unique content, but it is also difficult to maintain, as path-level rules can multiply quickly.

Public crawler rules can get confusing when considering private connections and commercial agreements. A publisher may block a public crawler while supplying content through a licensed feed, API or archive.

What AI licensing agreements tend to cover

Our database of publicly announced commercial deals contains 48 publisher-company records covering 30 publisher domains, to help set the robot.txt restriction into context:

Contracts may cover training, live retrieval, selected titles, current articles or an archive in any combination.


How to read the matrix

Each row pairs a publisher with a company it has a recorded AI or AI-search relationship with. Read across the row to find the coloured square. That square summarises how the publisher's public robots.txt treats the company's mapped crawlers on 14 July 2026.

Allowed means every mapped crawler is allowed. Partial means at least one mapped crawler faces path-level limits and none is fully blocked. Mixed means the mapped crawlers receive different treatment, including at least one full block. Blocked means every mapped crawler is fully blocked.

The matrix describes the public crawler route. A licensing agreement may provide access through a feed, API or archive.
relationship-dense-1

How to read the automated checks

These checks use the cached files fetched on 22 July 2026. They flag repeated user-agent groups; sitemap URLs that did not return a successful response to our checker; the absence of a Sitemap: declaration in robots.txt; and directives that our parser does not recognise.

The checks are diagnostic signals, not necessarily publishing errors. A failed sitemap check can reflect an access refusal or timeout, and no sitemap declaration does not prove that a site has no sitemap.

quality-dense-1

Based on the public announcements and reported coverage we reviewed, rights were usually defined by how the content could be used, not by individual user-agent strings. Crawler names change, and products often move faster than the contracts meant to cover them. The technical controls were then matched to those rights and to the routes used to deliver the content.

Key agreement terms included:

  1. Pre-training: Use of a defined large collection of data to develop general model capabilities.
  2. Fine-tuning: Later specialisation using publisher material.
  3. Retrieval and grounding.Use of current or archived content at query time.
  4. AI-search display: Summaries, quotations, citations and links shown in an answer.
  5. Agent access: User-initiated or automated requests for publisher pages.
  6. Internal publisher tools: Access to technology or models supplied through the agreement.
  7. Content scope: The titles, articles, archives or datasets covered by each permitted use.
  8. Products: The named products and services allowed to use the content.
  9. Territories: The geographic markets covered by the arrangement.
  10. Duration: The term of the right and any relevant archive period.
  11. Delivery route: The feed, API, archive or other authorised method used to provide the content.
  12. Excluded material: Content or uses outside the permission.
  13. New-product consent: Whether a new product fell within an existing right or required fresh consent.
  14. Payment structure: A minimum guarantee, usage-based payment or revenue share linked to the permitted use.
  15. Attribution and reporting: Attribution requirements, referral reporting and evidence connecting the use to payment.
  16. Audit and correction: Audit access and procedures for correcting errors or disputed usage.
  17. Termination: Rights to end or renegotiate the arrangement.
  18. Measurement unit: A defined dataset for training, an answer impression for retrieval or a verified referral outcome for AI search.

Terms can include minimum guarantees, usage-based payments, revenue share, attribution requirements, referral reporting, audit access, correction procedures and termination rights. Units vary by use: a defined dataset for training, an answer impression for retrieval or a verified referral outcome for AI search.

 

Recommendations for Publishers Robots.txt

Start with eight pieces of housekeeping:

  • Define the value of your content
    Identify the revenue it supports, including advertising, subscriptions and licensing. This will determine how much visibility you need and what requires protection.
  • Measure crawling demand and cost
    Track bot request volumes and their infrastructure cost. Repeated requests can overwhelm niche content providers, but they may also reveal demand worth licensing.
  • Read the current crawler policy
    Separate explicit crawler decisions from rules inherited from the default group. Check whether the result matches your intent for search, training, retrieval and agents.
  • Align your defence layers
    Make sure robots.txt, page-level signals, rights reservations and CDN or bot-blocking rules reflect the same priorities.
  • Match each contract to its technical route
    Record whether the partner uses public crawling, an API, a feed or another delivery method. A crawler name alone does not define the right.
  • Split the relationship types
    Keep AI training, AI search, News Showcase and distribution products in separate cohorts. Record start dates, end dates and verification dates.
  • Agree on the evidence before discussing price
    Ask what usage, attribution, referral and audit data the partner will provide. Measurement makes a right easier to value and enforce.
  • Put the next review in the calendar
    Crawler documentation, platform products and publisher policies change, so revisit the technical policy and the contract together.
 

Methodology and limitations

This is a point-in-time analysis of public rules. It compares what 70 publishers told named crawlers on 14 July 2026. It does not test whether crawlers followed those instructions or explain why a publisher chose them.

Snapshot and sample

We collected and cached one robots.txt file for each of 70 publishers on 14 July 2026. The cache gave every publisher the same observation date. Policies may have changed since then.

The sample was chosen to show a range of practices among large news organisations and publishers involved in public discussions about AI licensing or crawler access. It is not a representative sample of the global news industry. Every count and percentage in the article refers to these 70 publishers.

Crawlers and policy patterns

The crawler list covers publicly documented user agents used for model training, search indexing, live retrieval, answer generation and agent access. We classified each crawler by the function described in its operator's documentation. Googlebot was included where it provided a useful comparison with conventional search.

Some companies use a different crawler for each product. Others use one crawler for several functions. Our labels describe documented uses, not a common technical design. The list may also miss private, undocumented, new or renamed crawlers.

We assigned each publisher one of four policy patterns or a fallback data-quality label:

  • Open: AI crawlers generally retain access. Any restrictions are limited and do not amount to a full block.

  • Closed: Every AI crawler in the configured set is fully blocked.

  • Split by function: Treatment changes with the crawler's documented purpose. A publisher might block training while allowing search or retrieval.

  • Split by content: Treatment changes by site path or content section.

  • Unclassified: A data-quality label, not a policy pattern. It is used when the rules are incomplete, contradictory or do not fit one pattern closely enough.

These labels describe the published configuration. They are our analytical categories, not terms used by the publishers themselves.

We report two restriction measures because they answer different questions:

  • Any AI restriction counts a publisher when at least one AI crawler encounters a restriction, including rules inherited from a default group.

  • Explicit AI restriction counts a publisher only when a rule names an AI crawler or crawler family.

An explicit rule is stronger evidence of a stated position towards a named crawler. It is not evidence of the publisher's commercial motive or legal interpretation.

Commercial relationship evidence

The commercial dataset contains 48 publicly evidenced publisher-company relationships covering 30 publisher domains. Records came from publisher announcements, company disclosures and reported coverage.

We used these records to add context, not to explain individual robots.txt rules. A publisher can block a public crawler while supplying content through an API, feed or archive. Public crawler access does not prove that a commercial relationship exists.

This is not a complete market inventory. Confidential, undisclosed and unverified arrangements are excluded, so an absent record does not mean that no relationship exists.

What robots.txt can show

Robots.txt records the instructions a publisher made public for identified crawlers at a particular time. It can show the observable policy and support a description of its apparent pattern.

It cannot establish the publisher's underlying intent. Similar rules may result from different commercial, operational, technical, editorial or legal decisions.

Limits of the evidence

Robots.txt is one control among many. This analysis does not observe:

  • Contractual permissions, licensing rights or private delivery routes

  • Page-level directives, authentication, paywalls, CDN controls or firewall enforcement

  • When a policy changed or how it relates in time to a deal, product launch or dispute

  • Whether a crawler complied, or what content it crawled, indexed, retrieved or used for training

  • Separate permissions for different functions when one user agent performs several jobs

A restriction is a published instruction, not proof of technical prevention. If we found no applicable restriction, that means only that the robots.txt file contained no matching rule. It does not amount to contractual, copyright or legal permission.

The findings therefore describe the public configurations visible on 14 July 2026. They do not establish contractual rights, licensing status, crawler compliance, legal entitlement or infringement.

Sources

FT Strategies crawler snapshot and generated datasets, fetched 14 July 2026.

Source-backed commercial relationship evidence compiled from publisher announcements, partner disclosures, and reported coverage.

OpenAI publisher and developer FAQ

Google crawler documentation

Commercial research sources

This table brings together 35 public-source URLs used for the 48 publisher-company relationship records. It documents the evidence trail.

At FT Strategies, we help publishers build resilience against AI-driven disruption by understanding where their content, traffic and commercial models are most exposed. We combine exposure mapping with defensibility analysis to identify high-value assets, evaluate platform and AI dependencies, and prioritise opportunities for protection, differentiation, and monetisation.

If you are assessing how AI companies access and utilise your content, please get in touch with us. We can help you transition from making individual decisions about crawlers and licensing to developing a clear, data-led strategy to protect your value and discover opportunities for long-term growth.