It might be the summer break for publishers in the Northern Hemisphere, but one segment of the audience never goes on holiday: AI crawlers. 

For years, robots.txt files were how publishers allowed web spiders to crawl their website, and signalled to search engines their permission to index articles. Now it can also signal publishers’ approach to access by AI models and services.   

Robots.txt is often compared to a ‘do not walk on the lawn’ sign: a polite request that visitors are free to ignore. However, with licensing deals between AI and content companies negotiated behind closed doors, we decided to look at the robots.txt files of 70 major publishers to analyse their differing approaches and identify some best practices.

As you’ll read, some are prioritising visibility in AI search and responses. Others are taking a more defensive position, protecting content that they believe has value through subscriptions and licensing. Most — 60 of the 70 we looked at — restrict at least one AI crawler, most often those associated with model training.

Your AI crawler policy will depend on the importance of content discovery, the maturity of your subscription business and whether there is a market for content licensing. Robots.txt might be just an initial layer of protection, but research from Known Agents shows that 96% of bots respect its specifications, so it's worth checking whether your file is up to date. 

Whatever route you take, who you let crawl your content and how they do it are commercial  - and possibly legal - decisions that need careful consideration. 

Our eight pieces of housekeeping advice are a good place to start, while our recent two-part series on resilient content businesses in the age of AI (part one and part two) provide further food for thought.

Get in touch if you would like to discuss the analysis — or the trade-offs for your publication or outlet — in more detail. 

Beyond Reader Revenue is an interview series offering practical, behind-the-scenes insights into revenue diversification at the Financial Times. In this conversation, Aled John, Group Strategy Director at the Financial Times and former Deputy Managing Director of FT Strategies, discusses the formation and growth of FT Strategies.

FT Strategies was originally created in response to the growing demand from media organisations seeking to learn from the Financial Times' own transformation. Since then, it has developed into a global consulting business that supports organisations across the news, media, and other content-driven industries. Aled reflects on how the business identified its areas of expertise, developed the capabilities needed for growth, and adapted its offerings as industry needs evolved.

Featured Insights

ARTICLE

Robots.txt as strategic intent: Analysis of large publishers’ practices and policies

INTERVIEW

Beyond Reader Revenue: Interview with Angela Mackay, CCO and MD of Asia Pacific at the FT

ARTICLE

Resilient Content Businesses in the Age of AI: (Part 1) Publishing, AI and the curse of small stakes

ARTICLE

Resilient Content Businesses in the Age of AI: (Part 2) The New Valuation Logic of Media M&A

Recommended Reading

Strategy Development:

• Non-website audience is focus of revenue growth for new Business Insider CEO (Press Gazette)
• How Hello! fought back after saying goodbye to Google Discover traffic (Press Gazette)
• The Athletic teams up with sports creators to reach new (and younger) audiences (NiemanLab)
• Bloomberg Media launches Bloomberg India (InPublishing)
• Daily Mail appoints US general manager to build subscriptions and ad partnerships (Press Gazette)

Commercial Growth:

• Time has started serving ads to AI agents (Digiday)
• Months After Launching Subscriptions, Reach plc Wants to Launch Bundles (A Media Operator)
• Moving Down the Funnel: How Media Companies Can Facilitate Direct Buyer-Seller Connections (A Media Operator)
• Acast has acquired Backyard Ventures for $20m (Acast)
• MS NOW to launch membership program for ‘super fans’ (Variety)

Operational Transformation:

• USA Today EBITDA Falls 12% as it Touts Pivot From Scale, New Palantir Deal (A Media Operator)
• Reddit is experimenting with turning posts into audio and video content (The Verge)

AI, Technology & Data:

• Cloudflare says bot blocking is fuelling publisher AI deals (Press Gazette)
• The New York Times has begun testing a generative AI search tool for readers (Semafor)
• New York Post launches "Hamilton" AI chatbot (Axios)
• More AI Could Be Coming to Google Search Under Upcoming UK Regulations (A Media Operator)