Should you block AI crawlers to protect your content, or feed them to stay visible?
Last updated: September 14, 2026
robots.txt is a bargaining chip, and it has cashed out. News Corp, the Financial Times, the Guardian and TIME all blocked, then signed paid licensing deals with OpenAI. That is the publisher move, and it is rational. But if you run a business and your content exists to sell something else, the same block does the opposite of what a publisher gets: you lose the citation and never get a check. You disappear from the AI answers your buyers are already asking for, and nobody licenses a plumber's service page. For a business trying to get found, blocking AI crawlers is self-sabotage dressed up as caution.
The two moves, and the bots they act on
There are really three crawler jobs, and the docs keep them separate, which matters because you can block one without blocking the others. OpenAI's own crawler docs name them: GPTBot crawls to train the foundation models, OAI-SearchBot surfaces sites inside ChatGPT search, and ChatGPT-User fires when a user asks ChatGPT to go fetch a page. The last one is user-initiated, and OpenAI says robots.txt rules may not apply to it. Anthropic runs ClaudeBot; Perplexity runs PerplexityBot. All of them read robots.txt.
Blocking is trivial. Here is the whole thing for the training crawlers:
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: PerplexityBot Disallow: /
That's it. Three stanzas, and you are out of the training data for the three biggest assistants. The ease is exactly why the decision matters: it is one file edit, and people make it in a panic without asking what they're actually trading away.
Why a publisher blocks, and why it works for them
A publisher's articles are the inventory. If a model trains on the archive for free and then answers reader questions with a summary, the reader never visits, and the business that paid for the reporting eats the cost. So publishers block the crawler and treat access as a thing you buy. It is not a bluff. Look at who has already turned the block into a contract: News Corp signed a multi-year global partnership covering the Wall Street Journal, MarketWatch, Barron's and more; the Financial Times struck a licensing agreement to bring FT journalism into ChatGPT with attribution; the Guardian and TIME did the same. The leverage was the ability to say no. If your content is genuinely scarce and paid-for, no is a real card.
Why a business feeding the bots is the whole point
Now flip it. Your service pages, your comparison guides, your "how X works" posts exist to make a buyer choose you. They are not the product; they are the pitch. When someone asks ChatGPT "best CRM for a two-person agency" or "who does press releases for startups," the model answers from what it read. If you blocked it from reading you, you are not in the answer. There is no license coming, because nobody pays a SaaS company to train on its pricing page. You paid the full price of the block and collected none of the reward.
This is the part people underestimate: the block is symmetrical in effort and wildly asymmetrical in payoff. A publisher blocks and gains a negotiating position worth real money. A business blocks and gains a slightly smaller training corpus for OpenAI and a much smaller presence in the thing its customers now use to decide. One of these is a strategy. The other is a reflex.
The honest disagreement
The two camps are both loud and both partly right. The split isn't really "block vs feed," it's about what you think your content is worth on the open market:
School A: block, protect the asset, force the deal
Your content has value; giving it away for free trains the thing that will replace you. Block the crawlers, make AI companies pay to license, and control attribution. The publisher deals prove the leverage is real. Feeding the bots for "visibility" is giving away the store for a maybe.
School B: feed them, citation is the distribution
Being in the answer is the new being on page one. AI assistants are where buyers now ask, and a block removes you from that surface entirely. For anyone who isn't a media company, the "asset" being protected is marketing copy nobody will license anyway. Blocking trades a real benefit for a fake one.
Both are right, for different companies, and that is the actual answer, so let's be precise instead of mushy. The deciding line is whether your content is the product. News Corp's leverage came from owning something OpenAI wanted enough to pay for; the paid, attributed deals are the proof. A business whose content is marketing has no such leverage, because nobody licenses an ad. So: if you are a publisher with scarce, paid-for content, block and negotiate, the deals are real and the card is worth playing. If you are a business trying to get recommended, feed the bots, all of them, and spend your effort getting mentioned on the third-party pages they also read. The one caveat worth keeping: if a specific crawler is genuinely hammering your server and costing you money, throttle that one with a crawl-delay or block it for load reasons. That is an ops decision, not a visibility strategy, and it is the only version of "block" that makes sense for a get-found business.
Related entries
- Compare with Do backlinks still matter for AI visibility, or have mentions replaced them?
- Prerequisite Retrieval vs training: which one decides whether AI mentions you?
- Explains Why is my brand invisible to AI even though I rank on Google?
- Related What is co-occurrence, and why do mentions beat backlinks for AI?
Changelog
September 14, 2026: First published. Sourced to OpenAI, Anthropic and Perplexity crawler docs for the robots.txt tokens, and to OpenAI's own partnership announcements for the licensing deals.
Sources (primary): OpenAI, "Overview of OpenAI Crawlers" (GPTBot, OAI-SearchBot, OAI-AdsBot, ChatGPT-User, robots.txt directives); Anthropic, "Does Anthropic crawl data from the web, and how can site owners block the crawler?" (ClaudeBot, Disallow and Crawl-delay); Perplexity, "Perplexity Crawlers" (PerplexityBot, robots.txt handling). Licensing deals: OpenAI, News Corp partnership, Financial Times partnership, Guardian Media Group partnership, TIME partnership. AI behavior and deal terms evolve, so this entry is dated and revisited.