I have open capacity for freelance work starting November 2026. Get in touch

The »No Model Training« Signal

I’m definitely not the first to write about it but I found the blog post by Cloudflare too theoretical that I want to write it up myself. It’s all about how to set signals to agent bots / crawlers how AIs / LLMs can access your website’s content or train their models with your content.

While I appreciate the help of an LLM to solve some tasks, I do not endorse how they are trained: With the data of people investing lot of effort into writing articles, summarizing documents, creating demo tutorials. Nowadays all for a LLM that does not even attribute it properly, or worse, mixes content and outputs wrong facts.

So here’s my signal to agent bots that they can show it as search result, can search my content but are not allowed to use it for training and also not in search summaries:

# robots.txt
# Indicate no AI training signal
Content-Signal: ai-train=no, search=yes, ai-input=no

That’s the future. But we’re not there yet. As of now, only Cloudflare supports this, effectively meaning no one cares much about it yet. So let’s take care of the fallbacks:

# As long as apart from Cloudflare noone else supports the Content-Signal, disallow some bots
# Notice that this is only disallowing training, not necessarily summaries, search 
# https://searchscore.io/guides/ai-crawlers-robots-txt/#:~:text=The%20major%20AI%20crawler%20user%2Dagents
# First: Apple Training bot
User-agent: Applebot-Extended
Disallow: /
# Now: Google AI Training bot
User-agent: Google-Extended
Disallow: /
# Then: GPTBot (this is the training bot)
User-agent: GPTBot 
Disallow: /
# And: Claude
User-agent: ClaudeBot
Disallow: /
# And Bing / MS not here but in HTML head because they do not support it here in robots.txt

Unfortunately, that doesn’t seem to be enough yet. Some of the bots do not respect this but offer other methods, so let’s extend our HTML head:

<!-- To avoid archiving / AI training by Bing agent crawler -->
<meta name="robots" content="noarchive" />

There’s a IETF Internet Draft available that documents the new signals. And while their announcement post is pretty poor in information, Cloudflare has a dedicated Contentsignals site set up that explains this all pretty well and interactively.

Finally, to test your new signals, you can check it via curl by setting the user agent to the crawler’s name, e.g. curl -A "Google-Extended" -I https://yourwebsite.com/. If you get a status 200, it has allowance, otherwise not.

Get in touch

Tell me a bit about your project or question. I’ll get back to you as soon as I can.