· Notes for owners
Robots.txt AI limits: which page tags control Google's AI answers
Google says robots.txt governs crawling only. Here is what owners can do with snippet tags to limit what AI answers use from a page.
On October 22, 2026, Google's Gary Illyes said in a Search Central video that robots.txt, as RFC 9309 defines it, governs which URLs a crawler may request. It says nothing about what happens to the content afterward. Search Engine Journal covered the episode, including his advice that blocking one AI crawler means naming that crawler's user-agent token in the file.
What this means for you
The robots.txt file is not a master switch for AI, because RFC 9309 defines it as governing which URLs a crawler may request. It is a door policy. It decides who may fetch a page. Once a crawler is in, the file does not dictate what the system does with the data.
The Google-Extended token is a good example. It is a robots.txt token that controls whether content Google crawls can be used to train future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI. According to Google's list of crawlers, it does not affect your inclusion in Google Search and is not a ranking signal. Its meaning comes from Google's own product documentation, not from the robots.txt standard.
So robots.txt is one control, for one question. The question owners more often care about is different: how much of my page can an AI answer in Search show? For that, Google points to page-level controls. Per Google's page on AI features, nosnippet, data-nosnippet, max-snippet and noindex limit the information shown from a page in Search.
They work like this:
- nosnippet prevents a snippet from being shown, per Google's snippet guide.
- max-snippet sets the maximum length of that snippet.
- data-nosnippet is an attribute you put on individual HTML elements, so one paragraph or block can be held back while the rest of the page stays quotable. It is described in the robots meta tag specification.
- noindex limits the information shown from a page in Search.
The first three are the fine-grained tools. The last is the blunt one.
What to do, in order
Hand this list to whoever touches your website.
- Look at what is already on your pages. Open a few posts and your main service pages, view the page source, and search for nosnippet, max-snippet, noindex and data-nosnippet. Do the same for the template that generates every post, because that is where a single line affects every page.
- Decide, page type by page type, what you want. Blog posts and service pages that exist to win customers usually want to be quoted. Member-only material, internal documents or client lists may not. Write the decision down in one line per page type.
- Remove anything that contradicts the decision. If a page is meant to be found and cited, it should carry no nosnippet and no restrictive max-snippet.
- Use data-nosnippet for the exceptions. If one block on an otherwise public page should not be excerpted, mark that block and leave the page alone.
- Keep robots.txt for the crawl question. If you want a specific AI crawler kept out, name its user-agent token. A made-up line that no crawler recognizes does nothing. Google's specification supports four elements: User-Agent, Allow, Disallow and Sitemap, and ignores invalid lines.
- Percent-encode any non-English paths in your robots.txt rules. Google's Cherry Prommawin recommended this for crawlers other than Google's.
The trap
The most common mistake is a template-wide nosnippet or max-snippet left over from an old SEO experiment. Someone tests a setting, it works or it doesn't, and nobody removes it. From then on, every post on the site carries the restriction, and every post is held back from Google's AI answers. Nothing looks broken. Traffic from that source just never arrives.
A second one is noarchive or nocache on pages you want shown in Microsoft Copilot. Bing's webmaster guidelines cover how these affect Copilot. The safe habit is to remove both from any page you want cited unless you have a reason to keep them.
Where the controls stop
Two limits are worth knowing about.
First, the snippet controls above are documented for Google's products. Google's pages on them do not state that they apply to other companies' products, so treat nosnippet and max-snippet as Google controls.
Second, a standard way to state how content may be used does not exist yet. Illyes co-wrote an IETF draft that would add a Content-Usage rule to robots.txt and a matching HTTP header, so you could allow crawling and still say you do not want AI training. The current version is dated Aug. 19 and would update RFC 9309 only if approved. Do not write a Content-Usage line today and assume it protects anything. Whether it does depends on whether a crawler chooses to read it.
Where this leaves you
The practical picture is modest. Robots.txt handles who may fetch your pages. Snippet tags handle how much of a page Google's search features may show. Neither one is a general rule about AI, and Google documents its snippet controls for Google's own products. One review of your templates, with a written decision for each page type, sets these controls on purpose rather than by leftover.
Our work is being the technology department for owners who do not have one: we map how your business runs, build the software that fits it, and run it for you. If you would like someone to check what your site tells search and AI systems today, book a discovery call.