Glossary

AI Bot Blocking

What is AI Bot Blocking?

AI bot blocking means preventing AI providers' crawlers from reading your website, usually through rules in the robots.txt file for user agents such as GPTBot, ClaudeBot or Google-Extended. The file is a request, not a technical barrier; stricter control requires firewall or server rules. The trade-off: blocking protects content from being used for training but can reduce visibility in AI search and AI-generated answers. It often makes sense to treat training crawlers and search crawlers differently.

Topics

robots.txt
Crawlers
GEO
Visibility

Try KI.ordo

Agents, flows and templates in one workspace – free 24-hour demo.

Start demo

← All terms

Related terms

Generative engine optimisation (GEO)

Generative engine optimisation (GEO) covers measures that help content become visible in, and be cited as a source by, AI-powered search systems such as ChatGPT, Perplexity or the AI overviews in search engines. The term comes from a 2023 research paper. Key factors include clear, directly quotable answers, a clean structure, evidenced facts, up-to-date information and identifiable authors. GEO complements classic search engine optimisation (SEO) rather than replacing it.

AI search
SEO
+1

Template

llms.txt

llms.txt is a Markdown text file placed in a website’s root directory that gives language models a concise overview of the most important content, with links. The idea was put forward by Jeremy Howard (Answer.AI) in September 2024. It is a proposal rather than an official standard; major AI providers have not committed to reading it (as of October 2026). It complements robots.txt and the sitemap but does not replace them.

website
AI visibility
+1

Zero-Click Search

A zero-click search occurs when users get their answer directly on the results page, for example through featured answer boxes or AI-generated summaries, and no longer click through to a website. For businesses this reduces traffic even when their content serves as the source. Being visible as a cited source therefore becomes more important: with clear answers, supported facts and well-structured pages.

Template

AI Search

AI search refers to search methods that use AI to grasp the intent behind a query, weight results and often generate a summarised answer with sources rather than a plain list of links. Technically, they use keyword search, vector search over embeddings or a combination, frequently followed by a language model. Examples include answer engines on the web and internal knowledge search that helps staff find what they need in manuals and policies.

Search
Knowledge
+1