Tips & Guides
What Is a Robots.txt File and How to Create One That Blocks AI Crawlers in 2025
A robots.txt file controls which pages search engines and AI bots can crawl. Here is how to create one correctly — including how to block GPTBot and other AI crawlers.