Robots txt

Guides

Block AI crawlers: robots.txt, server rules and verification

How to block AI crawlers: which bots train and which cite you, copy-paste robots.txt and server rules, bot verification, and what to do about bots that lie.13 min read