Block search engines and AI from just one of my blogs

There are a few plug-ins in the directory if you search for robots.txt. The one I’ve built is called Custom Robots, and it gives you full control over the file’s content. There are many initiatives that maintain lists of AI-related crawlers, like ai.crawlers.txt. If you want to discourage all crawlers, including traditional search engines, you can use this simple rule: User-agent: * Disallow: / Or just install Manton’s plug-in, No Robots. If you want to stop crawlers for real, you’ll need to block their IP addresses from connecting to your web server. That’s something Manton needs to do, or you’ll need to put a reverse proxy in front of your blog. It’s pretty technical, and you’ll need to constantly keep up with which IP addresses the AI crawlers are using, as they will jump around as they get blocked. It’s a cat-and-mouse game.