Blocking Bad Bots With AbuseIPD Blacklist

Sean Conner at The Boston Diaries wrote about trying to block annoying and/or malicious bots from crawling his website. It is a good read.

Beginning in late 2024, I started noticing The New Leaf Journal going down periodically. The server logs suggested that it was being overrun by bots and crawlers. I tried various methods to ensure that NLJ would be up all the time, including fiddling with Apache configs and my .htaccess file. I stumbled upon the “solution” (for now, at least) in late January. I use Cloudron to manage this server (which is home to both this site and NLJ). Cloudron has a built-in firewall. Back on January 30, I published Using Cloudron’s IP Firewall. I populated its firewall with IPs from the AbuseIPD blacklist, which is readily available in LittleJake’s GitHub repository. NLJ’s resource usage went way down(see the chart in my linked post) and my crashing issues have been gone for more than two months. The AbuseIPD blacklist is updated approximately every six hours. I have been adding the IPs manually but I have been meaning to look into a way to automate the process.

You can reply to this article from your own site by sending a Webmention.