Easy PHP Blackhole Trap with WHOIS Lookup for Bad Bots

The concept is simple: include a hidden link to a robots.txt-forbidden directory somewhere on your pages. Bots that ignore or disobey your robots rules will crawl the link and fall into the trap, which then performs a WHOIS Lookup and records the event in the blackhole data file. Once added to the blacklist data file, bad bots immediately are denied access to your site. I call it the “one-strike” rule: bots have one chance to follow the robots.txt protocol, check the site’s robots.txt file, and obey its directives. Failure to comply results in immediate banishment. The best part is that the Blackhole only affects bad bots: normal users never see the hidden link, and good bots obey the robots rules in the first place.

1 UPs vs 0 DOWNs

Tags: php htaccess Bots scraping

Submitted by: JDStraughan Submitted on: Jul 20, 2010

Source: http://perishablepress.com/press/2010/07/14/blackhole-bad-bots/