How to block baidu spider!
Same problem with Baidu spider, that aggressive that my box ranked over 35 in my console using top. Obviously that even a fast computer cannot handle effectively outside requests running at 35….
Block baidu spider
I traced the number of IP’s (from that University building ????) to be several hundreds, with mainly two useragents)
Direct consequence ? As I have a cloud server I had to upgrade the same to higher memory in order to allow a decend response.
Previous answer :
Baidu seems totally unable to respect the robot.txt indication.
What I did:
I installed WP-Ban plugin for WordPress (free) and banned the following :
USER AGENTS :
Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
Furthermore using Wp Super Cache I re-address the relative error page to a static page, thus the whole wordpress installation does not / or at least only for the banned useragents check the Mysql datatable.
(This is standard WordPress blablabla, so everybody being able to install a WordPress Plugin can do it, as no coding or ftp access is required for this procedure)
I agree with everyone : Internet is free, banning whoever or whatever is absolutely the last thing anyone should do, but Baidoo today costs me USD 40 more/month , just to spider a webside written in Portughese, and I have some doubts if there are lots of Chinese people and visitors able to read and understand this language.