屏蔽恶意蜘蛛ajestic12.co.uk访问

作者:VPSAA技术部 发布时间:May 15, 2013 分类:教程

最近很多用户的访问日志中发现部分蜘蛛的痕迹,大量的占用资源,并且没有任何的意义。

比如大家看,这样子的访问日志:


108.59.8.80 - - [15/May/2013:16:37:17 +0800] "GET /tag/%E7%BE%8E%E4%B8%BD/feed HTTP/1.0" 200 1264 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)"
108.59.8.80 - - [15/May/2013:16:37:19 +0800] "GET /tag/%E8%80%8C%E5%A4%96/feed HTTP/1.0" 200 2414 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)"
108.59.8.80 - - [15/May/2013:16:37:36 +0800] "GET /tag/%E8%82%8C%E8%82%A4/feed HTTP/1.0" 200 4579 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)"
108.59.8.80 - - [15/May/2013:16:37:38 +0800] "GET /tag/%E8%84%B1%E7%9A%AE/feed HTTP/1.0" 200 4579 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)"
108.59.8.80 - - [15/May/2013:16:37:40 +0800] "GET /tag/%E8%8A%B1%E7%93%A3/feed HTTP/1.0" 200 1316 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)"
108.59.8.80 - - [15/May/2013:16:37:53 +0800] "GET /tag/%E8%9A%8A%E5%AD%90 HTTP/1.0" 200 1314 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)"
108.59.8.80 - - [15/May/2013:16:37:55 +0800] "GET /tag/%E8%9A%8A%E5%AD%90/feed HTTP/1.0" 200 6990 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)"
108.59.8.80 - - [15/May/2013:16:37:57 +0800] "GET /tag/%E8%A7%92%E8%B4%A8/feed HTTP/1.0" 500 589 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)"
108.59.8.80 - - [15/May/2013:16:38:25 +0800] "GET /tag/%E8%A7%A3%E6%95%91/feed HTTP/1.0" 200 5706 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)"
108.59.8.80 - - [15/May/2013:16:38:30 +0800] "GET /tag/%E8%AF%95%E7%94%A8/feed HTTP/1.0" 200 12297 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)"
108.59.8.80 - - [15/May/2013:16:38:57 +0800] "GET /tag/%E8%B4%9D%E5%A3%B3/feed HTTP/1.0" 200 3736 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)"
108.59.8.80 - - [15/May/2013:16:39:28 +0800] "GET /tag/%E8%B5%B0%E5%BC%80 HTTP/1.0" 200 3536 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)"
108.59.8.80 - - [15/May/2013:16:39:41 +0800] "GET /tag/%E8%B5%B0%E5%BC%80/feed HTTP/1.0" 200 4484 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)"
108.59.8.80 - - [15/May/2013:16:39:43 +0800] "GET /tag/%E8%BA%AB%E4%BD%93/feed HTTP/1.0" 200 4564 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)"
108.59.8.80 - - [15/May/2013:16:39:48 +0800] "GET /tag/%E8%BF%87%E6%95%8F/feed HTTP/1.0" 200 3734 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)"
108.59.8.80 - - [15/May/2013:16:39:50 +0800] "GET /tag/%E8%BF%90%E5%8A%A8/feed HTTP/1.0" 200 3735 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)"
108.59.8.80 - - [15/May/2013:16:39:57 +0800] "GET /tag/%E9%80%82%E7%94%A8/feed HTTP/1.0" 200 1235 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)"

上面是摘录的一个客户的访问日志中的片段。

大家可以看到大量的访问来路是http://www.majestic12.co.uk/bot.php?+

我们如何解决他呢?这个蜘蛛官方给了一个修改robots的方法,就是在robots.txt文件中加入:


User-agent: MJ12bot
Disallow: /

 

标签: www.majestic12.co.uk

添加新评论 »