2018各大搜索引擎蜘蛛特征表 (baidu蜘蛛,google蜘蛛,360蜘蛛,bing蜘蛛,yahoo蜘蛛)

  • 内容
  • 相关

转载请注明本文链接:https://blog.l9s.cc/.html  [生成短网址]

2018最新搜集各大搜索引擎蜘蛛特征

360Spider
Sosospider
SogouNewsSpider
Sogouwebspider
Sogouinstspider
Sogouspider2
Sogoublog
SogouOrionspider
sogouspider
Sogou web spider
YodaoBot
Googlebot
Bingbot
Slurp
Teoma
ia_archiver
twiceler
MSNBot
Scrubby
Robozilla
Gigabot
googlebot-image
googlebot-mobile
yahoo-mmcrawler
yahoo-blogs/v3.9
psbot
Nutch
naverbot
yeti
asterias
AhrefsBot
BLEXBot
MJ12bot
bingbot
netEstate
YisouSpider
Yahoo!SlurpChina
YoudaoBot
msnbot
msnbot-media
EasouSpider
JikeSpider
EtaoSpider
YandexBot
Google-Test

MegaIndex.ru

timg (1).jpg


1、百度蜘蛛:Baiduspider
网上的资料百度蜘蛛名称有BaiduSpider、baiduspider等,都洗洗睡吧,那是旧黄历了。百度蜘蛛最新名称为Baiduspider。日志中还发现了Baiduspider-image这个百度旗下蜘蛛,查了下资料(其实直接看名字就可以了……),是抓取图片的蜘蛛。
常见百度旗下同类型蜘蛛还有下面这些:Baiduspider-mobile(抓取wap)、Baiduspider-image(抓取图片)、Baiduspider-video(抓取视频)、Baiduspider-news(抓取新闻)。
注:以上百度蜘蛛目前常见的是Baiduspider和Baiduspider-image两种。

2、谷歌蜘蛛:Googlebot
这个争议较少,但也有说是GoogleBot的。谷歌蜘蛛最新名称为“compatible; Googlebot/2.1;”。还发现了Googlebot-Mobile,看名字是抓取wap内容的。

3、360蜘蛛:360Spider,它是一个很“勤奋抓爬”的蜘蛛。
4、SOSO蜘蛛:Sosospider,也可为它颁一个“勤奋抓爬”奖的蜘蛛。
5、雅虎蜘蛛:“Yahoo! Slurp China”或者Yahoo! 
名称中带“Slurp”和空格,名称有空格robots里名称可以使用“Slurp”或者“Yahoo”单词描述,不知道有效无效。

6、有道蜘蛛:YoudaoBot,YodaoBot(两个名字都有,中文拼音少了个U字母读音差别很大嘎,这都会少?)
7、搜狗蜘蛛:Sogou News Spider
搜狗蜘蛛还包括如下这些:Sogou web spider、Sogou inst spider、Sogou spider2、Sogou blog、Sogou News Spider、Sogou Orion spider,
(参考一些网站的robots文件,搜狗蜘蛛名称可以用Sogou概括,无法验证不知道有没有效)
看看最权威的百度的robots.txt ,http://www.baidu.com/robots.txt 就为Sogou搜狗蜘蛛费了不少字节,占了一大块领地。
“Sogou web spider;Sogou inst spider;Sogou spider2;Sogou blog;Sogou News Spider;Sogou Orion spider”目前6个,名称都带空格。
线上常见”Sogou web spider/4.0″ ;”Sogou News Spider/4.0″ ;”Sogou inst spider/4.0″ 可以为它颁个“占名为王”奖。
8、MSN蜘蛛:msnbot,msnbot-media(只见到msnbot-media在狂爬……)

9、必应蜘蛛:bingbot 
线上(compatible; bingbot/2.0;)
10、一搜蜘蛛:YisouSpider

11、Alexa蜘蛛:ia_archiver

12、宜sou蜘蛛:EasouSpider

13、即刻蜘蛛:JikeSpider

14、一淘网蜘蛛:EtaoSpider
“Mozilla/5.0 (compatible; EtaoSpider/1.0; http://省略/EtaoSpider)”
根据上述蜘蛛中选择几个常用的允许抓取,其余的都可以通过robots屏蔽抓取。如果你暂时空间流量还足够使用,等流量紧张了就保留几个常用的屏蔽掉其它蜘蛛以节省流量。至于那些蜘蛛抓取对网站能带来有利用的价值,网站的管理者眼睛是雪亮的。

另外还发现了如 YandexBot、AhrefsBot和ezooms.bot这些蜘蛛,据说这些蜘蛛国外,对中文网站用处很小。那不如就节省下资源。

最后再附上360搜索 so.com的蜘蛛IP段


101.226.166.195 101.226.169.228 180.153.236.113 182.118.25.105 182.118.20.228
101.226.166.196 101.226.169.229 180.153.236.114 169.254.95.120 182.118.20.215
101.226.166.197 101.226.169.230 180.153.236.115 182.118.25.20 182.118.20.216
101.226.166.198 180.153.231.111 180.153.236.116 182.118.29.148 182.118.20.217
101.226.166.199 101.226.166.219 180.153.236.59 182.118.35.16 182.118.20.218
101.226.166.200 101.226.166.220 180.153.236.60 182.118.35.17 182.118.20.219
101.226.166.201 101.226.166.221 180.153.236.61 182.118.35.18 182.118.22.144
101.226.166.202 101.226.166.222 180.153.236.62 182.118.35.19 182.118.25.216
180.153.231.56 101.226.166.223 180.153.236.63 182.118.25.15 182.118.25.217
101.226.169.215 101.226.166.224 180.153.236.64 182.118.29.147 182.118.25.218
101.226.169.216 101.226.166.225 180.153.236.65 182.118.35.11 182.118.25.219
101.226.169.217 101.226.166.226 180.153.236.66 182.118.35.12 182.118.25.220
101.226.169.218 180.153.231.77 180.153.227.108 182.118.35.13 182.118.22.145
101.226.169.219 101.226.166.243 180.153.236.189 182.118.35.14 182.118.25.221
101.226.169.220 101.226.166.244 180.153.236.190 169.254.95.120 182.118.25.222
101.226.169.221 101.226.166.245 180.153.236.191 182.118.29.150 182.118.25.223
101.226.169.222 101.226.166.246 180.153.236.192 182.118.35.26 182.118.25.224
180.153.231.108 101.226.166.247 180.153.236.193 182.118.35.27 182.118.25.225
101.226.166.251 101.226.166.248 180.153.236.194 182.118.35.28 182.118.22.221
101.226.166.252 101.226.166.249 180.153.236.195 182.118.35.29 182.118.22.222
101.226.166.253 101.226.166.250 180.153.236.196 182.118.35.30 182.118.22.223
101.226.166.254 180.153.231.89 180.153.227.105 169.254.95.120 182.118.22.224
101.226.167.195 101.226.167.207 180.153.236.165 182.118.25.25 182.118.22.225
101.226.167.196 101.226.167.208 180.153.236.166 182.118.29.149 182.118.22.231
101.226.167.197 101.226.167.209 180.153.236.167 182.118.35.21 182.118.22.232
101.226.167.198 101.226.167.210 180.153.236.168 182.118.35.22 182.118.22.233
180.153.231.90 101.226.167.211 180.153.236.169 182.118.35.23 182.118.22.234
101.226.168.195 101.226.167.212 180.153.236.170 182.118.35.24 182.118.22.235
101.226.168.196 101.226.167.213 180.153.236.171 182.118.25.186 182.118.22.236
101.226.168.197 101.226.167.214 180.153.236.172 182.118.25.187 182.118.22.237
101.226.168.198 180.153.231.92 180.153.236.51 182.118.25.188 182.118.22.238
101.226.168.199 101.226.166.203 180.153.236.52 182.118.25.189 182.118.22.239
101.226.168.200 101.226.166.204 180.153.236.53 182.118.25.190 182.118.22.240
101.226.168.201 101.226.166.205 180.153.236.54 182.118.25.181 182.118.22.241
101.226.168.202 101.226.166.206 180.153.236.55 182.118.25.182 182.118.22.242
180.153.231.98 101.226.166.207 180.153.236.56 182.118.25.183 182.118.22.243
101.226.167.247 101.226.166.208 180.153.236.57 182.118.25.184 182.118.22.244
101.226.167.248 101.226.166.209 180.153.236.58 182.118.25.185 182.118.22.245
101.226.167.249 101.226.166.210 180.153.227.104 182.118.25.136 182.118.22.246
101.226.167.250 180.153.231.57 180.153.236.157 182.118.25.137 182.118.22.247
101.226.167.251 101.226.167.215 180.153.236.158 182.118.25.138 182.118.22.248
101.226.167.252 101.226.167.216 180.153.236.159 182.118.25.139 182.118.22.249
101.226.167.253 101.226.167.217 180.153.236.160 182.118.25.140 182.118.22.250
101.226.167.254 101.226.167.218 180.153.236.161 182.118.25.141 182.118.22.141
180.153.231.97 101.226.167.219 180.153.236.162 182.118.25.142 182.118.25.201
101.226.168.203 101.226.167.220 180.153.236.163 182.118.25.143 182.118.25.202
101.226.168.204 101.226.167.221 180.153.236.164 182.118.25.144 182.118.25.203
101.226.168.205 101.226.167.222 180.153.236.149 182.118.25.145 182.118.25.204
101.226.168.206 180.153.231.93 180.153.236.150 182.118.25.176 182.118.25.205
101.226.168.207 101.226.169.199 180.153.236.151 182.118.25.177 182.118.20.203
101.226.168.208 101.226.169.200 180.153.236.152 182.118.25.178 182.118.20.229
101.226.168.209 101.226.169.201 180.153.236.153 182.118.25.179 182.118.20.230
101.226.168.210 101.226.169.202 180.153.236.154 182.118.25.180 182.118.20.231
180.153.231.99 101.226.169.203 180.153.236.155 182.118.25.171 182.118.20.232
101.226.167.223 101.226.169.204 180.153.236.156 182.118.25.172 182.118.20.202
101.226.167.224 101.226.169.205 180.153.236.133 182.118.25.173 182.118.20.220
101.226.167.225 101.226.169.206 180.153.236.134 182.118.25.174 182.118.20.221
101.226.167.226 180.153.231.106 180.153.236.135 182.118.25.175 182.118.20.222
101.226.167.227 101.226.168.243 180.153.236.136 182.118.25.196 182.118.20.223
101.226.167.228 101.226.168.244 180.153.236.137 182.118.25.197 182.118.22.146
101.226.167.229 101.226.168.245 180.153.236.138 182.118.25.198 182.118.25.226
101.226.167.230 101.226.168.246 180.153.236.139 182.118.25.199 182.118.25.227
180.153.231.94 101.226.168.247 180.153.236.140 182.118.25.200 182.118.25.228
101.226.168.211 101.226.168.248 180.153.227.107 182.118.25.191 182.118.25.229
101.226.168.212 101.226.168.249 180.153.236.181 182.118.25.192 182.118.25.230
101.226.168.213 101.226.168.250 180.153.236.182 182.118.25.193 182.118.22.226
101.226.168.214 180.153.231.104 180.153.236.183 182.118.25.194 182.118.22.227
101.226.168.215 101.226.167.231 180.153.236.184 182.118.25.195 182.118.22.228
101.226.168.216 101.226.167.232 180.153.236.185 182.118.22.143 182.118.22.229
101.226.168.217 101.226.167.233 180.153.236.186 182.118.25.211 182.118.22.230
101.226.168.218 101.226.167.234 180.153.236.187 182.118.25.212 182.118.22.216
180.153.231.100 101.226.167.235 180.153.236.188 182.118.25.213 182.118.22.217
101.226.166.211 101.226.167.236 180.153.236.101 182.118.25.214 182.118.22.218
101.226.166.212 101.226.167.237 180.153.236.102 182.118.25.215 182.118.22.219
101.226.166.213 101.226.167.238 180.153.236.103 182.118.20.233 182.118.22.220
101.226.166.214 180.153.231.95 180.153.236.104 182.118.20.234 182.118.22.142
101.226.166.215 101.226.167.199 180.153.236.105 182.118.20.235 182.118.25.206
101.226.166.216 101.226.167.200 180.153.236.106 182.118.20.236 182.118.25.207
101.226.166.217 101.226.167.201 180.153.236.107 182.118.20.237 182.118.25.208
101.226.166.218 101.226.167.202 180.153.236.108 182.118.22.147 182.118.25.209
180.153.231.58 101.226.167.203 180.153.236.117 182.118.25.231 182.118.25.210
101.226.166.235 101.226.167.204 180.153.236.118 182.118.25.232 182.118.20.201
101.226.166.236 101.226.167.205 180.153.236.119 182.118.25.233 182.118.20.211
101.226.166.237 101.226.167.206 180.153.236.120 182.118.25.234 182.118.20.212
101.226.166.238 180.153.231.91 180.153.236.121 182.118.25.235 182.118.20.213
101.226.166.239 101.226.167.239 180.153.236.122 182.118.21.231 182.118.20.214
101.226.166.240 101.226.167.240 180.153.236.123 182.118.21.232 182.118.22.211
101.226.166.241 101.226.167.241 180.153.236.124 182.118.21.233 182.118.22.212
101.226.166.242 101.226.167.242 180.153.236.67 182.118.21.234 182.118.22.213
180.153.231.88 101.226.167.243 180.153.236.68 182.118.21.235 182.118.22.214
101.226.168.235 101.226.167.244 180.153.236.69 182.118.21.236 182.118.22.215
101.226.168.236 101.226.167.245 180.153.236.70 182.118.21.237 182.118.20.206
101.226.168.237 101.226.167.246 180.153.236.71 182.118.21.238 182.118.21.220
101.226.168.238 180.153.231.96 180.153.236.72 182.118.21.239 182.118.21.221
101.226.168.239 101.226.168.251 180.153.236.73 182.118.21.240 182.118.21.222
101.226.168.240 101.226.168.252 180.153.236.74 182.118.20.205 182.118.21.223
101.226.168.241 101.226.168.253 180.153.236.35 182.118.21.201 182.118.21.224
101.226.168.242 101.226.168.254 180.153.236.36 182.118.21.202 182.118.21.225
180.153.231.103 101.226.169.195 180.153.236.37 182.118.21.203 182.118.21.226
101.226.166.227 101.226.169.196 180.153.236.38 182.118.21.204 182.118.21.227
101.226.166.228 101.226.169.197 180.153.236.39 182.118.21.205 182.118.21.228
101.226.166.229 101.226.169.198 180.153.236.40 182.118.21.206 182.118.21.229
101.226.166.230 180.153.231.105 180.153.236.41 182.118.21.207 182.118.21.230
101.226.166.231 180.153.236.19 180.153.236.42 182.118.21.208 182.118.21.245
101.226.166.232 180.153.236.20 180.153.227.106 182.118.21.209 182.118.21.246
101.226.166.233 180.153.236.21 180.153.236.173 182.118.21.210 182.118.21.247
101.226.166.234 180.153.236.22 180.153.236.174 182.118.21.211 182.118.21.248
180.153.231.78 180.153.236.23 180.153.236.175 182.118.21.212 182.118.21.249
101.226.169.207 180.153.236.24 180.153.236.176 182.118.21.213 182.118.21.250
101.226.169.208 180.153.236.25 180.153.236.177 182.118.21.214 182.118.21.251
101.226.169.209 180.153.236.26 180.153.236.178 182.118.21.215 182.118.21.252
101.226.169.210 180.153.236.125 180.153.236.179 182.118.21.216 182.118.21.253
101.226.169.211 180.153.236.126 180.153.236.180 182.118.21.217 182.118.21.254
101.226.169.212 180.153.236.127 180.153.236.141 182.118.21.218 182.118.22.148
101.226.169.213 180.153.236.128 180.153.236.142 182.118.21.219 182.118.25.236
101.226.169.214 180.153.236.129 180.153.236.143 182.118.20.207 182.118.25.237
180.153.231.107 180.153.236.130 180.153.236.144 182.118.20.249 182.118.25.238
101.226.168.227 180.153.236.131 180.153.236.145 182.118.20.250 182.118.25.239
101.226.168.228 180.153.236.132 180.153.236.146 182.118.20.251 182.118.25.240
101.226.168.229 180.153.231.113 180.153.236.147 182.118.20.252 182.118.25.131
101.226.168.230 180.153.236.11 180.153.236.148 182.118.20.253 182.118.25.132
101.226.168.231 180.153.236.12 182.118.25.151 182.118.20.254 182.118.25.133
101.226.168.232 180.153.236.13 182.118.25.152 182.118.21.241 182.118.25.134
101.226.168.233 180.153.236.14 182.118.25.153 182.118.21.242 182.118.25.135
101.226.168.234 180.153.236.15 182.118.25.154 182.118.21.243 182.118.22.149
180.153.231.102 180.153.236.16 182.118.25.155 182.118.21.244 182.118.25.241
101.226.168.219 180.153.236.17 182.118.25.161 182.118.22.201 182.118.25.242
101.226.168.220 180.153.236.18 182.118.25.162 182.118.22.202 182.118.25.243
101.226.168.221 180.153.236.43 182.118.25.163 182.118.22.203 182.118.25.244
101.226.168.222 180.153.236.44 182.118.25.164 182.118.22.204 182.118.25.245
101.226.168.223 180.153.236.45 182.118.25.165 182.118.22.205 182.118.25.146
101.226.168.224 180.153.236.46 182.118.22.196 182.118.22.206 182.118.25.147
101.226.168.225 180.153.236.47 182.118.22.197 182.118.22.207 182.118.25.148
101.226.168.226 180.153.236.48 182.118.22.198 182.118.22.208 182.118.25.149
180.153.231.101 180.153.236.49 182.118.22.199 182.118.22.209 182.118.25.150
101.226.169.223 180.153.236.50 182.118.22.200 182.118.22.210 182.118.25.156
101.226.169.224 180.153.236.109 182.118.25.101 182.118.20.224 182.118.25.157
101.226.169.225 180.153.236.110 182.118.25.102 182.118.20.225 182.118.25.158
101.226.169.226 180.153.236.111 182.118.25.103 182.118.20.226 182.118.25.159
101.226.169.227 180.153.236.112 182.118.25.104 182.118.20.227 182.118.25.160
42.236.99.2 42.236.99.122 42.236.99.62 42.236.99.185 42.236.99.56
42.236.99.3 42.236.99.123 42.236.99.63 42.236.99.186 42.236.99.57
42.236.99.4 42.236.99.124 42.236.99.64 42.236.99.187 42.236.99.58
42.236.99.5 42.236.99.125 42.236.99.65 42.236.99.188 42.236.99.59
42.236.99.6 42.236.99.126 42.236.99.66 42.236.99.189 42.236.99.60
42.236.99.7 42.236.99.130 42.236.99.67 42.236.99.190 42.236.99.61
42.236.99.8 42.236.99.131 42.236.99.68 42.236.99.194 42.236.99.250
42.236.99.9 42.236.99.132 42.236.99.69 42.236.99.195 42.236.99.253
42.236.99.10 42.236.99.133 42.236.99.70 42.236.99.196 42.236.99.179
42.236.99.11 42.236.99.134 42.236.99.71 42.236.99.197 42.236.99.180
42.236.99.12 42.236.99.135 42.236.99.72 42.236.99.198 42.236.99.181
42.236.99.13 42.236.99.136 42.236.99.73 42.236.99.199 42.236.99.182
42.236.99.14 42.236.99.137 42.236.99.74 42.236.99.200 42.236.99.183
42.236.99.15 42.236.99.138 42.236.99.75 42.236.99.201 42.236.99.184
42.236.99.16 42.236.99.139 42.236.99.76 42.236.99.202 42.236.99.251
42.236.99.17 42.236.99.140 42.236.99.77 42.236.99.203 42.236.99.252
42.236.99.18 42.236.99.141 42.236.99.78 42.236.99.204 42.236.99.116
42.236.99.19 42.236.99.142 42.236.99.79 42.236.99.205 42.236.99.117
42.236.99.20 42.236.99.143 42.236.99.80 42.236.99.206 42.236.99.118
42.236.99.21 42.236.99.144 42.236.99.81 42.236.99.207 42.236.99.119
42.236.99.22 42.236.99.145 42.236.99.82 42.236.99.208 42.236.99.120
42.236.99.23 42.236.99.146 42.236.99.83 42.236.99.209 42.236.99.121
42.236.99.24 42.236.99.147 42.236.99.84 42.236.99.210 42.236.99.254
42.236.99.25 42.236.99.148 42.236.99.85 42.236.99.211 42.236.99.249
42.236.99.26 42.236.99.149 42.236.99.86 42.236.99.212 42.236.99.242
42.236.99.27 42.236.99.150 42.236.99.87 42.236.99.213 42.236.99.243
42.236.99.28 42.236.99.151 42.236.99.88 42.236.99.214 42.236.99.244
42.236.99.29 42.236.99.152 42.236.99.89 42.236.99.215 42.236.99.245
42.236.99.30 42.236.99.153 42.236.99.90 42.236.99.216 42.236.99.246
42.236.99.31 42.236.99.154 42.236.99.91 42.236.99.217 42.236.99.247
42.236.99.32 42.236.99.155 42.236.99.92 42.236.99.218 42.236.99.248
42.236.99.33 42.236.99.156 42.236.99.93 42.236.99.219 42.236.99.53
42.236.99.34 42.236.99.157 42.236.99.94 42.236.99.220 42.236.99.54
42.236.99.35 42.236.99.158 42.236.99.95 42.236.99.221 42.236.99.55
42.236.99.36 42.236.99.159 42.236.99.96 42.236.99.222 42.236.99.176
42.236.99.37 42.236.99.160 42.236.99.97 42.236.99.223 42.236.99.177
42.236.99.38 42.236.99.161 42.236.99.98 42.236.99.224 42.236.99.178
42.236.99.39 42.236.99.162 42.236.99.99 42.236.99.225 42.236.99.113
42.236.99.40 42.236.99.163 42.236.99.100 42.236.99.226 42.236.99.114
42.236.99.41 42.236.99.164 42.236.99.101 42.236.99.227 42.236.99.115
42.236.99.42 42.236.99.165 42.236.99.102 42.236.99.228 42.236.99.239
42.236.99.43 42.236.99.166 42.236.99.103 42.236.99.229 42.236.99.240
42.236.99.44 42.236.99.167 42.236.99.104 42.236.99.230 42.236.99.241
42.236.99.45 42.236.99.168 42.236.99.105 42.236.99.231 42.236.99.52
42.236.99.46 42.236.99.169 42.236.99.106 42.236.99.232 42.236.99.175
42.236.99.47 42.236.99.170 42.236.99.107 42.236.99.233 42.236.99.112
42.236.99.48 42.236.99.171 42.236.99.108 42.236.99.234 42.236.99.238
42.236.99.49 42.236.99.172 42.236.99.109 42.236.99.235 42.236.99.237
42.236.99.50 42.236.99.173 42.236.99.110 42.236.99.236 42.236.99.111
42.236.99.51 42.236.99.174



本文标签:

版权声明:若无特殊注明,本文皆为《王晓明》原创,转载请保留文章出处。

本文链接:2018各大搜索引擎蜘蛛特征表 (baidu蜘蛛,google蜘蛛,360蜘蛛,bing蜘蛛,yahoo蜘蛛) - https://blog.l9s.cc/SEO优化/132.html

发表评论

电子邮件地址不会被公开。 必填项已用*标注