今天跟大家分享下最近研究的python爬取易車網(wǎng)的車型數(shù)據(jù). 本來app上有現(xiàn)成的api來獲取,但是沒有停售的車型數(shù)據(jù),所以只好現(xiàn)學現(xiàn)用了.
先上代碼傳送門
思路
品牌:
關閉Chrome的JavaScript支持,打開車型頁發(fā)現(xiàn)品牌數(shù)據(jù)并沒有加載出來,由此可以斷定品牌數(shù)據(jù)是通過JavaScript動態(tài)加載出來的,打開Chrome的開發(fā)工具分析js
image.png
可以看出品牌數(shù)據(jù)來源于接口 http://api.car.bitauto.com/CarInfo/getlefttreejson.ashx?tagtype=chexing&pagetype=masterbrand&objid=0,解析該接口返回的數(shù)據(jù)就可以拿到所有的品牌數(shù)據(jù). 但是有點小坑的是返回的數(shù)據(jù)不是標準的json,需要截取里面的json部分并為所有key加上雙引號才能正常解析.
車系:
在網(wǎng)站上隨便點開一個品牌,比如http://car.bitauto.com/tree_chexing/mb_9/,通過修改瀏覽器的JavaScript設置然后重載頁面,發(fā)現(xiàn)車系的數(shù)據(jù)沒有受到影響,那么我們就可以正常使用response.xpath()來爬取頁面. 查看網(wǎng)頁的源代碼找到元素位置
image.png
查看不同的品牌發(fā)現(xiàn)有的車系有供應商而有的車系則沒有,那么處理這部分標簽的時候需要區(qū)別對待.
車型:
網(wǎng)頁上進入車型詳情頁可以點擊左側的車型進入,也可以點擊車系詳情中的車型進入詳情頁,所以我們在解析車系的時候就可以順便爬取車型頁了.
image.png
車型的數(shù)據(jù)分為在售車型和停售車型,在售車型可以通過解析頁面的標簽獲取,但是停售車型需要通過接口http://car.bitauto.com/AjaxNew/GetNoSaleSerailListByYear.ashx?csID=2593&year=2017來取得,這個接口返回來的是標準的json可以直接使用.
Spider的全部代碼
# coding=utf-8
import json
import re
import scrapy
from scrapy.spiders import Rule
from scrapy.linkextractors import LinkExtractor
from yiche.items import BrandItem, SerialItem, ModelItem
# json替換key
def replacea(matched):
return '\"' + matched.group('value') + '\":'
# 解析車系Item
def parse_serial_item(serial, bid, vendor):
item = SerialItem()
item['id'] = serial.xpath('div/div/a/@id')[0].re(r'n(\d+)')[0]
item['bid'] = bid
item['name'] = serial.xpath('div/div/a/@title')[0].extract()
item['vendor'] = vendor
item['logo'] = serial.xpath('div/div/a/img/@src')[0].extract()
sell = serial.xpath('div/ul/li[@class="price"]/a/text()')[0].re(ur'停售')
item['sell'] = '0' if sell else '1'
return item
class YiCheSpider(scrapy.Spider):
name = "yiche"
rules = (
# 所有車系
Rule(LinkExtractor(allow=(r'http://car\.bitauto\.com/tree_chexing/mb_\d+/$',)), callback='parse_serial', follow=True),
# 在售車型
Rule(LinkExtractor(allow=(r'http://car\.bitauto\.com/\w+/$',)), callback='parse_model', follow=True),
# 停售車型
Rule(LinkExtractor(allow=(r'http://car\.bitauto\.com/AjaxNew/GetNoSaleSerailListByYear\.ashx?csID=\d+&year=\d+$',)),
callback='parse_model_selled', follow=True),
)
def start_requests(self):
url = 'http://api.car.bitauto.com/CarInfo/getlefttreejson.ashx?tagtype=chexing&pagetype=masterbrand&objid=0'
yield scrapy.Request(url, callback=self.parse, dont_filter=True)
# 解析品牌
def parse(self, response):
print '==> %s' % response.url
result = re.sub('(?P<value>\w+):', replacea, response.text[response.text.find('{'):response.text.rfind('}') + 1])
data = json.loads(result)
for char in data['char']:
print '==> %s' % char
try:
for brand in data['brand']['%s' % char]:
item = BrandItem()
item['id'] = brand['id']
item['name'] = brand['name']
item['logo'] = 'http://image.bitautoimg.com/bt/car/default/images/logo/masterbrand/png/100/m_%s_100.png' % item['id']
item['initial'] = char
yield item
url = 'http://car.bitauto.com/tree_chexing/mb_%s/' % item['id']
request = scrapy.Request(url, callback=self.parse_serial, dont_filter=True)
request.meta['bid'] = item['id']
yield request
except KeyError:
pass
# 解析車系
def parse_serial(self, response):
print '==> %s' % response.url
bid = response.meta['bid']
# 品牌下的全部車系及車型
brands = response.xpath('//*[@id="divCsLevel_0"]/*')
size = len(brands)
if size % 2 == 0:
for i in range(size / 2):
vendor = brands[i * 2].xpath('a/text()')[0].re(r'(\w+)>>')[0]
for serial in brands[i * 2 + 1].xpath('div'):
item = parse_serial_item(serial, bid, vendor)
yield item
url = 'http://car.bitauto.com%s' % serial.xpath('div/div/a/@href')[0].extract()
request = scrapy.Request(url, callback=self.parse_model, dont_filter=True)
request.meta['sid'] = item['id']
yield request
else:
for serial in brands.xpath('div'):
item = parse_serial_item(serial, bid, '')
yield item
url = 'http://car.bitauto.com%s' % serial.xpath('div/div/a/@href')[0].extract()
request = scrapy.Request(url, callback=self.parse_model, dont_filter=True)
request.meta['sid'] = item['id']
yield request
# 解析車型
def parse_model(self, response):
print '==> %s' % response.url
sid = response.meta['sid']
# 在售車型
classify = ''
for tr in response.xpath('//*[@id="compare_sale"]/tbody/*'):
tit = tr.xpath('@class')
if len(tit) == 0:
item = ModelItem()
item['id'] = tr.xpath('td[1]/a[1]/@href')[0].re(r'/m(\d+)/')[0]
item['sid'] = sid
item['name'] = tr.xpath('td[1]/a[1]/text()')[0].extract()
item['classify'] = classify
item['sell'] = '1'
yield item
else:
ths = tr.xpath('th[1]/text()')
th = ths[1].extract().rstrip() if len(ths) > 1 else ths[0].extract().rstrip()
strong = tr.xpath('th[1]/strong/text()')[0].extract()
classify = strong + (('/' + th) if len(th) > 0 else '')
# 停售車型
years = response.xpath('//*[@id="carlist_nosaleyear"]/a/@id').extract()
for year in years:
url = 'http://car.bitauto.com/AjaxNew/GetNoSaleSerailListByYear.ashx?csID=%s&year=%s' % (sid, year)
request = scrapy.Request(url=url, callback=self.parse_model_selled, dont_filter=True)
request.meta['sid'] = sid
yield request
@staticmethod
def parse_model_selled(response):
print '==> %s' % response.url
sid = response.meta['sid']
try:
datas = json.loads(response.body_as_unicode())
for data in datas:
classify = data['Engine_Exhaust'] + '/' + data['MaxPower'] + ' ' + data['InhaleType']
for car in data['carList']:
item = ModelItem()
item['id'] = car['CarID']
item['sid'] = sid
item['name'] = car['YearType'] + ' ' + car['Name']
item['classify'] = classify
item['sell'] = '0'
yield item
except ValueError:
print 'model parse error,serial_id[%s].' % sid
pass
