大師兄的Python學(xué)習(xí)筆記(三十）: 爬蟲（十一）
大師兄的Python學(xué)習(xí)筆記(三十二）: 爬蟲（十三）

十一、Scrapy框架

10. 使用Selenium抓舉動(dòng)態(tài)頁面

• 可以使用Scrapy對(duì)接Selenium模擬瀏覽器抓取動(dòng)態(tài)頁面蓖柔。
• 以爬取華為商城搜索頁面為例辰企。

1) 定義Item

• 定義要爬取的Field和數(shù)據(jù)庫collection, 這里我們只爬取商品名和價(jià)格。

# items.py
>>>import scrapy

>>>class HuaweiMainPageItem(scrapy.Item):
>>>    collection = 'items'
>>>    title = scrapy.Field()
>>>    price = scrapy.Field()

2) 初步實(shí)現(xiàn)爬蟲

• 這里爬蟲只打開搜索頁面况鸣。

# huawei.py
>>import scrapy
>>from huawei_main_page.items import HuaweiMainPageItem

>>class HuaweiSpider(scrapy.Spider):
>>    name = 'huawei'
>>    allowed_domains = ['www.vmall.com']
>>    url = 'https://www.vmall.com/'

>>    def start_requests(self):
>>        yield scrapy.Request(url=self.url, callback=self.parse)

• 順便在settings.py中配置要搜索庫

# settings.py
KEYWORD = ['筆記本']

3) 通過下載器中間件實(shí)現(xiàn)對(duì)接Selenium

• 首先配置中間件和Selenium參數(shù)

# settings.py
... ...
SELENIUM_TIMEOUT = 20
MAX_PAGE=10
... ...
DOWNLOADER_MIDDLEWARES = {
  'huawei_main_page.middlewares.HuaweiMainPageDownloaderMiddleware': 543,
}

• 編寫中間件牢贸，用selenium實(shí)現(xiàn)輸入搜索內(nèi)容，之后點(diǎn)擊跳轉(zhuǎn)頁面镐捧，并將跳轉(zhuǎn)的頁面發(fā)送給下載器潜索。
• 我使用的是火狐瀏覽器。

# middlewares.py
>>>from selenium import webdriver
>>>from selenium.webdriver.firefox.options import Options
>>>from selenium.common.exceptions import TimeoutException
>>>from selenium.webdriver.common.by import By
>>>from selenium.webdriver.support.ui import WebDriverWait
>>>from selenium.webdriver.support import expected_conditions as EC
>>>from scrapy.http import HtmlResponse
>>>from logging import getLogger

>>>class HuaweiMainPageDownloaderMiddleware(object):

>>>    def __init__(self, kw,timeout=None):
>>>        self.logger = getLogger(__name__)
>>>        self.kw = kw
>>>        self.timeout = timeout
>>>        self.options = Options()
>>>        self.options.add_argument('-headless')
>>>        self.browser = webdriver.Firefox(firefox_options=self.options)
>>>        self.browser.set_window_size(1980, 1200)
>>>        self.browser.set_page_load_timeout(self.timeout)
>>>        self.wait = WebDriverWait(self.browser, self.timeout)

>>>    @classmethod
>>>    def from_crawler(cls, crawler):
>>>        return cls(timeout=crawler.settings.get('SELENIUM_TIMEOUT'),kw=crawler.settings.get('KEYWORD'))

>>>    def process_request(self, request, spider):
>>>        self.logger.debug('PhantomJS is Starting')

>>>        try:
>>>            page = self.browser.get(request.url)
>>>            search = self.wait.until(
>>>                EC.presence_of_element_located((By.CSS_SELECTOR,'.text'))
>>>            )
>>>            search.clear()
>>>            search.send_keys(self.kw)

>>>            button = self.wait.until(
>>>                EC.presence_of_element_located((By.CSS_SELECTOR,'.button'))
>>>            )
>>>            button.click()

>>>            self.browser.switch_to.window(self.browser.window_handles[-1])
>>>            self.wait.until(
>>>                EC.presence_of_all_elements_located((By.CSS_SELECTOR,'.pro-panels'))
>>>            )

>>>            return HtmlResponse(url=self.browser.current_url,body=self.browser.page_source,request=request,encoding='utf-8',status=200)

>>>        except TimeoutException:
>>>            return HtmlResponse(url=request.url,status=500,request=request)

>>>    def spider_opened(self, spider):
>>>        spider.logger.info('Spider opened: %s' % spider.name)

4) 繼續(xù)在爬蟲中編寫解析方法

• 解析下載器返回的頁面數(shù)據(jù)懂酱。

# huawei.py
>>>import scrapy
>>>from huawei_main_page.items import HuaweiMainPageItem

>>>class HuaweiSpider(scrapy.Spider):
>>>    name = 'huawei'
>>>    allowed_domains = ['www.vmall.com']
>>>    url = 'https://www.vmall.com/'

>>>    def start_requests(self):
>>>        yield scrapy.Request(url=self.url, callback=self.parse)

>>>    def parse(self, response):
>>>        products = response.css('.pro-panels')
>>>        for product in products:
>>>            item = HuaweiMainPageItem()
>>>            item['title'] = product.css('.p-name::text').extract_first()
>>>            item['price'] = product.css('.p-price b::text').extract_first()
>>>            yield item

5) 通過Pipeline將爬取結(jié)果保存到數(shù)據(jù)庫

• 首先需要在settings.py中配置數(shù)據(jù)庫信息竹习。

# settings.py
>>>ITEM_PIPELINES = {
>>>   'huawei_main_page.pipelines.HuaweiMainPagePipeline': 300,
>>>}

>>>MONGO_URI='localhost'
>>>MONGO_DB='project_huawei'

• 之后在pipeline.py中實(shí)現(xiàn)存儲(chǔ)。

>>>import pymongo

>>>class HuaweiMainPagePipeline(object):
>>>    def __init__(self,mongo_uri,mongo_db):
>>>        self.mongo_uri = mongo_uri
>>>        self.mongo_db = mongo_db

>>>    @classmethod
>>>    def from_crawler(cls,crawler):
>>>        return >>>>cls(mongo_uri=crawler.settings.get('MONGO_URI'),mongo_db=crawler.settings.get('MONGO_DB'))

>>>    def open_spider(self,spider):
>>>        self.client = pymongo.MongoClient(self.mongo_uri)
>>>        self.db = self.client[self.mongo_db]

>>>    def process_item(self, item, spider):
>>>        self.db[item.collection].insert(dict(item))
>>>        return item

>>>    def close_spider(self,spider):
>>>        self.client.close()

參考資料

• https://blog.csdn.net/u010138758/article/details/80152151 J-Ombudsman
• https://www.cnblogs.com/zhuluqing/p/8832205.html moisiet
• https://www.runoob.com 菜鳥教程
• http://www.tulingxueyuan.com/ 北京圖靈學(xué)院
• http://www.imooc.com/article/19184?block_id=tuijian_wz#child_5_1 兩點(diǎn)水
• https://blog.csdn.net/weixin_44213550/article/details/91346411 python老菜鳥
• https://realpython.com/python-string-formatting/ Dan Bader
• https://www.liaoxuefeng.com/ 廖雪峰
• https://blog.csdn.net/Gnewocean/article/details/85319590 新海說
• https://www.cnblogs.com/Nicholas0707/p/9021672.html Nicholas
• https://www.cnblogs.com/dalaoban/p/9331113.html 超天大圣
• https://blog.csdn.net/zhubao124/article/details/81662775 zhubao124
• https://blog.csdn.net/z59d8m6e40/article/details/72871485 z59d8m6e40
• http://www.reibang.com/p/2b04f5eb5785 MR_ChanHwang
• 《Python學(xué)習(xí)手冊(cè)》Mark Lutz
• 《Python編程 從入門到實(shí)踐》Eric Matthes
• 《Python3網(wǎng)絡(luò)爬蟲開發(fā)實(shí)戰(zhàn)》崔慶才

本文作者：大師兄(superkmi)