一、準(zhǔn)備工作
1.安裝splash
在windows環(huán)境下鼠次，splash可通過docker進(jìn)行安裝撵溃，安裝方法在 這篇文章 中有詳細(xì)講解，在此不再贅述掂咒。
安裝完成后在powershell中運(yùn)行以下命令安裝splash：
#從docker hub下載相關(guān)鏡像文件
docker pull scrapinghub/splash才沧、
使用docker啟動(dòng)服務(wù)命令啟動(dòng)Splash服務(wù)迈喉，在8050、8051温圆、5023端口開啟splash服務(wù)：
#啟動(dòng)splash服務(wù)挨摸，并通過http，https岁歉，telnet提供服務(wù)
#通常一般使用http模式 得运，可以只啟動(dòng)一個(gè)8050就好
#Splash 將運(yùn)行在 0.0.0.0 at ports 8050 (http), 8051 (https) and 5023 (telnet).
docker run -p 5023:5023 -p 8050:8050 -p 8051:8051 scrapinghub/splash
在瀏覽器中輸入10.0.75.1:8050查看服務(wù)啟動(dòng)情況：
QQ截圖20180214144552.png
2.安裝scrapy-splash
使用pip安裝scrapy-splash：
pip install scrapy-splash
完成以上步驟之后就可以在scrapy中使用splash了
二、編寫爬蟲
動(dòng)態(tài)頁面中的部分內(nèi)容是瀏覽器運(yùn)行頁面中的javascript腳本生成的锅移，相較于普通靜態(tài)頁面澈圈，爬取難度更大。以網(wǎng)站 http://quotes.toscrape.com/js/為例帆啃，爬取動(dòng)態(tài)生成的名人名言信息瞬女。
在項(xiàng)目文件settings.py中配置scrapy-splash：
#定義splash服務(wù)器地址
SPLASH_URL = 'http://10.0.75.1:8050/'
#開啟splash的下載中間件
DOWNLOADER_MIDDLEWARES = {
'scrapy_splash.SplashCookiesMiddleware': 723,
'scrapy_splash.SplashMiddleware': 725,
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
}
#設(shè)置去重
DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'
編碼實(shí)現(xiàn)：
# -*- coding: utf-8 -*-
import scrapy
from scrapy_splash import SplashRequest
class QuoteSpider(scrapy.Spider):
name = 'quote'
start_urls = ['http://quotes.toscrape.com/js/']
def start_requests(self):
for url in self.start_urls:
yield SplashRequest(url, args={'images': 0})
def parse(self, response):
quotes = response.xpath('//div[@class="quote"]')
for quote in quotes:
text = quote.xpath('.//*[@class="text"]/text()').extract_first()
author = quote.xpath('.//*[@class="author"]/text()').extract_first()
yield {
'text': text,
'author': author,
}
next_page = response.xpath('//a[contains(text(),"Next")]/@href').extract_first()
if next_page:
absolute_url = response.urljoin(next_page)
yield SplashRequest(absolute_url, args={'images': 0})
執(zhí)行結(jié)果：
2018-02-14 15:17:31 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/js/>
{'text': '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”', 'author': 'Albert Einstein'}
2018-02-14 15:17:31 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/js/>
{'text': '“It is our choices, Harry, that show what we truly are, far more than our abilities.”', 'author': 'J.K. Rowling'}
....
2018-02-14 15:17:38 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/js/page/10/>
{'text': "“A person's a person, no matter how small.”", 'author': 'Dr. Seuss'}
2018-02-14 15:17:38 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/js/page/10/>
{'text': '“... a mind needs books as a sword needs a whetstone, if it is to keep its edge.”', 'author': 'George R.R. Martin'}
結(jié)果顯示我們已成功爬取動(dòng)態(tài)生成的名人名言信息。
