docker 安裝
Mac 安裝
brew cask install docker
ubuntu 安裝
- 安裝
wget -qO- https://get.docker.com/ | sh
- 啟動
sudo service docker start
- 修改倉庫源
- 配置 /etc/docker/daemon.js
{
“registry-mirrors”: [“https://registry.docker-cn.com“]
}
- 重啟
# 機(jī)器重啟
sudo reboot
splash 安裝
sudo docker pull scrapinghub/splash
運(yùn)行 splash
sudo docker run -p 8050:8050 scrapinghub/splash
scrapy 使用 splash
- 配置之前先得安裝
pip install scrapy-splash
- 配置settings.py
# 添加 splash 服務(wù)器地址
SPLASH_URL = 'http://127.0.0.1:8050'
# 配置下載中間件
DOWNLOADER_MIDDLEWARES = {
# 'github.middlewares.SeleniumMiddleware': 543,
'scrapy_splash.SplashCookiesMiddleware': 723,
'scrapy_splash.SplashMiddleware': 725,
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
}
- 使用 splash 提供的請求對象
