上一次寫了scrapy-redis分布式爬蟲的環(huán)境搭建寓涨，現(xiàn)在以毒舌電影社區(qū)為例子編寫毒舌電影社區(qū)的分布式爬蟲例子。如果對于scrapy-redis的環(huán)境搭建不熟悉氯檐，可以直接參考 scrapy-redisf實(shí)現(xiàn)分布式爬蟲戒良；分析毒舌電影社區(qū)需要Fiddler進(jìn)行抓包，這部分可以直接參考 fiddler抓取摩拜單車數(shù)據(jù)包教程冠摄。下面直接給出電影評論接口地址以及參數(shù)糯崎。
- 1.通過Fiddler抓包分析，可以得到電影評論的接口河泳，以及傳遞的參數(shù)如下沃呢，其中count表示每次調(diào)用接口返回的點(diǎn)評條數(shù)，startIndex就是下一條的起始index拆挥，userid是用戶id(如果為0薄霜，代表用戶沒有登錄)
接口地址：
https://dswxapp.dushemovie.com/dsmovieapi/ssl/daily_recmd/list_daily_recmd_dynamic/3
傳遞參數(shù)：
{"count":"20","startIndex":"0","userId":"0"}
-
毒舌電影社區(qū)基本架構(gòu)圖，數(shù)據(jù)庫Redis維持公共的requestUrl隊(duì)列纸兔，實(shí)現(xiàn)去重和保存爬取到的item惰瓜，MongoDB保存所有的數(shù)據(jù)。Master主機(jī)首先提取url到redis汉矿，slave從中提取url崎坊，并且把爬取到url保存會(huì)redis。Master同樣提取url到redis洲拇，并且從中獲取url進(jìn)行爬取奈揍。
scrapy-redis架構(gòu)圖
-
- 文件如下：
文件分布
- 4.爬取邏輯spiders源碼,spider直接繼承scrapy-reids的RedisSpider痹届，保存在Redis數(shù)據(jù)庫的DuzhemovieSpider的starturl中：
# -*- coding: utf-8 -*-
import json
from scrapy import Request
from time import sleep
import logging
from scrapy_redis.spiders import RedisSpider
class DuzhemovieSpider(RedisSpider):
name = "duzhemovie"
# allowed_domains = ["dushemovie.com"]
# start_urls = ['http://dushemovie.com/']
redis_key:"DuzhemovieSpider:start_urls"
url="https://dswxapp.dushemovie.com/dsmovieapi/ssl/daily_recmd/list_daily_recmd_dynamic/3"
postBody={"count":"20","startIndex":"0","userId":"0"}
def start_requests(self):
yield Request(url=self.url,method="POST",body=str(self.postBody),callback=self.parse)
def parse(self, response):
jsonData=json.loads(response.body.decode('UTF-8'))
yield jsonData
while True:
sleep(1)
self.postBody["startIndex"]=str(int(self.postBody['startIndex'])+20)
yield Request(url=self.url,method="POST",body=str(self.postBody),callback=self.parse)
- 5.pipelines保存數(shù)據(jù)item在遠(yuǎn)程阿里云自建數(shù)據(jù)庫MongoDB上，因?yàn)閿?shù)據(jù)都是json格式打月，所以item模塊沒有定義字段值队腐，直接把json格式數(shù)據(jù)寫進(jìn)MongoDB數(shù)據(jù)庫里面。源碼如下：
# -*- coding: utf-8 -*-
# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: http://doc.scrapy.org/en/latest/topics/item-pipeline.html
import pymongo
from scrapy.exceptions import DropItem
class MongoPipeline(object):
collection_name="users"
def __init__(self,mongo_uri,mongo_db,mongo_user,mongo_pass):
self.mongo_uri=mongo_uri
self.mongo_db=mongo_db
self.mongo_user=mongo_user
self.mongo_pass=mongo_pass
@classmethod
def from_crawler(cls,crawler):
return cls(mongo_uri=crawler.settings.get('MONGO_URI'),mongo_db=crawler.settings.get('MONGO_DATABASE'),mongo_user=crawler.settings.get("MONGO_USER"),mongo_pass=crawler.settings.get("MONGO_PASS"))
def open_spider(self, spider):
self.client = pymongo.MongoClient(self.mongo_uri)
self.db = self.client[self.mongo_db]
self.db.authenticate(self.mongo_user,self.mongo_pass)
def close_spider(self, spider):
self.client.close()
def process_item(self, item, spider):
# self.db[self.collection_name].update({'url_token': item['url_token']}, {'$set': dict(item)}, True)
# return item
if item["dynamicDataList"]==None:
raise DropItem("Data is None")
else:
self.db[self.collection_name].insert(dict(item))
return item
- 6.項(xiàng)目設(shè)置模塊奏篙，主要設(shè)置MongoDB的地址柴淘，用戶，密碼以及Redis數(shù)據(jù)庫的用戶名與密碼秘通，以及常見的一些scrapy控制爬取速率和item为严，middleWare順序。
lines (94 sloc) 4 KB
# -*- coding: utf-8 -*-
# Scrapy settings for dushe project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
# http://doc.scrapy.org/en/latest/topics/settings.html
# http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html
# http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html
BOT_NAME = 'dushe'
SPIDER_MODULES = ['dushe.spiders']
NEWSPIDER_MODULE = 'dushe.spiders'
# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'dushe (+http://www.yourdomain.com)'
# Obey robots.txt rules
ROBOTSTXT_OBEY = False
# Configure maximum concurrent requests performed by Scrapy (default: 16)
#CONCURRENT_REQUESTS = 32
# Configure a delay for requests for the same website (default: 0)
# See http://scrapy.readthedocs.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
#DOWNLOAD_DELAY = 3
# The download delay setting will honor only one of:
#CONCURRENT_REQUESTS_PER_DOMAIN = 16
#CONCURRENT_REQUESTS_PER_IP = 16
# Disable cookies (enabled by default)
#COOKIES_ENABLED = False
# Disable Telnet Console (enabled by default)
#TELNETCONSOLE_ENABLED = False
# Override the default request headers:
DEFAULT_REQUEST_HEADERS = {
'Host': 'dswxapp.dushemovie.com',
'Content-Type': 'application/json',
'Accept-Language':' zh-cn',
'Accept-Encoding': 'gzip, deflate',
'Connection':' keep-alive',
'Accept': '*/*',
'User-Agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 10_0_1 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) Mobile/14A403 MicroMessenger/6.5.16 NetType/WIFI Language/zh_CN',
'Referer': 'https://servicewechat.com/wxae1df0a33ef19e00/18/page-frame.html',
'acw_tc':'AQAAAGEFNWv+5AsARUVJ3wWxzmZzc+SK; Path=/; HttpOnly',
'Expires': 'Thu, 01 Jan 1970 00:00:00 GMT',
'di': {"uid":0,"sysType":1,"versionCode":1,"channelCode":"WeChat Small App","lang":"zh_CN","local":"CN","sign":"null"}
}
# Enable or disable spider middlewares
# See http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html
#SPIDER_MIDDLEWARES = {
# 'dushe.middlewares.DusheSpiderMiddleware': 543,
#}
# Enable or disable downloader middlewares
# See http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html
#DOWNLOADER_MIDDLEWARES = {
# 'dushe.middlewares.MyCustomDownloaderMiddleware': 543,
#}
# Enable or disable extensions
# See http://scrapy.readthedocs.org/en/latest/topics/extensions.html
#EXTENSIONS = {
# 'scrapy.extensions.telnet.TelnetConsole': None,
#}
# Configure item pipelines
# See http://scrapy.readthedocs.org/en/latest/topics/item-pipeline.html
ITEM_PIPELINES = {
'dushe.pipelines.MongoPipeline': 300,
'scrapy_redis.pipelines.RedisPipeline': 301
}
# Enable and configure the AutoThrottle extension (disabled by default)
# See http://doc.scrapy.org/en/latest/topics/autothrottle.html
#AUTOTHROTTLE_ENABLED = True
# The initial download delay
#AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
#AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
#AUTOTHROTTLE_DEBUG = False
# Enable and configure HTTP caching (disabled by default)
# See http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
#HTTPCACHE_ENABLED = True
#HTTPCACHE_EXPIRATION_SECS = 0
#HTTPCACHE_DIR = 'httpcache'
#HTTPCACHE_IGNORE_HTTP_CODES = []
#HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'
HTTPERROR_ALLOWED_CODES = [404]
DOWNLOAD_DELAY = 3
MONGO_URI="remoteAddress"
MONGO_DATABASE='databaseName'
MONGO_USER="username"
MONGO_PASS="password"
SCHEDULER = "scrapy_redis.scheduler.Scheduler"
DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter"
REDIS_URL = 'redis://username:password@remoteAddress:port'
SCHEDULER_IDLE_BEFORE_CLOSE = 10
SCHEDULER_PERSIST=False
-
爬取的數(shù)據(jù)展示如下：
數(shù)據(jù)庫展示
