中間件位于引擎與下載器凿试、引擎與spider之間剪撬，是處理scrapy中兩個重要對象Request炫加、Response及數據數據對象Item的重要的擴展鸿捧。

image

那么中間件分類兩類就不難理解了涛菠，其中一類在引擎與下載器之間我們可以稱之為下載中間件莉御、另一個在引擎與spider之間我們可以稱之為爬蟲中間件；下載中間件和spider中間件都對Request碗暗、Response請求處理颈将，根據位置不同，他們主要負責的職能也不同言疗。

spider中間件（主職過濾）對Request晴圾、Response的主要作用在過濾，可以對特定路徑的URL請求丟棄噪奄、對特定頁面響應過濾死姚、同時對一些不含有指定信息的item過濾，當然pipeline也能實現(xiàn)item的過濾勤篮。

下載中間件（主職加工）主要作用是加工都毒，如給Request添加代理、添加UA碰缔、添加cookie账劲，對Response返回數據編碼解碼、壓縮解壓縮金抡、格式化等預處理瀑焦。

下面我們將從這兩個中間件出發(fā)來詳細講解其中奧妙。

spider中間件

在創(chuàng)建scrapy項目后會自動在middlewares.py文件下生成一個spider中間件和下載中間件模板梗肝，查看代碼：

class ProxyExampleSpiderMiddleware(object):
    # Not all methods need to be defined. If a method is not defined,
    # scrapy acts as if the spider middleware does not modify the
    # passed objects.

    @classmethod
    def from_crawler(cls, crawler):
        # This method is used by Scrapy to create your spiders.
        s = cls()
        crawler.signals.connect(s.spider_opened, signal=signals.spider_opened)
        return s

    def process_spider_input(self, response, spider):
        # Called for each response that goes through the spider
        # middleware and into the spider.

        # Should return None or raise an exception.
        return None

    def process_spider_output(self, response, result, spider):
        # Called with the results returned from the Spider, after
        # it has processed the response.

        # Must return an iterable of Request, dict or Item objects.
        for i in result:
            yield i

    def process_spider_exception(self, response, exception, spider):
        # Called when a spider or process_spider_input() method
        # (from other spider middleware) raises an exception.

        # Should return either None or an iterable of Response, dict
        # or Item objects.
        pass

    def process_start_requests(self, start_requests, spider):
        # Called with the start requests of the spider, and works
        # similarly to the process_spider_output() method, except
        # that it doesn’t have a response associated.

        # Must return only requests (not items).
        for r in start_requests:
            yield r

    def spider_opened(self, spider):
        spider.logger.info('Spider opened: %s' % spider.name)

• process_spider_input(response, spider)
  當Response傳遞給spider的解析函數之前榛瓮，該函數執(zhí)行，返回結果為None或異常
• process_spider_output(response, result, spider)
  當解析函數完成對Response處理后巫击，該函數執(zhí)行禀晓，接受被解析的Response響應及其對應解析出來的迭代對象result（result可以使yield Request或者yield Item）
• process_spider_exception(response, exception, spider)
  當spider中間件拋出異常及spider解析函數出現(xiàn)異常，這個方法被調用坝锰，返回None或可迭代對象的Request粹懒、dict、Item顷级，如果返回None將繼續(xù)被其他spider中間件的異常處理
• from_crawler(cls, crawler)
• 讀取配置文件中的參數進行中間件配置

看一個scrapy內置的spider中間件源碼

scrapy.spidermiddlewares.httperror.HttpErrorMiddleware

過濾出所有失敗(錯誤)的HTTP response崎淳，因此spider不需要處理這些reques

class HttpErrorMiddleware(object):

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.settings)

    def __init__(self, settings):
        self.handle_httpstatus_all = settings.getbool('HTTPERROR_ALLOW_ALL')
        self.handle_httpstatus_list = settings.getlist('HTTPERROR_ALLOWED_CODES')

    def process_spider_input(self, response, spider):
        if 200 <= response.status < 300:  # common case
            return
        meta = response.meta
        if 'handle_httpstatus_all' in meta:
            return
        if 'handle_httpstatus_list' in meta:
            allowed_statuses = meta['handle_httpstatus_list']
        elif self.handle_httpstatus_all:
            return
        else:
            allowed_statuses = getattr(spider, 'handle_httpstatus_list', self.handle_httpstatus_list)
        if response.status in allowed_statuses:
            return
        raise HttpError(response, 'Ignoring non-200 response')

    def process_spider_exception(self, response, exception, spider):
        if isinstance(exception, HttpError):
            spider.crawler.stats.inc_value('httperror/response_ignored_count')
            spider.crawler.stats.inc_value(
                'httperror/response_ignored_status_count/%s' % response.status
            )
            logger.info(
                "Ignoring response %(response)r: HTTP status code is not handled or not allowed",
                {'response': response}, extra={'spider': spider},
            )
            return []

開啟自定義spider中間件方式，在配置文件setting.py中添加命名為SPIDER_MIDDLEWARES的字典，其中key為下載器路徑拣凹，value為優(yōu)先級森爽，數字越小越靠近引擎，process_spider_input()優(yōu)先處理嚣镜，數字越大越靠近spider爬迟，process_spider_output()優(yōu)先處理

內置spider 中間件

SPIDER_MIDDLEWARES_BASE = {
    # Engine side
    'scrapy.spidermiddlewares.httperror.HttpErrorMiddleware': 50,
    'scrapy.spidermiddlewares.offsite.OffsiteMiddleware': 500,
    'scrapy.spidermiddlewares.referer.RefererMiddleware': 700,
    'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware': 800,
    'scrapy.spidermiddlewares.depth.DepthMiddleware': 900,
    # Spider side
}

下載中間件

下載中間件的使用頻率遠高于spider中間件，他是我們設置反爬蟲措施的主要戰(zhàn)場菊匿，同樣在創(chuàng)建scrapy項目的同時會生成一個下載中間件的模板付呕，如下：

class ProxyExampleDownloaderMiddleware(object):
    # Not all methods need to be defined. If a method is not defined,
    # scrapy acts as if the downloader middleware does not modify the
    # passed objects.

    @classmethod
    def from_crawler(cls, crawler):
        # This method is used by Scrapy to create your spiders.
        s = cls()
        crawler.signals.connect(s.spider_opened, signal=signals.spider_opened)
        return s

    def process_request(self, request, spider):
        # Called for each request that goes through the downloader
        # middleware.

        # Must either:
        # - return None: continue processing this request
        # - or return a Response object
        # - or return a Request object
        # - or raise IgnoreRequest: process_exception() methods of
        #   installed downloader middleware will be called
        return None

    def process_response(self, request, response, spider):
        # Called with the response returned from the downloader.

        # Must either;
        # - return a Response object
        # - return a Request object
        # - or raise IgnoreRequest
        return response

    def process_exception(self, request, exception, spider):
        # Called when a download handler or a process_request()
        # (from other downloader middleware) raises an exception.

        # Must either:
        # - return None: continue processing this exception
        # - return a Response object: stops process_exception() chain
        # - return a Request object: stops process_exception() chain
        pass

    def spider_opened(self, spider):
        spider.logger.info('Spider opened: %s' % spider.name)

from_crawler(cls, crawler)
這個類方法通常是訪問settings和signals的入口函數

process_request(self, request, spider)

當引擎將請求發(fā)送給下載器之前調用，用于對request請求加工跌捆，返回值為None徽职、Request、Response佩厚、IgnoreRequest異常姆钉。

返回為None時其他的下載中間件的process_request方法執(zhí)行，直到內置的一個下載器方法返回Request對象為止

當返回Request對象后抄瓦，其他的process_request方法不再執(zhí)行潮瓶，而是將翻譯的Request發(fā)往調度器排隊等待新一輪的process_request依次執(zhí)行。

當返回Response钙姊，這時更加簡單了毯辅，相當于告訴引擎我已經得到結果了，不需要其他process_request執(zhí)行煞额，接下來依次執(zhí)行process_resposne

• process_response(request, response, spider)
  當process_request返回resposne對象后思恐，該函數的流水線登場對resposne處理
  若返回Response對象，它會被下個中間件中的process_response()處理
  若返回Request對象膊毁，中間鏈停止胀莹，然后返回的Request會被重新調度下載
  拋出IgnoreRequest，回調函數 Request.errback將會被調用處理媚媒，若沒處理，將會忽略
• 當然如果所有的中間件都沒意見那么將交給引擎轉交給spider的解析函數
• process_exception(request, exception, spider)
  當下載處理模塊或process_request()拋出一個異常（包括IgnoreRequest異常）時涩僻，該方法被調用通常返回None,它會一直處理異常
• 總結：下載中間件缭召，是兩條流水線，一條是process_request逆日、一條是process_requests嵌巷，這兩條流水線的最末端是內置的方法，當一個Request請求發(fā)來室抽，將被篇process_request流水線上的工位逐個檢查處理搪哪，有的說缺少材料，補一份清單流水線重新檢查坪圾，有的說這個已經是response成品了不用檢查了晓折，并貼上response標簽惑朦，有的說這個是殘次品按照殘次品處理，當然如果所有工位都沒有意見那么最后一個工位的員工將按照最后流程貼上一個response成品標簽漓概，從而進入Process_response流水線漾月。
• 在process_resposne流水線上同上處理，如果有的工位說還是Request半成品胃珍，那么返回process_request流水線在加工梁肿；如果有的說確實是response產品，那么給下一個工位逐一確定觅彰，直到最后一個內置工位確認確實是response產品吩蔑，那么交給引擎。

中間件就是這樣的流水線思想填抬，我們自定義的中間件只不過是流水線上的一個工位烛芬，當然你可以給這個工位分配順序和權限，可以直接給產品貼標簽痴奏，貼上不同的標簽蛀骇，將進入不同的流水線再處理。

常用自定義中間件源碼：

user-agent中間件

from faker import Faker

class UserAgent_Middleware():

    def process_request(self, request, spider):
        f = Faker()
        agent = f.firefox()
        request.headers['User-Agent'] = agent

代理ip中間件

class Proxy_Middleware():

    def process_request(self, request, spider):

        try:
            xdaili_url = spider.settings.get('XDAILI_URL')

            r = requests.get(xdaili_url)
            proxy_ip_port = r.text
            request.meta['proxy'] = 'https://' + proxy_ip_port
        except requests.exceptions.RequestException:
            print('獲取訊代理ip失敹敛稹擅憔！')
            spider.logger.error('獲取訊代理ip失敗檐晕！')

scrapy中對接selenium

from scrapy.http import HtmlResponse
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from gp.configs import *

class ChromeDownloaderMiddleware(object):

    def __init__(self):
        options = webdriver.ChromeOptions()
        options.add_argument('--headless')  # 設置無界面
        if CHROME_PATH:
            options.binary_location = CHROME_PATH
        if CHROME_DRIVER_PATH:
            self.driver = webdriver.Chrome(chrome_options=options, executable_path=CHROME_DRIVER_PATH)  # 初始化Chrome驅動
        else:
            self.driver = webdriver.Chrome(chrome_options=options)  # 初始化Chrome驅動

    def __del__(self):
        self.driver.close()

    def process_request(self, request, spider):
        try:
            print('Chrome driver begin...')
            self.driver.get(request.url)  # 獲取網頁鏈接內容
            return HtmlResponse(url=request.url, body=self.driver.page_source, request=request, encoding='utf-8',
                                status=200)  # 返回HTML數據
        except TimeoutException:
            return HtmlResponse(url=request.url, request=request, encoding='utf-8', status=500)
        finally:
            print('Chrome driver end...')

scrapy對接cookie中間件

class WeiBoMiddleWare(object):
    def __init__(self, cookies_pool_url):
        self.logging = logging.getLogger("WeiBoMiddleWare")
        self.cookies_pool_url = cookies_pool_url

    def get_random_cookies(self):
        try:
            response = requests.get(self.cookies_pool_url)
        except Exception as e:
            self.logging.info('Get Cookies failed: {}'.format(e))
        else:
            # 在中間件中暑诸，設置請求頭攜帶的Cookies值，必須是一個字典辟灰，不能直接設置字符串个榕。
            cookies = json.loads(response.text)
            self.logging.info('Get Cookies success: {}'.format(response.text))
            return cookies

    @classmethod
    def from_settings(cls, settings):
        obj = cls(
            cookies_pool_url=settings['WEIBO_COOKIES_URL']
        )
        return obj

    def process_request(self, request, spider):
        request.cookies = self.get_random_cookies()
        return None

上面源碼來自（https://blog.csdn.net/BF02jgtRS00XKtCx/article/details/82141627、https://blog.csdn.net/qq_42336549/article/details/80991814）用以示例芥喇。

使用方法西采，在setting.py文件中配置DOWNLOADERMIDDLEWARES 字典，鍵為中間件路徑继控，值是優(yōu)先級械馆，數字越小，越靠近引擎武通，數字越大越靠近下載器霹崎，所以數字越小的，process_request()優(yōu)先處理冶忱；數字越大的尾菇，process_response()優(yōu)先處理.

內置下載中間件如下：

image

最后再附上一個內置的下載中間件源碼：

默認開啟，Scrapy將記錄所有在request(Cookie 請求頭)發(fā)送的cookies及response接收到的cookies(Set-Cookie 接收頭)。

class CookiesMiddleware(object):
    """This middleware enables working with sites that need cookies"""

    def __init__(self, debug=False):
        self.jars = defaultdict(CookieJar)
        self.debug = debug

    @classmethod
    def from_crawler(cls, crawler):
        if not crawler.settings.getbool('COOKIES_ENABLED'):
            raise NotConfigured
        return cls(crawler.settings.getbool('COOKIES_DEBUG'))

    def process_request(self, request, spider):
        if request.meta.get('dont_merge_cookies', False):
            return

        cookiejarkey = request.meta.get("cookiejar")
        jar = self.jars[cookiejarkey]
        cookies = self._get_request_cookies(jar, request)
        for cookie in cookies:
            jar.set_cookie_if_ok(cookie, request)

        # set Cookie header
        request.headers.pop('Cookie', None)
        jar.add_cookie_header(request)
        self._debug_cookie(request, spider)

    def process_response(self, request, response, spider):
        if request.meta.get('dont_merge_cookies', False):
            return response

        # extract cookies from Set-Cookie and drop invalid/expired cookies
        cookiejarkey = request.meta.get("cookiejar")
        jar = self.jars[cookiejarkey]
        jar.extract_cookies(response, request)
        self._debug_set_cookie(response, spider)

        return response

    def _debug_cookie(self, request, spider):
        if self.debug:
            cl = [to_native_str(c, errors='replace')
                  for c in request.headers.getlist('Cookie')]
            if cl:
                cookies = "\n".join("Cookie: {}\n".format(c) for c in cl)
                msg = "Sending cookies to: {}\n{}".format(request, cookies)
                logger.debug(msg, extra={'spider': spider})

    def _debug_set_cookie(self, response, spider):
        if self.debug:
            cl = [to_native_str(c, errors='replace')
                  for c in response.headers.getlist('Set-Cookie')]
            if cl:
                cookies = "\n".join("Set-Cookie: {}\n".format(c) for c in cl)
                msg = "Received cookies from: {}\n{}".format(response, cookies)
                logger.debug(msg, extra={'spider': spider})

    def _format_cookie(self, cookie):
        # build cookie string
        cookie_str = '%s=%s' % (cookie['name'], cookie['value'])

        if cookie.get('path', None):
            cookie_str += '; Path=%s' % cookie['path']
        if cookie.get('domain', None):
            cookie_str += '; Domain=%s' % cookie['domain']

        return cookie_str

    def _get_request_cookies(self, jar, request):
        if isinstance(request.cookies, dict):
            cookie_list = [{'name': k, 'value': v} for k, v in \
                    six.iteritems(request.cookies)]
        else:
            cookie_list = request.cookies

        cookies = [self._format_cookie(x) for x in cookie_list]
        headers = {'Set-Cookie': cookies}
        response = Response(request.url, headers=headers)

        return jar.make_cookies(response, request)

·END·

Python之戰(zhàn)

深入Python的心

image

微信號：djdldy

image

轉載是一種動力 分享是一種美德