pyspider是一個(gè)爬蟲(chóng)架構(gòu)的開(kāi)源化實(shí)現(xiàn)

主要的功能需求是：

• 抓取、更新調(diào)度多站點(diǎn)的特定的頁(yè)面
• 需要對(duì)頁(yè)面進(jìn)行結(jié)構(gòu)化信息提取
• 靈活可擴(kuò)展香缺，穩(wěn)定可監(jiān)控 而這也是絕大多數(shù)python爬蟲(chóng)的需求 —— 定向抓取鹉动，結(jié)構(gòu)化化解析。但是面對(duì)結(jié)構(gòu)迥異的各種網(wǎng)站，單一的抓取模式并不一定能滿(mǎn)足懈玻，靈活的抓取控制是必須的。為了達(dá)到這個(gè)目的乾颁，單純的配置文件往往不夠靈活涂乌，于是，通過(guò)腳本去控制抓取是我最后的選擇英岭。 而去重調(diào)度湾盒，隊(duì)列，抓取诅妹，異常處理罚勾，監(jiān)控等功能作為框架，提供給抓取腳本吭狡，并保證靈活性尖殃。最后加上web的編輯調(diào)試環(huán)境，以及web任務(wù)監(jiān)控划煮，即成為了這套框架分衫。

pyspider的設(shè)計(jì)基礎(chǔ)是：以python腳本驅(qū)動(dòng)的抓取環(huán)模型爬蟲(chóng)

• 通過(guò)python腳本進(jìn)行結(jié)構(gòu)化信息的提取，follow鏈接調(diào)度抓取控制般此，實(shí)現(xiàn)最大的靈活性
• 通過(guò)web化的腳本編寫(xiě)蚪战、調(diào)試環(huán)境。web展現(xiàn)調(diào)度狀態(tài)
• 抓取環(huán)模型成熟穩(wěn)定铐懊，模塊間相互獨(dú)立邀桑，通過(guò)消息隊(duì)列連接，從單進(jìn)程到多機(jī)分布式靈活拓展

安裝pyspider:

添加依賴(lài)

sudo apt-get install python python-dev python-distribute python-pip libcurl4-openssl-dev libxml2-dev libxslt1-dev python-lxml libssl-dev zlib1g-dev

sudo apt-get install phantomjs

pip3 install pyspider

啟動(dòng):

pyspider all

以鏈家為例 ：

#!/usr/bin/env python
# -*- encoding: utf-8 -*-
# Created on 2019-01-09 14:03:20
# Project: lianjia

from pyspider.libs.base_handler import *
import json
import pymongo
import pymysql


class Handler(BaseHandler):
    #pyspider爬蟲(chóng)的主類(lèi)科乎，在這里進(jìn)行爬取壁畸，解析，和存儲(chǔ)數(shù)據(jù)
    
    #crawl_config:在這個(gè)參數(shù)中可以做全局的設(shè)置(UA,Headers,proxy..)
    crawl_config = {
        'User-Agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36',
        #'itags':'v2'
    }
    
    #創(chuàng)建mongodb數(shù)據(jù)庫(kù)連接
    mongo_client = pymongo.MongoClient('127.0.0.1',27017)
    #獲取要湊走的數(shù)據(jù)庫(kù)
    db = mongo_client['lianjia']
    #獲取數(shù)據(jù)庫(kù)下的集合
    col = db['lianjiacol']
    
    
    #創(chuàng)建數(shù)據(jù)庫(kù)連接
    mysql_cli = pymysql.Connect(
        '127.0.0.1','root','ljh1314',
        'lianjia',3306,charset='utf8'
    )
    #創(chuàng)建游標(biāo)
    cursor = mysql_cli.Cursor()

    @every(minutes=24 * 60)
    #定時(shí):每隔一天進(jìn)行重復(fù)請(qǐng)求（重新執(zhí)行on_start）
    def on_start(self):
        #crawl:根據(jù)crawl發(fā)起請(qǐng)求(url,proxy,itag,user-agent,headers,exetime,callback,save,parmas,data,fetch_type,)
        self.crawl('https://bj.lianjia.com/ershoufang/', callback=self.index_page)

    @config(age=10 * 24 * 60 * 60)
    def index_page(self, response):
        
        #response.doc:是一個(gè)pyquery對(duì)象
        #response.etree:返回的是一個(gè)lxml
        #提取每一個(gè)房源的詳情url地址
        hourse_infos = response.doc('ul.sellListContent li.clear.LOGCLICKDATA')
        
        for hourse in hourse_infos.items():
            detail_url = hourse('div.title a').attr.href
            self.crawl(detail_url,callback=self.detail_page)
            
        #提取下一頁(yè)發(fā)起請(qǐng)求
        #data = response.doc('div.page-box.house-lst-page-box').attr.page-data
        data = response.etree.xpath('//div[@class="page-box house-lst-page-box"]/@page-data')[0]
        print(data)
        json_data = json.loads(data)
        #print('nextdata',json_data)
        cur_page = int(json_data['curPage'])
        total_page = int(json_data['totalPage'])
        if cur_page < total_page:
            #發(fā)起下一頁(yè)請(qǐng)求
            next_page = cur_page+1
            next_url = 'https://bj.lianjia.com/ershoufang/pg%s/' % str(next_page)
            self.crawl(next_url,callback=self.index_page)
            
        #next_url = response.doc()
       # for each in response.doc('a[href^="http"]').items():
       #     self.crawl(each.attr.href, callback=self.detail_page)

    @config(priority=2)
    def detail_page(self, response):
        print('二手房詳情獲取成功')
        #獲取二手房詳情的數(shù)據(jù)
        info = {}
        #標(biāo)題(獲取標(biāo)簽的屬性和文本)
        #info['title'] = response.doc('h1.main').attr.title
        info['title'] = response.doc('h1.main').text()
        #其他信息
        info['subTitle'] = response.doc('div.sub').text()
        #關(guān)注人數(shù)
        info['attenNum'] = response.doc('#favCount').text()
        #觀看人數(shù)
        info['cartCount'] = response.doc('#cartCount').text()
        #總價(jià)
        info['totalPrice'] = response.doc('div.price  .total').text()
        #單價(jià)
        info['price'] = response.doc('span.unitPriceValue').text()
        #規(guī)模
        info['model'] = response.doc('div.room .mainInfo').text()
        #樓層
        info['foolr'] = response.doc('div.room .subInfo').text()
        #小區(qū)
        info['communityName'] = response.doc('div.communityName a.info ').text()
        
        #print(info)
        
        return info
    
    def on_result(self,result):
        print('獲取到了結(jié)果',result)
        if result:
            try:
                #self.col.insert(result)
                sql = """
                INSERT INTO lianjiadb(%s)
                VALUES (%s)
                """ % (','.join(result.keys()),','.join(['%s']*len(result)))
                data = list(result.values())
                self.cursor.exectue(sql,data)
                print('數(shù)據(jù)存儲(chǔ)成功')
            except Exception as err:
                print('數(shù)據(jù)插入失敗',err)
        
        #return {
        #    "url": response.url,
        #    "title": response.doc('title').text(),
        #}


        ```