概括

通過(guò)Python爬蟲實(shí)現(xiàn)多線程對(duì)于36氪首頁(yè)氪視頻的爬取攒钳。

實(shí)現(xiàn)

36氪首頁(yè)：https://36kr.com/

36氪首頁(yè)

• 獲取Ajax請(qǐng)求的API

通過(guò)右鍵 > 檢查跃赚，或F12打開(kāi)瀏覽器調(diào)試模式笆搓，選擇Network > XHR，此時(shí)滾動(dòng)鼠標(biāo)滑輪纬傲，直到自動(dòng)刷新出新的文章（或者點(diǎn)擊底部的“瀏覽更多”按鈕）满败，就可以獲取動(dòng)態(tài)數(shù)據(jù)包。

獲取Ajax請(qǐng)求的接口

獲取到的響應(yīng)

• 通過(guò) requests 發(fā)送請(qǐng)求

import json
import jsonpath
import requests
import re
import time
from queue import Queue
from threading import Thread

class Krspider(object):
    def __init__(self):
        # 留下page入口以實(shí)現(xiàn)獲取多頁(yè)數(shù)據(jù)
        self.base_url = 'https://36kr.com/api/search-column/mainsite?per_page=20&page={}'
        self.video_url = 'https://36kr.com/video/{}'  # 拼接氪視頻詳情頁(yè)url
        self.headers = {'User-Agent':"Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko"}
        self.url_queue = Queue()
        self.send_request_queue = Queue()
        self.parse_detail_queue = Queue()
        self.detail_data_queue = Queue()
        self.parse_video_queue = Queue()
        self.count = 0

    def send_request(self):
        while True:
            url = self.url_queue.get()
            str_json = requests.get(url, headers=self.headers).content.decode()
            self.send_request_queue.put(str_json)
            self.url_queue.task_done() 

通過(guò) requests 的 get 方法向目標(biāo) url 發(fā)送 get 請(qǐng)求继低，str_json 即為響應(yīng)的 Json 字符串熬苍。收發(fā)網(wǎng)絡(luò)請(qǐng)求是耗時(shí)的，會(huì)產(chǎn)生阻塞袁翁，使用多線程柴底，也用到隊(duì)列模塊 Queue，把每個(gè)步驟封裝成函數(shù)粱胜，分別用線程去執(zhí)行柄驻，每個(gè)步驟間通過(guò)隊(duì)列相互通信，也對(duì)函數(shù)間解耦焙压。

    def run(self):
        total_page = int(input('輸入要抓取的頁(yè)數(shù)：'))
        for page in range(1, total_page + 1):
            url = self.base_url.format(page)
            self.url_queue.put(url)

通過(guò)遍歷頁(yè)數(shù)鸿脓，得到每一頁(yè)的 url，同時(shí)將各個(gè) url 放入到 url 隊(duì)列 self.url_queue 中涯曲。發(fā)送請(qǐng)求并接受響應(yīng)的方法 send_request 會(huì)不斷的從 url 隊(duì)列中拿出 每一頁(yè)的 url野哭，并發(fā)送請(qǐng)求。將收到的每一頁(yè)返回的Json字符串放入到 self.send_request_queue 隊(duì)列中幻件。

• 解析氪視頻頁(yè)面的url

data_dict = json.loads(data)  

再通過(guò) jsonpath 將所氪視頻文章取出涧衙，返回一個(gè)列表哪工，列表中每個(gè)字典就是每一個(gè)氪視頻文章數(shù)據(jù)。

# 通過(guò) column_id = "18"取出當(dāng)前頁(yè)面所有氪視頻文章的字典
video_news_list = jsonpath.jsonpath(data_dict, '$..items[?(@.column_id=="18")]')  

遍歷 video_news_list 列表绍撞，在每個(gè)氪視頻文章字典中通過(guò)“title”正勒、“id”兩個(gè)鍵取出對(duì)應(yīng)的標(biāo)題和 id，這個(gè) id 用于拼接氪視頻詳情頁(yè)面的 url

for video_dict in video_news_list:
    title = video_dict['title']
    url = self.video_url.format(video_dict['id'])

解析全過(guò)程：

def parse_detail(self):
    while True:
        data = self.send_request_queue.get()
        data_dict = json.loads(data)
        video_news_list = jsonpath.jsonpath(data_dict, '$..items[?(@.column_id=="18")]')
        for video_dict in video_news_list:
            title = video_dict['title']
            url = self.video_url.format(video_dict['id'])
            # 將標(biāo)題和 url 組成的列表放入 self.parse_detail_queue 隊(duì)列中
            self.parse_detail_queue.put([title, url])
        self.send_request_queue.task_done()

• 解析視頻MP4文件的url

• 向詳情頁(yè)發(fā)送請(qǐng)求傻铣，獲取響應(yīng)

def send_detail_request(self):
    while True:
        video_list = self.parse_detail_queue.get()
        data = requests.get(video_list[1], headers=self.headers).content.decode()
        # 將響應(yīng)字符串和標(biāo)題組成的列表放入隊(duì)列
        self.detail_data_queue.put([data, video_list[0]])
        self.parse_detail_queue.task_done()

• 解析詳情頁(yè)響應(yīng)章贞，獲取MP4文件的url

def parse_video_url(self):
    while True:
        list = self.detail_data_queue.get()
        pattern = re.compile('http://video\.chuangkr\.china\.com\.cn/.*vb1152\.mp4?')
        try:
            # 響應(yīng)字符串正則匹配，獲得 MP4文件的 url
            str = pattern.search(list[0]).group()
            video_url = str.split(',')[-1].lstrip('"url_1152":"')
        except AttributeError:
            pass
        else:
            if video_url:
                # 將 MP4 文件 url 和標(biāo)題組成列表放入隊(duì)列
                self.parse_video_queue.put([video_url, list[1]])
            else:
                pass
        self.detail_data_queue.task_done()

響應(yīng)字符串中有多個(gè) MP4 文件的 url非洲，但是清晰度卻不同鸭限，分別以“vb_384.mp4”、“vb_512.mp4”两踏、“vb_1152.mp4”結(jié)尾败京，這里獲取清晰度最高的以“vb_1152.mp4”結(jié)尾的文件 url

str.split(',')[-1]

str.split(',')[-1].lstrip('"url_1152":"')

• 獲取 MP4 文件數(shù)據(jù)并保存

有了 MP4 文件的url，最后一步就是發(fā)送請(qǐng)求獲取響應(yīng)數(shù)據(jù)并保存梦染。

def receive_down_load_video(self):
    while True:
        list = self.parse_video_queue.get()
        video_url = list[0]
        title = list[1]
        print('開(kāi)始下載：[{}]'.format(title))
        start = time.time()
        data = requests.get(video_url, headers=self.headers, stream=True).content
        file_name = title[:10]  # 標(biāo)題前8位作為文件名
        file_path = 'video_36kr/' + file_name + '.mp4'
        with open(file_path, 'wb') as f:
             f.write(data)
        end = time.time()
        print('\n' + '[%s]下載完成赡麦，用時(shí)%.2f秒' % (title, (end - start)))
        self.count += 1
        self.parse_video_queue.task_done()

• run() 方法開(kāi)啟多線程

def run(self):
    total_page = int(input('輸入要抓取的頁(yè)數(shù)：'))
    start = time.time()
    for page in range(1, total_page + 1):
        url = self.base_url.format(page)
        self.url_queue.put(url)

        th_list = []
        for i in range(3):
            send_th = Thread(target=self.send_request)
            th_list.append(send_th)

            parse_th = Thread(target=self.parse_detail)
            th_list.append(parse_th)

            send_detail_th = Thread(target=self.send_detail_request)
            th_list.append(send_detail_th)

            parse_video_th = Thread(target=self.parse_video_url)
            th_list.append(parse_video_th)

            download_th = Thread(target=self.receive_down_load_video)
            th_list.append(download_th)

        for th in th_list:
            th.setDaemon(True)  # 把子線程設(shè)置為守護(hù)線程，主線程結(jié)束帕识，子線程也結(jié)束
            th.start()

        for q in [self.url_queue, self.send_request_queue, self.parse_detail_queue, self.detail_data_queue, self.parse_video_queue]:
            q.join()  # 隊(duì)列計(jì)數(shù)不為0的時(shí)候讓主線程阻塞等待泛粹，隊(duì)列計(jì)數(shù)為0的時(shí)候主線程才會(huì)繼續(xù)往后執(zhí)行
    end = time.time()
    print('>>>全部下載完成，總耗時(shí)%s秒<<<' % (end - start))
    print('共下載視頻個(gè)數(shù)：{}'.format(self.count))

把每個(gè)子線程都設(shè)置為守護(hù)線程肮疗，主線程結(jié)束晶姊，所有子線程結(jié)束。而當(dāng)每一個(gè)任務(wù)隊(duì)列計(jì)數(shù)不為0伪货，即還有任務(wù)沒(méi)有被執(zhí)行時(shí)们衙，主線程阻塞，當(dāng)所有隊(duì)列計(jì)數(shù)都為0碱呼，即所有任務(wù)被執(zhí)行蒙挑，主線程往后執(zhí)行并結(jié)束，所有的子線程也隨之結(jié)束（while True 循環(huán)停止）愚臀。

最后來(lái)嘗試運(yùn)行一下程序脆荷，抓取前5頁(yè)的視頻：

播放一個(gè)視頻：

最后是整個(gè)程序的代碼：

import json
import jsonpath
import requests
import re
import time
from queue import Queue
from threading import Thread

class Krspider(object):
    def __init__(self):
        # 留下page入口以實(shí)現(xiàn)獲取多頁(yè)數(shù)據(jù)
        self.base_url = 'https://36kr.com/api/search-column/mainsite?per_page=20&page={}'
        self.video_url = 'https://36kr.com/video/{}'  # 拼接氪視頻詳情頁(yè)url
        self.headers = {'User-Agent':"Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko"}
        self.url_queue = Queue()
        self.send_request_queue = Queue()
        self.parse_detail_queue = Queue()
        self.detail_data_queue = Queue()
        self.parse_video_queue = Queue()
        self.count = 0

    def send_request(self):
        while True:
            url = self.url_queue.get()
            str_json = requests.get(url, headers=self.headers).content.decode()
            self.send_request_queue.put(str_json)
            self.url_queue.task_done()

    def parse_detail(self):
        while True:
            data = self.send_request_queue.get()
            data_dict = json.loads(data)
            video_news_list = jsonpath.jsonpath(data_dict, '$..items[?(@.column_id=="18")]')
            for video_dict in video_news_list:
                title = video_dict['title']
                url = self.video_url.format(video_dict['id'])
                # 將標(biāo)題和 url 組成的列表放入 self.parse_detail_queue 隊(duì)列中
                self.parse_detail_queue.put([title, url])
            self.send_request_queue.task_done()

    def send_detail_request(self):
        while True:
            video_list = self.parse_detail_queue.get()
            data = requests.get(video_list[1], headers=self.headers).content.decode()
            # 將響應(yīng)字符串和標(biāo)題組成的列表放入隊(duì)列
            self.detail_data_queue.put([data, video_list[0]])
            self.parse_detail_queue.task_done()

    def parse_video_url(self):
        while True:
            list = self.detail_data_queue.get()
            pattern = re.compile('http://video\.chuangkr\.china\.com\.cn/.*vb1152\.mp4?')
            try:
                # 響應(yīng)字符串正則匹配，獲得 MP4文件的 url
                str = pattern.search(list[0]).group()
                video_url = str.split(',')[-1].lstrip('"url_1152":"')
            except AttributeError:
                pass
            else:
                if video_url:
                    # 將 MP4 文件 url 和標(biāo)題組成列表放入隊(duì)列
                    self.parse_video_queue.put([video_url, list[1]])
                else:
                    pass
            self.detail_data_queue.task_done()

    def receive_down_load_video(self):
        while True:
            list = self.parse_video_queue.get()
            video_url = list[0]
            title = list[1]
            print('開(kāi)始下載：[{}]'.format(title))
            start = time.time()
            data = requests.get(video_url, headers=self.headers, stream=True).content
            file_name = title[:10]  # 標(biāo)題前8位作為文件名
            file_path = 'video_36kr/' + file_name + '.mp4'
            with open(file_path, 'wb') as f:
                f.write(data)
            end = time.time()
            print('\n' + '[%s]下載完成，用時(shí)%.2f秒' % (title, (end - start)))
            self.count += 1
            self.parse_video_queue.task_done()

    def run(self):
        total_page = int(input('輸入要抓取的頁(yè)數(shù)：'))
        start = time.time()
        for page in range(1, total_page + 1):
            url = self.base_url.format(page)
            self.url_queue.put(url)

            th_list = []
            for i in range(3):
                send_th = Thread(target=self.send_request)
                th_list.append(send_th)

                parse_th = Thread(target=self.parse_detail)
                th_list.append(parse_th)

                send_detail_th = Thread(target=self.send_detail_request)
                th_list.append(send_detail_th)

                parse_video_th = Thread(target=self.parse_video_url)
                th_list.append(parse_video_th)

                download_th = Thread(target=self.receive_down_load_video)
                th_list.append(download_th)

            for th in th_list:
                th.setDaemon(True)  # 把子線程設(shè)置為守護(hù)線程蜓谋，主線程結(jié)束，子線程也結(jié)束
                th.start()

            for q in [self.url_queue, self.send_request_queue, self.parse_detail_queue, self.detail_data_queue,
                      self.parse_video_queue]:
                q.join()  # 隊(duì)列計(jì)數(shù)不為0的時(shí)候讓主線程阻塞等待炭分，隊(duì)列計(jì)數(shù)為0的時(shí)候主線程才會(huì)繼續(xù)往后執(zhí)行
        end = time.time()
        print('>>>全部下載完成桃焕，總耗時(shí)%s秒<<<' % (end - start))
        print('共下載視頻個(gè)數(shù)：{}'.format(self.count))

if __name__ == '__main__':
    Krspider().run()