問(wèn)題的由來(lái)
??前幾天抬探,在微信公眾號(hào)(Python爬蟲(chóng)及算法)上有個(gè)人問(wèn)了筆者一個(gè)問(wèn)題针史,如何利用爬蟲(chóng)來(lái)實(shí)現(xiàn)如下的需求劝枣,需要爬取的網(wǎng)頁(yè)如下(網(wǎng)址為:https://www.wikidata.org/w/index.php?title=Special:WhatLinksHere/Q5&limit=500&from=0):
??我們的需求為爬取紅色框框內(nèi)的名人(有500條記錄男韧,圖片只展示了一部分)的 名字以及其介紹氢哮,關(guān)于其介紹皇型,點(diǎn)擊該名人的名字即可诬烹,如下圖:
這就意味著我們需要爬取500個(gè)這樣的頁(yè)面助析,即500個(gè)HTTP請(qǐng)求(暫且這么認(rèn)為吧),然后需要提取這些網(wǎng)頁(yè)中的名字和描述椅您,當(dāng)然有些不是名人外冀,也沒(méi)有描述,我們可以跳過(guò)掀泳。最后雪隧,這些網(wǎng)頁(yè)的網(wǎng)址在第一頁(yè)中的名人后面可以找到,如George Washington的網(wǎng)頁(yè)后綴為Q23.
??爬蟲(chóng)的需求大概就是這樣员舵。
爬蟲(chóng)的4種姿勢(shì)
??首先脑沿,分析來(lái)爬蟲(chóng)的思路:先在第一個(gè)網(wǎng)頁(yè)(https://www.wikidata.org/w/index.php?title=Special:WhatLinksHere/Q5&limit=500&from=0)中得到500個(gè)名人所在的網(wǎng)址,接下來(lái)就爬取這500個(gè)網(wǎng)頁(yè)中的名人的名字及描述马僻,如無(wú)描述庄拇,則跳過(guò)。
??接下來(lái)韭邓,我們將介紹實(shí)現(xiàn)這個(gè)爬蟲(chóng)的4種方法措近,并分析它們各自的優(yōu)缺點(diǎn),希望能讓讀者對(duì)爬蟲(chóng)有更多的體會(huì)女淑。實(shí)現(xiàn)爬蟲(chóng)的方法為:
- 一般方法(同步瞭郑,requests+BeautifulSoup)
- 并發(fā)(使用concurrent.futures模塊以及requests+BeautifulSoup)
- 異步(使用aiohttp+asyncio+requests+BeautifulSoup)
- 使用框架Scrapy
一般方法
??一般方法即為同步方法,主要使用requests+BeautifulSoup鸭你,按順序執(zhí)行屈张。完整的Python代碼如下:
import requests
from bs4 import BeautifulSoup
import time
# 開(kāi)始時(shí)間
t1 = time.time()
print('#' * 50)
url = "http://www.wikidata.org/w/index.php?title=Special:WhatLinksHere/Q5&limit=500&from=0"
# 請(qǐng)求頭部
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36'}
# 發(fā)送HTTP請(qǐng)求
req = requests.get(url, headers=headers)
# 解析網(wǎng)頁(yè)
soup = BeautifulSoup(req.text, "lxml")
# 找到name和Description所在的記錄
human_list = soup.find(id='mw-whatlinkshere-list')('li')
urls = []
# 獲取網(wǎng)址
for human in human_list:
url = human.find('a')['href']
urls.append('https://www.wikidata.org'+url)
# 獲取每個(gè)網(wǎng)頁(yè)的name和description
def parser(url):
req = requests.get(url)
# 利用BeautifulSoup將獲取到的文本解析成HTML
soup = BeautifulSoup(req.text, "lxml")
# 獲取name和description
name = soup.find('span', class_="wikibase-title-label")
desc = soup.find('span', class_="wikibase-descriptionview-text")
if name is not None and desc is not None:
print('%-40s,\t%s'%(name.text, desc.text))
for url in urls:
parser(url)
t2 = time.time() # 結(jié)束時(shí)間
print('一般方法,總共耗時(shí):%s' % (t2 - t1))
print('#' * 50)
輸出的結(jié)果如下(省略中間的輸出袱巨,以......代替):
##################################################
George Washington , first President of the United States
Douglas Adams , British author and humorist (1952–2001)
......
Willoughby Newton , Politician from Virginia, USA
Mack Wilberg , American conductor
一般方法阁谆,總共耗時(shí):724.9654655456543
##################################################
使用同步方法,總耗時(shí)約725秒愉老,即12分鐘多场绿。
??一般方法雖然思路簡(jiǎn)單,容易實(shí)現(xiàn)俺夕,但效率不高裳凸,耗時(shí)長(zhǎng)。那么劝贸,使用并發(fā)試試看姨谷。
并發(fā)方法
??并發(fā)方法使用多線(xiàn)程來(lái)加速一般方法,我們使用的并發(fā)模塊為concurrent.futures模塊映九,設(shè)置多線(xiàn)程的個(gè)數(shù)為20個(gè)(實(shí)際不一定能達(dá)到梦湘,視計(jì)算機(jī)而定)。完整的Python代碼如下:
import requests
from bs4 import BeautifulSoup
import time
from concurrent.futures import ThreadPoolExecutor, wait, ALL_COMPLETED
# 開(kāi)始時(shí)間
t1 = time.time()
print('#' * 50)
url = "http://www.wikidata.org/w/index.php?title=Special:WhatLinksHere/Q5&limit=500&from=0"
# 請(qǐng)求頭部
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36'}
# 發(fā)送HTTP請(qǐng)求
req = requests.get(url, headers=headers)
# 解析網(wǎng)頁(yè)
soup = BeautifulSoup(req.text, "lxml")
# 找到name和Description所在的記錄
human_list = soup.find(id='mw-whatlinkshere-list')('li')
urls = []
# 獲取網(wǎng)址
for human in human_list:
url = human.find('a')['href']
urls.append('https://www.wikidata.org'+url)
# 獲取每個(gè)網(wǎng)頁(yè)的name和description
def parser(url):
req = requests.get(url)
# 利用BeautifulSoup將獲取到的文本解析成HTML
soup = BeautifulSoup(req.text, "lxml")
# 獲取name和description
name = soup.find('span', class_="wikibase-title-label")
desc = soup.find('span', class_="wikibase-descriptionview-text")
if name is not None and desc is not None:
print('%-40s,\t%s'%(name.text, desc.text))
# 利用并發(fā)加速爬取
executor = ThreadPoolExecutor(max_workers=20)
# submit()的參數(shù): 第一個(gè)為函數(shù), 之后為該函數(shù)的傳入?yún)?shù)捌议,允許有多個(gè)
future_tasks = [executor.submit(parser, url) for url in urls]
# 等待所有的線(xiàn)程完成哼拔,才進(jìn)入后續(xù)的執(zhí)行
wait(future_tasks, return_when=ALL_COMPLETED)
t2 = time.time() # 結(jié)束時(shí)間
print('并發(fā)方法,總共耗時(shí):%s' % (t2 - t1))
print('#' * 50)
輸出的結(jié)果如下(省略中間的輸出瓣颅,以......代替):
##################################################
Larry Sanger , American former professor, co-founder of Wikipedia, founder of Citizendium and other projects
Ken Jennings , American game show contestant and writer
......
Antoine de Saint-Exupery , French writer and aviator
Michael Jackson , American singer, songwriter and dancer
并發(fā)方法倦逐,總共耗時(shí):226.7499692440033
##################################################
使用多線(xiàn)程并發(fā)后的爬蟲(chóng)執(zhí)行時(shí)間約為227秒,大概是一般方法的三分之一的時(shí)間宫补,速度有了明顯的提升懊世选!多線(xiàn)程在速度上有明顯提升粉怕,但執(zhí)行的網(wǎng)頁(yè)順序是無(wú)序的健民,在線(xiàn)程的切換上開(kāi)銷(xiāo)也比較大,線(xiàn)程越多贫贝,開(kāi)銷(xiāo)越大秉犹。
??關(guān)于多線(xiàn)程與一般方法在速度上的比較,可以參考文章:Python爬蟲(chóng)之多線(xiàn)程下載豆瓣Top250電影圖片稚晚。
異步方法
??異步方法在爬蟲(chóng)中是有效的速度提升手段崇堵,使用aiohttp可以異步地處理HTTP請(qǐng)求,使用asyncio可以實(shí)現(xiàn)異步IO蜈彼,需要注意的是筑辨,aiohttp只支持3.5.3以后的Python版本。使用異步方法實(shí)現(xiàn)該爬蟲(chóng)的完整Python代碼如下:
import requests
from bs4 import BeautifulSoup
import time
import aiohttp
import asyncio
# 開(kāi)始時(shí)間
t1 = time.time()
print('#' * 50)
url = "http://www.wikidata.org/w/index.php?title=Special:WhatLinksHere/Q5&limit=500&from=0"
# 請(qǐng)求頭部
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36'}
# 發(fā)送HTTP請(qǐng)求
req = requests.get(url, headers=headers)
# 解析網(wǎng)頁(yè)
soup = BeautifulSoup(req.text, "lxml")
# 找到name和Description所在的記錄
human_list = soup.find(id='mw-whatlinkshere-list')('li')
urls = []
# 獲取網(wǎng)址
for human in human_list:
url = human.find('a')['href']
urls.append('https://www.wikidata.org'+url)
# 異步HTTP請(qǐng)求
async def fetch(session, url):
async with session.get(url) as response:
return await response.text()
# 解析網(wǎng)頁(yè)
async def parser(html):
# 利用BeautifulSoup將獲取到的文本解析成HTML
soup = BeautifulSoup(html, "lxml")
# 獲取name和description
name = soup.find('span', class_="wikibase-title-label")
desc = soup.find('span', class_="wikibase-descriptionview-text")
if name is not None and desc is not None:
print('%-40s,\t%s'%(name.text, desc.text))
# 處理網(wǎng)頁(yè)幸逆,獲取name和description
async def download(url):
async with aiohttp.ClientSession() as session:
try:
html = await fetch(session, url)
await parser(html)
except Exception as err:
print(err)
# 利用asyncio模塊進(jìn)行異步IO處理
loop = asyncio.get_event_loop()
tasks = [asyncio.ensure_future(download(url)) for url in urls]
tasks = asyncio.gather(*tasks)
loop.run_until_complete(tasks)
t2 = time.time() # 結(jié)束時(shí)間
print('使用異步,總共耗時(shí):%s' % (t2 - t1))
print('#' * 50)
輸出結(jié)果如下(省略中間的輸出暮现,以......代替):
##################################################
Frédéric Tadde? , French journalist and TV host
Gabriel Gonzáles Videla , Chilean politician
......
Denmark , sovereign state and Scandinavian country in northern Europe
Usain Bolt , Jamaican sprinter and soccer player
使用異步还绘,總共耗時(shí):126.9002583026886
##################################################
顯然,異步方法使用了異步和并發(fā)兩種提速方法栖袋,自然在速度有明顯提升拍顷,大約為一般方法的六分之一。異步方法雖然效率高塘幅,但需要掌握異步編程昔案,這需要學(xué)習(xí)一段時(shí)間。
??關(guān)于異步方法與一般方法在速度上的比較电媳,可以參考文章:利用aiohttp實(shí)現(xiàn)異步爬蟲(chóng)踏揣。
??如果有人覺(jué)得127秒的爬蟲(chóng)速度還是慢,可以嘗試一下異步代碼(與之前的異步代碼的區(qū)別在于:僅僅使用了正則表達(dá)式代替BeautifulSoup來(lái)解析網(wǎng)頁(yè)匾乓,以提取網(wǎng)頁(yè)中的內(nèi)容):
import requests
from bs4 import BeautifulSoup
import time
import aiohttp
import asyncio
import re
# 開(kāi)始時(shí)間
t1 = time.time()
print('#' * 50)
url = "http://www.wikidata.org/w/index.php?title=Special:WhatLinksHere/Q5&limit=500&from=0"
# 請(qǐng)求頭部
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36'}
# 發(fā)送HTTP請(qǐng)求
req = requests.get(url, headers=headers)
# 解析網(wǎng)頁(yè)
soup = BeautifulSoup(req.text, "lxml")
# 找到name和Description所在的記錄
human_list = soup.find(id='mw-whatlinkshere-list')('li')
urls = []
# 獲取網(wǎng)址
for human in human_list:
url = human.find('a')['href']
urls.append('https://www.wikidata.org' + url)
# 異步HTTP請(qǐng)求
async def fetch(session, url):
async with session.get(url) as response:
return await response.text()
# 解析網(wǎng)頁(yè)
async def parser(html):
# 利用正則表達(dá)式解析網(wǎng)頁(yè)
try:
name = re.findall(r'<span class="wikibase-title-label">(.+?)</span>', html)[0]
desc = re.findall(r'<span class="wikibase-descriptionview-text">(.+?)</span>', html)[0]
print('%-40s,\t%s' % (name, desc))
except Exception as err:
pass
# 處理網(wǎng)頁(yè)捞稿,獲取name和description
async def download(url):
async with aiohttp.ClientSession() as session:
try:
html = await fetch(session, url)
await parser(html)
except Exception as err:
print(err)
# 利用asyncio模塊進(jìn)行異步IO處理
loop = asyncio.get_event_loop()
tasks = [asyncio.ensure_future(download(url)) for url in urls]
tasks = asyncio.gather(*tasks)
loop.run_until_complete(tasks)
t2 = time.time() # 結(jié)束時(shí)間
print('使用異步(正則表達(dá)式),總共耗時(shí):%s' % (t2 - t1))
print('#' * 50)
輸出的結(jié)果如下(省略中間的輸出,以......代替):
##################################################
Dejen Gebremeskel , Ethiopian long-distance runner
Erik Kynard , American high jumper
......
Buzz Aldrin , American astronaut
Egon Krenz , former General Secretary of the Socialist Unity Party of East Germany
使用異步(正則表達(dá)式)娱局,總共耗時(shí):16.521944999694824
##################################################
16.5秒彰亥,僅僅為一般方法的43分之一,速度如此之快衰齐,令人咋舌(感謝某人提供的嘗試)任斋。筆者雖然自己實(shí)現(xiàn)了異步方法,但用的是BeautifulSoup來(lái)解析網(wǎng)頁(yè)耻涛,耗時(shí)127秒废酷,沒(méi)想到使用正則表達(dá)式就取得了如此驚人的效果∪冢可見(jiàn)锦积,BeautifulSoup解析網(wǎng)頁(yè)雖然快,但在異步方法中歉嗓,還是限制了速度丰介。但這種方法的缺點(diǎn)為,當(dāng)你需要爬取的內(nèi)容比較復(fù)雜時(shí)鉴分,一般的正則表達(dá)式就難以勝任了哮幢,需要另想辦法。
爬蟲(chóng)框架Scrapy
??最后志珍,我們使用著名的Python爬蟲(chóng)框架Scrapy來(lái)解決這個(gè)爬蟲(chóng)橙垢。我們創(chuàng)建的爬蟲(chóng)項(xiàng)目為wikiDataScrapy,項(xiàng)目結(jié)構(gòu)如下:
在settings.py中設(shè)置“ROBOTSTXT_OBEY = False”. 修改items.py伦糯,代碼如下:
# -*- coding: utf-8 -*-
import scrapy
class WikidatascrapyItem(scrapy.Item):
# define the fields for your item here like:
name = scrapy.Field()
desc = scrapy.Field()
然后柜某,在spiders文件夾下新建wikiSpider.py,代碼如下:
import scrapy.cmdline
from wikiDataScrapy.items import WikidatascrapyItem
import requests
from bs4 import BeautifulSoup
# 獲取請(qǐng)求的500個(gè)網(wǎng)址敛纲,用requests+BeautifulSoup搞定
def get_urls():
url = "http://www.wikidata.org/w/index.php?title=Special:WhatLinksHere/Q5&limit=500&from=0"
# 請(qǐng)求頭部
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36'}
# 發(fā)送HTTP請(qǐng)求
req = requests.get(url, headers=headers)
# 解析網(wǎng)頁(yè)
soup = BeautifulSoup(req.text, "lxml")
# 找到name和Description所在的記錄
human_list = soup.find(id='mw-whatlinkshere-list')('li')
urls = []
# 獲取網(wǎng)址
for human in human_list:
url = human.find('a')['href']
urls.append('https://www.wikidata.org' + url)
# print(urls)
return urls
# 使用scrapy框架爬取
class bookSpider(scrapy.Spider):
name = 'wikiScrapy' # 爬蟲(chóng)名稱(chēng)
start_urls = get_urls() # 需要爬取的500個(gè)網(wǎng)址
def parse(self, response):
item = WikidatascrapyItem()
# name and description
item['name'] = response.css('span.wikibase-title-label').xpath('text()').extract_first()
item['desc'] = response.css('span.wikibase-descriptionview-text').xpath('text()').extract_first()
yield item
# 執(zhí)行該爬蟲(chóng)喂击,并轉(zhuǎn)化為csv文件
scrapy.cmdline.execute(['scrapy', 'crawl', 'wikiScrapy', '-o', 'wiki.csv', '-t', 'csv'])
輸出結(jié)果如下(只包含最后的Scrapy信息總結(jié)部分):
{'downloader/request_bytes': 166187,
'downloader/request_count': 500,
'downloader/request_method_count/GET': 500,
'downloader/response_bytes': 18988798,
'downloader/response_count': 500,
'downloader/response_status_count/200': 500,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2018, 10, 16, 9, 49, 15, 761487),
'item_scraped_count': 500,
'log_count/DEBUG': 1001,
'log_count/INFO': 8,
'response_received_count': 500,
'scheduler/dequeued': 500,
'scheduler/dequeued/memory': 500,
'scheduler/enqueued': 500,
'scheduler/enqueued/memory': 500,
'start_time': datetime.datetime(2018, 10, 16, 9, 48, 44, 58673)}
可以看到,已成功爬取500個(gè)網(wǎng)頁(yè)淤翔,耗時(shí)31秒翰绊,速度也相當(dāng)OK。再來(lái)看一下生成的wiki.csv文件旁壮,它包含了所有的輸出的name和description监嗜,如下圖:
可以看到,輸出的CSV文件的列并不是有序的抡谐。至于如何解決Scrapy輸出的CSV文件有換行的問(wèn)題裁奇,請(qǐng)參考stackoverflow上的回答:https://stackoverflow.com/questions/39477662/scrapy-csv-file-has-uniform-empty-rows/43394566#43394566 。
??Scrapy來(lái)制作爬蟲(chóng)的優(yōu)勢(shì)在于它是一個(gè)成熟的爬蟲(chóng)框架童叠,支持異步框喳,并發(fā)课幕,容錯(cuò)性較好(比如本代碼中就沒(méi)有處理找不到name和description的情形),但如果需要頻繁地修改中間件五垮,則還是自己寫(xiě)個(gè)爬蟲(chóng)比較好乍惊,而且它在速度上沒(méi)有超過(guò)我們自己寫(xiě)的異步爬蟲(chóng),至于能自動(dòng)導(dǎo)出CSV文件這個(gè)功能放仗,還是相當(dāng)實(shí)在的润绎。
總結(jié)
??本文內(nèi)容較多,比較了4種爬蟲(chóng)方法诞挨,每種方法都有自己的利弊莉撇,已在之前的陳述中給出,當(dāng)然惶傻,在實(shí)際的問(wèn)題中棍郎,并不是用的工具或方法越高級(jí)就越好,具體問(wèn)題具體分析嘛~
??本文到此結(jié)束银室,感謝閱讀哦~
注意:本人現(xiàn)已開(kāi)通微信公眾號(hào): Python爬蟲(chóng)與算法(微信號(hào)為:easy_web_scrape)涂佃, 歡迎大家關(guān)注哦~~