用標(biāo)題中的四種方式解析網(wǎng)頁(yè)咒循，比較其解析速度。復(fù)習(xí)PyQuery和PySpider绞愚，PySpider這個(gè)項(xiàng)目有點(diǎn)老了叙甸，現(xiàn)在還是使用被淘汰的PhantomJS。
系統(tǒng)配置位衩、Python版本對(duì)解析速度也有影響裆蒸，下面是我的結(jié)果（lxml與xpath最快，bs最慢）：
==== Python version: 3.6.7 (v3.6.7:6ec5cf24b7, Oct 20 2018, 03:02:14)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] =====
==== Total trials: 10000 =====
bs4 total time: 7.7
pq total time: 0.9
lxml (cssselect) total time: 0.9
lxml (xpath) total time: 0.6
regex total time: 1.0 (doesn't find all p)
拷貝下面代碼可以自測(cè)：
import re
import sys
import time
import requests
from lxml.html import fromstring
from pyquery import PyQuery as pq
from bs4 import BeautifulSoup as bs
def Timer():
a = time.time()
while True:
c = time.time()
yield time.time()-a
a = c
timer = Timer()
url = "http://www.python.org/"
html = requests.get(url).text
num = 10000
print ('\n==== Python version: %s =====' %sys.version)
print ('\n==== Total trials: %s =====' %num)
next(timer)
soup = bs(html, 'lxml')
for x in range(num):
paragraphs = soup.findAll('p')
t = next(timer)
print ('bs4 total time: %.1f' %t)
d = pq(html)
for x in range(num):
paragraphs = d('p')
t = next(timer)
print ('pq total time: %.1f' %t)
tree = fromstring(html)
for x in range(num):
paragraphs = tree.cssselect('p')
t = next(timer)
print ('lxml (cssselect) total time: %.1f' %t)
tree = fromstring(html)
for x in range(num):
paragraphs = tree.xpath('.//p')
t = next(timer)
print ('lxml (xpath) total time: %.1f' %t)
for x in range(num):
paragraphs = re.findall('<[p ]>.*?</p>', html)
t = next(timer)
print ('regex total time: %.1f (doesn\'t find all p)\n' %t)
借PyQuery復(fù)習(xí)CSS選擇器糖驴。
PyQuery支持下載網(wǎng)頁(yè)為文本僚祷，是通過(guò)urllib或Requests實(shí)現(xiàn)的：
from pyquery import PyQuery as pq
url = 'https://www.feixiaohao.com/currencies/bitcoin/'
headers = {
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 '
'(KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36'
}
doc = pq(url=url, headers=headers, method='get')
# 也支持發(fā)送表單數(shù)據(jù)
# pq(your_url, {'q': 'foo'}, method='post', verify=True)
# btc價(jià)格
btc_price = doc('.mainPrice span.convert').text()
print(btc_price)
# btc logo
btc_logo = doc('img.coinLogo').attr('src')
print(btc_logo)
也可以直接加載文檔字符串或html文檔：
from pyquery import PyQuery as pq
html = '''
<div>
<ul>
<li class="item-0">first item</li>
<li class="item-1"><a href="link2.html">second item</a></li>
<li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>
<li class="item-1 active"><a href="link4.html">fourth item</a></li>
<li class="item-0"><a href="link5.html">fifth item</a></li>
</ul>
</div>
'''
doc = pq(html)
# doc = pq(filename='demo.html')
# 使用eq可以按次序選擇
print(doc('li').eq(1))
查找上下級(jí)元素可以通過(guò)find()哪痰，children()，parent()久妆，parents()晌杰，siblings()。CSS選擇器舉例如下：
Pyspider的選擇器是PyQuery筷弦。下面的例子是使用PySpider抓取IMDB250信息肋演，fetch_type設(shè)為了js，存入MongoDB烂琴。
#!/usr/bin/env python
# -*- encoding: utf-8 -*-
# Created on 2019-01-30 16:22:03
# Project: imdb
from pyspider.libs.base_handler import *
class Handler(BaseHandler):
crawl_config = {
}
@every(minutes=5)
def on_start(self):
self.crawl('https://www.imdb.com/chart/top', callback=self.index_page)
@config(age=10 * 24 * 60 * 60)
def index_page(self, response):
for each in response.doc('.titleColumn > a').items():
self.crawl(each.attr.href, callback=self.detail_page, fetch_type='js')
@config(priority=2)
def detail_page(self, response):
return {
"title": response.doc('h1').text(),
"voters": response.doc('[itemprop="aggregateRating"] > a > span').text(),
"score": response.doc('strong > span').text()
}
# 需要再init中定義mongoclient
def on_result(self, result):
self.mongo.insert_result(result)
super(Handler, self).on_result(result)
PySpider的文檔
http://docs.pyspider.org/en/latest/
PySpider目前還不支持Python3.7爹殊，所以只好用Python3.6。
在MacOSX上安裝Pyspider時(shí)奸绷，反復(fù)彈出這個(gè)錯(cuò)誤Failed building wheel for pycurl和error: command 'gcc' failed with exit status 1梗夸。是SSL導(dǎo)致的錯(cuò)誤，解決方法如下：
(env)$ pip uninstall pycurl
(env)$ pip install --upgrade pip
(env)$ export LDFLAGS=-L/usr/local/opt/openssl/lib
(env)$ export CPPFLAGS=-I/usr/local/opt/openssl/include
(env)$ export PYCURL_SSL_LIBRARY=openssl
(env)$ pip install pycurl
