Python網(wǎng)絡數(shù)據(jù)采集1-Beautifulsoup的使用
來自此書: [美]Ryan Mitchell 《Python網(wǎng)絡數(shù)據(jù)采集》，例子是照搬的，覺得跟著敲一遍還是有作用的昙沦，所以記錄下來。
import requests
from bs4 import BeautifulSoup
res = requests.get('https://www.pythonscraping.com/pages/page1.html')
soup = BeautifulSoup(res.text, 'lxml')
print(soup.h1)
<h1>An Interesting Title</h1>
使用urllib訪問頁面是這樣的载荔，read返回的是字節(jié)盾饮，需要解碼為utf-8的文本。像這樣a.read().decode('utf-8')懒熙，不過在使用bs4解析時候丘损，可以直接傳入urllib庫返回的響應對象。
import urllib.request
a = urllib.request.urlopen('https://www.pythonscraping.com/pages/page1.html')
soup = BeautifulSoup(a, 'lxml')
print(soup.h1)
<h1>An Interesting Title</h1>
抓取所有CSS class屬性為green的span標簽工扎，這些是人名徘钥。
import requests
from bs4 import BeautifulSoup
res = requests.get('https://www.pythonscraping.com/pages/warandpeace.html')
soup = BeautifulSoup(res.text, 'lxml')
green_names = soup.find_all('span', class_='green')
for name in green_names:
print(name.string)
Anna
Pavlovna Scherer
Empress Marya
Fedorovna
Prince Vasili Kuragin
Anna Pavlovna
St. Petersburg
the prince
Anna Pavlovna
Anna Pavlovna
...
孩子(child)和后代(descendant)是不一樣的。孩子標簽就是父標簽的直接下一代肢娘，而后代標簽則包括了父標簽下面所有的子子孫孫呈础。通俗來說，descendant包括了child橱健。
import requests
from bs4 import BeautifulSoup
res = requests.get('https://www.pythonscraping.com/pages/page3.html')
soup = BeautifulSoup(res.text, 'lxml')
gifts = soup.find('table', id='giftList').children
for name in gifts:
print(name)
<tr><th>
Item Title
</th><th>
Description
</th><th>
Cost
</th><th>
Image
</th></tr>
<tr class="gift" id="gift1"><td>
Vegetable Basket
</td><td>
This vegetable basket is the perfect gift for your health conscious (or overweight) friends!
<span class="excitingNote">Now with super-colorful bell peppers!</span>
</td><td>
$15.00
</td><td>
</td></tr>
<tr class="gift" id="gift2"><td>
Russian Nesting Dolls
</td><td>
Hand-painted by trained monkeys, these exquisite dolls are priceless! And by "priceless," we mean "extremely expensive"! <span class="excitingNote">8 entire dolls per set! Octuple the presents!</span>
</td><td>
$10,000.52
</td><td>
</td></tr>
找到表格后而钞，選取當前結(jié)點為tr，并找到這個tr之后的兄弟節(jié)點拘荡，由于第一個tr為表格標題臼节，這樣的寫法能提取出所有除開表格標題的正文數(shù)據(jù)。
import requests
from bs4 import BeautifulSoup
res = requests.get('https://www.pythonscraping.com/pages/page3.html')
soup = BeautifulSoup(res.text, 'lxml')
gifts = soup.find('table', id='giftList').tr.next_siblings
for name in gifts:
print(name)
<tr class="gift" id="gift1"><td>
Vegetable Basket
</td><td>
This vegetable basket is the perfect gift for your health conscious (or overweight) friends!
<span class="excitingNote">Now with super-colorful bell peppers!</span>
</td><td>
$15.00
</td><td>
</td></tr>
<tr class="gift" id="gift2"><td>
Russian Nesting Dolls
</td><td>
Hand-painted by trained monkeys, these exquisite dolls are priceless! And by "priceless," we mean "extremely expensive"! <span class="excitingNote">8 entire dolls per set! Octuple the presents!</span>
</td><td>
$10,000.52
</td><td>
</td></tr>
查找商品的價格，可以根據(jù)商品的圖片找到其父標簽<td>网缝，其上一個兄弟標簽就是價格巨税。
import requests
from bs4 import BeautifulSoup
res = requests.get('https://www.pythonscraping.com/pages/page3.html')
soup = BeautifulSoup(res.text, 'lxml')
price = soup.find('img', src='../img/gifts/img1.jpg').parent.previous_sibling.string
print(price)
$15.00
采集所有商品圖片，為了避免其他圖片亂入粉臊。使用正則表達式精確搜索草添。
import re
import requests
from bs4 import BeautifulSoup
res = requests.get('https://www.pythonscraping.com/pages/page3.html')
soup = BeautifulSoup(res.text, 'lxml')
imgs= soup.find_all('img', src=re.compile(r'../img/gifts/img.*.jpg'))
for img in imgs:
print(img['src'])
../img/gifts/img1.jpg
../img/gifts/img2.jpg
../img/gifts/img3.jpg
../img/gifts/img4.jpg
../img/gifts/img6.jpg
find_all()還可以傳入函數(shù)，對這個函數(shù)有個要求：就是其返回值必須是布爾類型扼仲，若是True則保留果元，若是False則剔除。
import re
import requests
from bs4 import BeautifulSoup
res = requests.get('https://www.pythonscraping.com/pages/page3.html')
soup = BeautifulSoup(res.text, 'lxml')
# lambda tag: tag.name=='img'
tags = soup.find_all(lambda tag: tag.has_attr('src'))
for tag in tags:
print(tag)
tag是一個Element對象犀盟，has_attr用來判斷是否有該屬性。tag.name則是獲取標簽名蝇狼。在上面的網(wǎng)頁中阅畴，下面的寫法返回的結(jié)果一樣。
lambda tag: tag.has_attr('src')或lambda tag: tag.name=='img'
by @sunhaiyu
2017.7.14
