任務(wù)2.1 學(xué)習(xí)BeautifulSoup
英語生詞本
parser n. 剖析器;
prettify v. 修飾;
sibling n. 兄弟猜扮，姐妹; [生] 同科涮母，同屬; [人] 氏族成員;
在cmd命令行窗口安裝BeautifulSoup庫：
pip install beautifulsoup4
如何使用BeautifulSoup
from bs4 import BeautifulSoup
soup = BeautifulSoup('<p>data</p>' , 'html.parser')
BeautifulSoup庫的安裝小測
演示HTML頁面地址：http://python123.io/ws/demo.html
>>> r = requests.get("http://python123.io/ws/demo.html")
>>> r.text
'<html><head><title>This is a python demo page</title></head>\r\n<body>\r\n<p class="title"><b>The demo python introduces several python courses.</b></p>\r\n<p class="course">Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:\r\n<a class="py1" id="link1">Basic Python</a> and <a class="py2" id="link2">Advanced Python</a>.</p>\r\n</body></html>'
>>> demo = r.text
>>> from bs4 import BeautifulSoup
>>> soup = BeautifulSoup(demo, "html.parser")
>>> print(soup.prettify())
<html>
<head>
<title>
This is a python demo page
</title>
</head>
<body>
<p class="title">
<b>
The demo python introduces several python courses.
</b>
</p>
<p class="course">
Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:
<a class="py1" id="link1">
Basic Python
</a>
and
<a class="py2" id="link2">
Advanced Python
</a>
.
</p>
</body>
</html>
>>>
2.1.1 BeautifulSoup庫的基本元素
HTML文檔 <==> 標(biāo)簽樹 <==> BeautifulSoup類
即洼裤，BeautifulSoup類 對應(yīng)一個HTML/XML文檔的全部內(nèi)容。
BeautifulSoup類有5種基本元素：
-
Tag
標(biāo)簽期犬，最基本的信息組織單元，分別用<></>標(biāo)明開頭和結(jié)尾 -
Name
標(biāo)簽的名字采够，<p>...</p>的名字是'p'喊递，格式：<tag>.name -
Attributes
標(biāo)簽的屬性疤剑，字典形式組織滑绒，格式：<tag>.attrs -
NavigableString
標(biāo)簽內(nèi)非屬性字符串，<>...</>中字符串隘膘，格式：<tag>.string -
Comment
標(biāo)簽內(nèi)字符串的注釋部分疑故，一種特殊的Comment類型
標(biāo)簽樹的平行遍歷: （上行遍歷，下行遍歷略）
for sibling in soup.a.next_siblings:
print(sibling) #遍歷后續(xù)節(jié)點
for sibling in soup.a.previous_siblings:
print(sibling) #遍歷前續(xù)節(jié)點
問題：如何讓<html>內(nèi)容更加“友好”的顯示弯菊？
---bs4庫的prettify()方法
>>> print(soup.a.prettify())
<a class="py1" id="link1">
Basic Python
</a>
2.1.2 BeautifulSoup庫實踐案例
使用beautifulsoup提取丁香園論壇的回復(fù)內(nèi)容
- 用戶瀏覽器訪問目標(biāo)網(wǎng)站并檢查目標(biāo)內(nèi)容所在標(biāo)簽
目標(biāo)網(wǎng)址如下：http://www.dxy.cn/bbs/thread/626626#626626
用Chrome訪問的纵势，按F12可看見網(wǎng)站結(jié)構(gòu)及回復(fù)內(nèi)容所在標(biāo)簽如下圖：
- 獲取回復(fù)內(nèi)容
我們所需的評論內(nèi)容就在td class="postbody"標(biāo)簽下，利用BeautifulSoup獲取內(nèi)容
content = data.find("td", class_="postbody").text
參考代碼：
import urllib.request
from bs4 import BeautifulSoup as bs
def main():
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/58.0.3029.110 Safari/537.36 SE 2.X MetaSr 1.0"
}
url = 'http://www.dxy.cn/bbs/thread/626626'
request = urllib.request.Request(url, headers=headers)
response = urllib.request.urlopen(request).read().decode("utf-8")
html = bs(response, 'lxml')
getItem(html)
def getItem(html):
datas = [] # 用來存放獲取的用戶名和評論
for data in html.find_all("tbody"):
try:
userid = data.find("div", class_="auth").get_text(strip=True)
print(userid)
content = data.find("td", class_="postbody").get_text(strip=True)
print(content)
datas.append((userid,content))
except:
pass
print(datas)
if __name__ == '__main__':
main()
參考鏈接：https://blog.csdn.net/wwq114/article/details/88085875
說實話管钳，沒看懂钦铁。。才漆。還要認(rèn)真學(xué)習(xí)呀
任務(wù)2.1 學(xué)習(xí)xpath
要交作業(yè)了牛曹，來不及學(xué)了，待補(bǔ)充醇滥。黎比。超营。
