python中解析網(wǎng)頁(yè)內(nèi)容基本步驟
- 使用BeautifulSoup解析網(wǎng)頁(yè)
Soup = BeautifulSoup(html, 'lxml') - 描述要爬取得東西在哪里
=Soup.select(‘路徑’) - 從標(biāo)簽中獲得需要的信息，按一定格式裝在數(shù)據(jù)容器中（字典的列表）到推，便于查詢
<p></p>
[{title=Something, rate=4.0},{title=Something, rate=4.0},{title=Something, rate=4.0},{title=Something, rate=4.0}]
兩種不同的路徑描述方式
- CSS Selector:
例如：body>div.main-content>ul>li:nth-child(1)>img - XPATH
例如：/html/body/div[2]/ul/li(1)/img
作業(yè)代碼
from bs4 import BeautifulSoup
info= []
#讀取本地HTML文件并找到要爬取的片段
with open('E:/工作盤Workshop/a3-編程練習(xí)/Python Practice/0824/1.2/index.html','r',) as web_data:
Soup = BeautifulSoup(web_data,'lxml')
images = Soup.select('body > div:nth-of-type(1)> div > div.col-md-9 > div:nth-of-type(2) > div > div > img')
titles = Soup.select('body > div:nth-of-type(1) > div > div.col-md-9 > div:nth-of-type(2) > div > div > div.caption > h4 > a')
prices = Soup.select('body > div:nth-of-type(1) > div > div.col-md-9 > div:nth-of-type(2) > div > div > div.caption > h4.pull-right')
votes = Soup.select('body > div:nth-of-type(1) > div > div.col-md-9 > div:nth-of-type(2) > div > div > div.ratings > p.pull-right')
stars = Soup.select('body > div:nth-of-type(1) > div > div.col-md-9 > div:nth-of-type(2) > div > div > div.ratings > p:nth-of-type(2)')
#print(images,titles,prices,votes,stars,sep='\n------------------\n')
#從片段中提取有效數(shù)據(jù)考赛，每組數(shù)據(jù)生成一個(gè)字典，再把字典放入列表
for title,image,price,vote,star in zip(titles,images,prices,votes,stars):
data = {
'title':title.get_text(),
'price':price.get_text(),
'star': len(star.find_all("span", "glyphicon glyphicon-star")),
'vote':vote.get_text()[:-8],
'image':image.get('src')
}
info.append(data)
#因列表較長(zhǎng)莉测，用一個(gè)函數(shù)逐條打印各元素
def print_lol(the_list):
for each_item in the_list:
print(each_item)
print_lol(info)
輸出效果
1.jpg
遺留問題
- Chrome瀏覽器中copy selector得到的路徑中有一處錯(cuò)誤颜骤，手工更正后才能找到元素。元素實(shí)際位置是
body > div:nth-of-type(1)下面捣卤，瀏覽器復(fù)制出來(lái)的路徑卻是body > div:nth-of-type(2)忍抽。 - 能否找到更高效的方法把-child替換為-of-type，手工修改確實(shí)很浪費(fèi)時(shí)間董朝。
