1.前期準(zhǔn)備
具體請(qǐng)查看上一篇
2.準(zhǔn)備庫(kù)requests,BeautifulSoup,xlwt,lxml
1.BeautifulSoup：是專業(yè)的網(wǎng)頁(yè)爬取庫(kù)，方便抓取網(wǎng)頁(yè)信息
2.xlwt：生成excel表格
3.lxml：xml解析庫(kù)
3.具體思路
企查查網(wǎng)站具有一定的反爬機(jī)制党饮，直接爬取會(huì)受到網(wǎng)站阻攔劝评，所以我們需要模擬瀏覽器請(qǐng)求络拌，繞過(guò)反爬機(jī)制研侣，打開企查查網(wǎng)站矢劲，獲取cookie及一系列請(qǐng)求頭文件蒲拉，然后使用BeautifulSoup分析網(wǎng)頁(yè)節(jié)點(diǎn)捕捉需要的信息
4.源碼
# encoding: utf-8
import requests
from bs4 import BeautifulSoup
import lxml
import xlwt
import re
def craw():
file = xlwt.Workbook()
table = file.add_sheet('sheet1', cell_overwrite_ok=True)
print('正在爬取潭苞，請(qǐng)稍等....')
for n in range(1,500):
print('第'+ str(n) + '頁(yè)......')
url = 'https://www.qichacha.com/g_JS_' + str(n) + '.html'
user_agent = 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36'
headers = {
'Host': 'www.qichacha.com',
'User-Agent': r'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
'Accept-Language': 'zh-CN,zh;q=0.9',
'Accept-Encoding': 'gzip, deflate, br',
'Referer': 'http://www.qichacha.com/',
'Cookie': r'zg_did=%7B%22did%22%3A%20%22166870cd07f60d-0c80294526eac7-36664c08-1fa400-166870cd0801af%22%7D; acw_tc=3af3b59815398640670163813ec3ddf30042b9b31607691a7b8d249c27; UM_distinctid=166870d292d85-016e1a972f471f-36664c08-1fa400-166870d292f349; _uab_collina=153986407973326937715323; QCCSESSID=g0gqbq7t1r8ksn94j8ii1qpbq1; CNZZDATA1254842228=364260894-1539860390-https%253A%252F%252Fwww.qichacha.com%252F%7C1540383468; Hm_lvt_3456bee468c83cc63fb5147f119f1075=1539864081,1540384169; zg_de1d1a35bfa24ce29bbf2c7eb17e6c4f=%7B%22sid%22%3A%201540384168992%2C%22updated%22%3A%201540384533698%2C%22info%22%3A%201539864055943%2C%22superProperty%22%3A%20%22%7B%7D%22%2C%22platform%22%3A%20%22%7B%7D%22%2C%22utm%22%3A%20%22%7B%7D%22%2C%22referrerDomain%22%3A%20%22%22%7D; Hm_lpvt_3456bee468c83cc63fb5147f119f1075=1540384534',
'Connection': 'keep-alive',
'If-Modified-Since': 'Wed, 24 Oct 2018 12:35:27 GMT',
'If-None-Match': '"59*******"',
'Cache-Control': 'private',
}
response = requests.get(url, headers=headers)
if response.status_code != 200:
response.encoding = 'utf-8'
print(response.status_code)
print('ERROR')
# soup = BeautifulSoup(response.text, 'lxml')
html_str = response.text
soup = BeautifulSoup(html_str, 'html.parser')
list = []
# list = soup.findAll(class_='panel panel-default')
list = soup.findAll(class_='panel panel-default')
for i in range(len(list)):
text = list[i]
soup2 = BeautifulSoup(str(text), 'lxml')
icon = soup2.find('img').attrs['src']
table.write((n - 1) * 10 + i, 1, str(icon))
name = soup2.find(class_='name').text
table.write((n - 1) * 10 + i, 2, name)
try:
content = soup2.findAll(class_='text-muted clear text-ellipsis m-t-xs')[0].text
address = soup2.findAll(class_='text-muted clear text-ellipsis m-t-xs')[1].text
table.write((n - 1) * 10 + i, 3, content)
table.write((n - 1) * 10 + i, 4, address)
except:
print('第'+str(n)+'頁(yè)第'+str(i)+'行錯(cuò)誤')
file.save('D:/qcc.xls')
if __name__ == '__main__':
craw()
5.結(jié)果
image.png
****6.申明
本次代碼僅供學(xué)習(xí)使用忽冻，不應(yīng)用于任何商業(yè)用途，若被查處此疹，概不負(fù)責(zé)僧诚。
--完成
