篇幅有限
完整內(nèi)容及源碼關(guān)注公眾號：ReverseCode，發(fā)送 沖
題目
http://match.yuanrenxue.com/match/3
抓取下列5頁商標(biāo)的數(shù)據(jù)颗搂，并將出現(xiàn)頻率最高的申請?zhí)柼钊氪鸢钢?/p>
抓包
抓包
分析
http://match.yuanrenxue.com/match/3 請求原始網(wǎng)頁后請求一堆js/css,并沒有攜帶cookie和特殊的返回
初始請求返回
http://match.yuanrenxue.com/logo 每次請求頁數(shù)的時候都會先請求logo并set了一個cookie靶草，說明cookie是從服務(wù)器返回的
獲取cookie
http://match.yuanrenxue.com/api/match/3 請求返回頁面json數(shù)據(jù)蹄胰，攜帶logo返回的cookie
返回json
沒有帶cookie不能訪問http://match.yuanrenxue.com/api/match/3
使用請求頭加引號.py 將fiddler的請求頭包上
請求頭加引號.py
import re
old_headers ='''
Connection: keep-alive
Accept: application/json, text/javascript, */*; q=0.01
X-Requested-With: XMLHttpRequest
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.70 Safari/537.36
Referer: http://match.yuanrenxue.com/match/3
Accept-Encoding: gzip, deflate
Accept-Language: zh-CN,zh;q=0.9
Cookie: sessionid=7ly07o9fclh1llfsposkhh9jqvujxkth
'''
pattern = '^(.*?):[\s]*(.*?)$'
headers = ""
for line in old_headers.splitlines():
headers += (re.sub(pattern,'\'\\1\': \'\\2\',',line)) + '\n'
print(headers[:-2])
加上cookie使用python請求抓取返回一堆js代碼，因?yàn)閏ookie是由服務(wù)器生成的奕翔，所以這一段返回的js沒有意義
headers = {
'Connection': 'keep-alive',
'Accept': 'application/json, text/javascript, */*; q=0.01',
'X-Requested-With': 'XMLHttpRequest',
'User-Agent': 'yuanrenxue.project',
'Referer': 'http://match.yuanrenxue.com/match/3',
'Accept-Encoding': 'gzip, deflate',
'Accept-Language': 'zh-CN,zh;q=0.9',
'Cookie': 'sessionid=7ly07o9fclh1llfsposkhh9jqvujxkth'
}
url = 'http://match.yuanrenxue.com/api/match/3'
res = requests.get(url=url, headers=headers)
print(res.text)
爬蟲
規(guī)律：請求完logo后再請求api則正常返回裕寨，同理請求第二頁
session = requests.session()
headers = {
'Connection': 'keep-alive',
'Accept': 'application/json, text/javascript, */*; q=0.01',
'X-Requested-With': 'XMLHttpRequest',
'User-Agent': 'yuanrenxue.project',
'Referer': 'http://match.yuanrenxue.com/match/3',
'Accept-Encoding': 'gzip, deflate',
'Accept-Language': 'zh-CN,zh;q=0.9'
}
session.headers = headers
url_logo = 'http://match.yuanrenxue.com/logo'
res = session.post(url_logo)
print(res, res.cookies)
url = 'http://match.yuanrenxue.com/api/match/3?page=1'
res = session.get(url=url)
print(res.text)
本文由博客群發(fā)一文多發(fā)等運(yùn)營工具平臺 OpenWrite 發(fā)布
