參考文獻:利用EditThisCookie獲取cookie跳過登陸驗證(醫(yī)脈通)
EditThisCookie是一款可以管理Chrome瀏覽器中cookies的插件,用戶可以利用EditThisCookie添加,刪除,編輯,搜索翩隧,鎖定和屏蔽cookies。
我們手動登陸醫(yī)脈通后,利用 EditThisCookie 導(dǎo)出 cookie腾窝,并將其保存到一個txt文件中
這時cookie已被我們成功獲取
下一步將cookie附入我們每次的請求,如下代碼所示
cookie_jar = cookielib.MozillaCookieJar()
cookies = open('cookie.txt').read()
for cookie in json.loads(cookies):
cookie_jar.set_cookie(cookielib.Cookie(version=0, name=cookie['name'],
value=cookie['value'], port=None, port_specified=False,
domain=cookie['domain'], domain_specified=False,
domain_initial_dot=False, path=cookie['path'], path_specified=True,
secure=cookie['secure'], expires=None, discard=True, comment=None,
comment_url=None, rest={'HttpOnly': None}, rfc2109=False))
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookie_jar),urllib2.HTTPHandler)
urllib2.install_opener(opener)
現(xiàn)在我們已經(jīng)將cokie放入urllib2包中了居砖,以后每次訪問的時候cookielib都會幫我們自動管理cookie虹脯,不需要我們操心啦!悯蝉!
爬蟲
其實cookie不用我們管后归形,爬蟲就是很普通的urllib2的爬蟲了,沒有什么特別的鼻由。
代碼
# -*- coding: utf-8 -*-
import json
import urllib2
import cookielib
from pyquery import PyQuery as pq
class YMT(object):
def __init__(self):
"""
讀取從 EditThisCookie 上獲取的cookie
并將其綁定到 urllib2 上
以后每次爬蟲訪問網(wǎng)站時都會自動帶上cookie暇榴,不再需要人工參與
"""
cookie_jar = cookielib.MozillaCookieJar()
cookies = open('cookie.txt').read()
for cookie in json.loads(cookies):
cookie_jar.set_cookie(cookielib.Cookie(version=0, name=cookie['name'], value=cookie['value'], port=None, port_specified=False, domain=cookie['domain'], domain_specified=False, domain_initial_dot=False, path=cookie['path'], path_specified=True, secure=cookie['secure'], expires=None, discard=True, comment=None, comment_url=None, rest={'HttpOnly': None}, rfc2109=False))
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookie_jar),urllib2.HTTPHandler)
urllib2.install_opener(opener)
def _get(self, url):
"""
:param url: 需要訪問的url
:return: 返回該url的網(wǎng)頁內(nèi)容
"""
request = urllib2.Request(url=url)
request.add_header('Referer', 'http://meddic.medlive.cn/search/search.do')
response = urllib2.urlopen(request)
data = response.read()
return data
def get_word_translate(self,word):
"""
:param word: 需要翻譯的英文單詞
:return: 如果有對應(yīng)英文的中文,則返回該中文翻譯蕉世,否則返回None
"""
url = 'http://meddic.medlive.cn/search/search.do?word=%s' % word.replace(' ','+')
html = self._get(url)
# 用 pyquery 定位翻譯
q = pq(html)
paraphrase_list = q('.paraphrase_list')
for paraphrase in paraphrase_list:
p = pq(paraphrase)
if p('.dictionary').text() == '英漢醫(yī)學(xué)短語詞典':
return p('.dictionary_message').text()
return None
if __name__ == '__main__':
ymt = YMT()
word = 'Whipple disease'
print ymt.get_word_translate(word)