在開發(fā)爬蟲過程中經(jīng)常會遇到IP被封掉的情況汇鞭，這時就需要用到代理IP
-
1.requests用代理
import requests
url = "http://www.baidu.com"
proxies = {
"http": "http://10.10.1.10:3128",
"https": "http://10.10.1.10:1080",
}
response = requests.get(url, proxies=proxies)
print response.content
-
2.加頭文件
import requests
url = "http://www.baidu.com"
headers = {
'User-Agent':'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6'
}
response = requests.get(url,headers = headers)
print response.content
