在之前的文章中攒磨,提到如何爬取雪球網(wǎng)用戶的股票數(shù)據(jù),但是由于爬取過(guò)程中汤徽,會(huì)存在一些問(wèn)題娩缰,比如由于網(wǎng)站設(shè)有反爬蟲(chóng)機(jī)制,所以會(huì)導(dǎo)致在爬取過(guò)程中收到403的response谒府,因此在爬取的時(shí)候需要做一些偽裝拼坎,首先要做的就是建立一個(gè)代理ip池,通過(guò)代理ip對(duì)網(wǎng)站進(jìn)行訪問(wèn)完疫,從而防止由于單一ip短時(shí)間內(nèi)頻繁訪問(wèn)而被網(wǎng)站給ban掉泰鸡。
但是要獲取穩(wěn)定又多的ip,而且實(shí)現(xiàn)每次的ip更新壳鹤,因此我們要先寫一個(gè)爬蟲(chóng)盛龄,把代理ip網(wǎng)站的ip爬下來(lái),然后封裝進(jìn)我們的ip池里芳誓。
這里我選擇的是西刺代理余舶,大家也可以自行選擇其他的代理ip網(wǎng)站。
以上是西刺代理的頁(yè)面截圖锹淌,頁(yè)面相對(duì)規(guī)整匿值,首先進(jìn)行測(cè)試,發(fā)現(xiàn)該網(wǎng)站沒(méi)有采用json的傳參方式赂摆,但是由于頁(yè)面十分的規(guī)整挟憔,因此這里考慮直接對(duì)html頁(yè)面進(jìn)行爬取,首先先看一下頁(yè)面的html源碼库正。
通過(guò)以上我們可以發(fā)現(xiàn)曲楚,頁(yè)面相對(duì)而言較為規(guī)整,每個(gè)tr下面就包含了一樣數(shù)量的td褥符,且龙誊,每個(gè)tr里的第二個(gè)td和第三個(gè)td以及倒數(shù)第三個(gè)td內(nèi)包含了我們所需要的ip,端口以及ip類型喷楣。
接下來(lái)就直接上代碼了趟大。
#設(shè)置ip池#
def get_ip():
url = 'http://www.xicidaili.com/'
headers = {
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36',
'Cookie':'_free_proxy_session=BAh7B0kiD3Nlc3Npb25faWQGOgZFVEkiJTI3MGZjZTNkM2I2NWNhY2M5MWZkYmY0NDhkMGU4MTExBjsAVEkiEF9jc3JmX3Rva2VuBjsARkkiMXhjRldkci9BWGxZTHEzSittMGczMVBiYzhIcVdHTVlKaGFkamRQK1RZVXc9BjsARg%3D%3D--0f372ba6b8e9956e504653078cc02c497684f04e; Hm_lvt_0cf76c77469e965d2957f0553e6ecf59=1539521254,1539521263,1539567080,1539577822; Hm_lpvt_0cf76c77469e965d2957f0553e6ecf59=1539578075'
}
html = requests.get(url,headers=headers)
soup = BeautifulSoup(html.text)
ips = soup.find_all('tr')
result_ls = []
for i in range(2,len(ips)):
ip_info = ips[i]
tds = ip_info.find_all('td')
if len(tds) > 0:
result_ls.append(tds[1].text+':'+tds[2].text)
else:
continue
return result_ls
首先爬取頁(yè)面上所有的td,將里面的第一個(gè)與第二個(gè)td取出铣焊,并進(jìn)行拼接逊朽,然后再把結(jié)果存入到數(shù)組中,最后返回result_ls這個(gè)數(shù)組作為ip池曲伊。
def result(user_id,ip):
#根據(jù)獲取的uid分別爬取用戶的股票信息#
url2 ='https://xueqiu.com/v4/stock/portfolio/stocks.json'
user_agent_list=[ "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 "
"(KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1",
"Mozilla/5.0 (X11; CrOS i686 2268.111.0) AppleWebKit/536.11 "
"(KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11",
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.6 "
"(KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6",
"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.6 "
"(KHTML, like Gecko) Chrome/20.0.1090.0 Safari/536.6",
"Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.1 "
"(KHTML, like Gecko) Chrome/19.77.34.5 Safari/537.1",
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.5 "
"(KHTML, like Gecko) Chrome/19.0.1084.9 Safari/536.5",
"Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 "
"(KHTML, like Gecko) Chrome/19.0.1084.36 Safari/536.5",
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 "
"(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
"Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.3 "
"(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_0) AppleWebKit/536.3 "
"(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 "
"(KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 "
"(KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",
"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 "
"(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 "
"(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
"Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.3 "
"(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 "
"(KHTML, like Gecko) Chrome/19.0.1061.0 Safari/536.3",
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.24 "
"(KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24",
"Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 "
"(KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24"]
#設(shè)置參數(shù)#
params = {
'size':10000,
'type':1,
'pid':-1,
'category':2,
'uid':str(user_id)
}
#請(qǐng)求頭文件#
headers2 = {
'User-Agent':choice(user_agent_list),
'Referer': 'https://xueqiu.com/u/'+str(user_id),
'Connection': 'keep-alive',
'cache-control':'no-cache',
'Cookie':cookies
}
#設(shè)置代理#
proxies = {'http':ip}
#發(fā)送請(qǐng)求#
response2 = requests.get(url2,headers=headers2,params=params,proxies=proxies)
time.sleep(5)
print("響應(yīng)碼:"+str(response2.status_code) + '使用ip:'+ip)
stockInfo = []
if response2.status_code == 400:
pass
else:
text2 = json.loads(response2.text)
stockInfo = text2['stocks']
return stockInfo
之后再?gòu)膇p池中選取一個(gè)ip作為參數(shù)傳入到代理中叽讳。
最后上完整代碼追他,有問(wèn)題或者改進(jìn)意見(jiàn)歡迎私信共同探討。
# -*- coding: utf-8 -*-
"""
Created on Sat Oct 13 18:33:35 2018
@author: Chen.Xiao
"""
import requests
import json
import pandas as pd
import numpy as np
import time
import ssl
from random import choice
from bs4 import BeautifulSoup
#防止ssl報(bào)錯(cuò)#
ssl._create_default_https_context = ssl._create_unverified_context
cookies = '_ga=GA1.2.1962796314.1538825138; device_id=516f0c95e19bf48ba37ca2f3e68811ed; __utmz=1.1538825152.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); s=fi17cigo1a; bid=5ab5c3e86b4b2c95f32bb04bd50357c3_jn791euz; aliyungf_tc=AQAAAMdnpDWO5QEA/c5CMbWjR/czsdk4; Hm_lvt_1db88642e346389874251b5a1eded6e3=1554460329; __utmc=1; xq_a_token=3450822dc3b6c0b631c3ba4768fcddac23c054d7; xq_a_token.sig=f1ZlcbP6BFkUA32I1cexLDa2KTk; xq_r_token=0de8d3b6155ce156310ff6d4e214d4532198ccec; xq_r_token.sig=8xgxBORSc2oawdjl5r3ksadcr9s; _gid=GA1.2.684281849.1555120269; u=231555120270276; _gat=1; __utma=1.1962796314.1538825138.1554640094.1555121068.23; __utmt=1; __utmb=1.2.10.1555121068; Hm_lpvt_1db88642e346389874251b5a1eded6e3=1555121080'
#設(shè)置ip池#
def get_ip():
url = 'http://www.xicidaili.com/'
headers = {
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36',
'Cookie':'_free_proxy_session=BAh7B0kiD3Nlc3Npb25faWQGOgZFVEkiJTI3MGZjZTNkM2I2NWNhY2M5MWZkYmY0NDhkMGU4MTExBjsAVEkiEF9jc3JmX3Rva2VuBjsARkkiMXhjRldkci9BWGxZTHEzSittMGczMVBiYzhIcVdHTVlKaGFkamRQK1RZVXc9BjsARg%3D%3D--0f372ba6b8e9956e504653078cc02c497684f04e; Hm_lvt_0cf76c77469e965d2957f0553e6ecf59=1539521254,1539521263,1539567080,1539577822; Hm_lpvt_0cf76c77469e965d2957f0553e6ecf59=1539578075'
}
html = requests.get(url,headers=headers)
soup = BeautifulSoup(html.text)
ips = soup.find_all('tr')
result_ls = []
for i in range(2,len(ips)):
ip_info = ips[i]
tds = ip_info.find_all('td')
if len(tds) > 0:
result_ls.append(tds[1].text+':'+tds[2].text)
else:
continue
return result_ls
ips = get_ip()
print(ips)
#存儲(chǔ)股票id#
stocks = {
'code':[],
'stockName':[],
'exchange':[]
}
def get_uid():
#設(shè)置url_1獲取投資者的id#
url = 'https://xueqiu.com/recommend/user/industry.json?id=47&_=1555121107441'
#設(shè)置請(qǐng)求頭文件#
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36',
'Cookie': cookies,
'Referer': 'https://xueqiu.com/people'
}
#獲取userid#
user_id=[]
#發(fā)送請(qǐng)求#
print('開(kāi)始發(fā)送請(qǐng)求...')
response = requests.get(url,headers=headers)
print("響應(yīng)碼:"+str(response.status_code))
if response.status_code == 400:
pass
#print(response.text)
else:
text = json.loads(response.text)
users = text['industries'][0]['users']
for i in range(len(users)):
user_id.append(users[i]['id'])
print(user_id)
return user_id
def result(user_id,ip):
#根據(jù)獲取的uid分別爬取用戶的股票信息#
url2 ='https://xueqiu.com/v4/stock/portfolio/stocks.json'
user_agent_list=[ "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 "
"(KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1",
"Mozilla/5.0 (X11; CrOS i686 2268.111.0) AppleWebKit/536.11 "
"(KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11",
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.6 "
"(KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6",
"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.6 "
"(KHTML, like Gecko) Chrome/20.0.1090.0 Safari/536.6",
"Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.1 "
"(KHTML, like Gecko) Chrome/19.77.34.5 Safari/537.1",
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.5 "
"(KHTML, like Gecko) Chrome/19.0.1084.9 Safari/536.5",
"Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 "
"(KHTML, like Gecko) Chrome/19.0.1084.36 Safari/536.5",
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 "
"(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
"Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.3 "
"(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_0) AppleWebKit/536.3 "
"(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 "
"(KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 "
"(KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",
"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 "
"(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 "
"(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
"Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.3 "
"(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
"Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 "
"(KHTML, like Gecko) Chrome/19.0.1061.0 Safari/536.3",
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.24 "
"(KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24",
"Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 "
"(KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24"]
#設(shè)置參數(shù)#
params = {
'size':10000,
'type':1,
'pid':-1,
'category':2,
'uid':str(user_id)
}
#請(qǐng)求頭文件#
headers2 = {
'User-Agent':choice(user_agent_list),
'Referer': 'https://xueqiu.com/u/'+str(user_id),
'Connection': 'keep-alive',
'cache-control':'no-cache',
'Cookie':cookies
}
#設(shè)置代理#
proxies = {'http':ip}
#發(fā)送請(qǐng)求#
response2 = requests.get(url2,headers=headers2,params=params,proxies=proxies)
time.sleep(5)
print("響應(yīng)碼:"+str(response2.status_code) + '使用ip:'+ip)
stockInfo = []
if response2.status_code == 400:
pass
else:
text2 = json.loads(response2.text)
stockInfo = text2['stocks']
return stockInfo
def main():
uid = get_uid()
for i in range(len(uid)):
print('采集第'+str(i)+'個(gè)數(shù)據(jù)')
ip = choice(ips)
user_stockInfo = result(uid[i],ip)
for j in range(len(user_stockInfo)):
stocks['code'].append(user_stockInfo[j]['code'])
stocks['stockName'].append(user_stockInfo[j]['stockName'])
stocks['exchange'].append(user_stockInfo[j]['exchange'])
stocks_df = pd.DataFrame(stocks)
stocks_df.to_csv('./stocksInfo.csv')
if __name__ =="__main__":
#main()
uid = get_uid()
for i in range(len(uid)):
print('采集第'+str(i)+'個(gè)數(shù)據(jù)')
ip = choice(ips)
user_stockInfo = result(uid[i],ip)a
for j in range(len(user_stockInfo)):
stocks['code'].append(user_stockInfo[j]['code'])
stocks['stockName'].append(user_stockInfo[j]['stockName'])
stocks['exchange'].append(user_stockInfo[j]['exchange'])
stocks_df = pd.DataFrame(stocks)
stocks_df.to_csv('./stocksInfo.csv')