爬取雪球網(wǎng)優(yōu)化之代理池

在之前的文章中攒磨,提到如何爬取雪球網(wǎng)用戶的股票數(shù)據(jù),但是由于爬取過(guò)程中汤徽,會(huì)存在一些問(wèn)題娩缰,比如由于網(wǎng)站設(shè)有反爬蟲(chóng)機(jī)制,所以會(huì)導(dǎo)致在爬取過(guò)程中收到403的response谒府,因此在爬取的時(shí)候需要做一些偽裝拼坎,首先要做的就是建立一個(gè)代理ip池,通過(guò)代理ip對(duì)網(wǎng)站進(jìn)行訪問(wèn)完疫,從而防止由于單一ip短時(shí)間內(nèi)頻繁訪問(wèn)而被網(wǎng)站給ban掉泰鸡。

但是要獲取穩(wěn)定又多的ip,而且實(shí)現(xiàn)每次的ip更新壳鹤,因此我們要先寫一個(gè)爬蟲(chóng)盛龄,把代理ip網(wǎng)站的ip爬下來(lái),然后封裝進(jìn)我們的ip池里芳誓。

這里我選擇的是西刺代理余舶,大家也可以自行選擇其他的代理ip網(wǎng)站。

西刺代理頁(yè)面截圖

以上是西刺代理的頁(yè)面截圖锹淌,頁(yè)面相對(duì)規(guī)整匿值,首先進(jìn)行測(cè)試,發(fā)現(xiàn)該網(wǎng)站沒(méi)有采用json的傳參方式赂摆,但是由于頁(yè)面十分的規(guī)整挟憔,因此這里考慮直接對(duì)html頁(yè)面進(jìn)行爬取,首先先看一下頁(yè)面的html源碼库正。


頁(yè)面部分html源碼

通過(guò)以上我們可以發(fā)現(xiàn)曲楚,頁(yè)面相對(duì)而言較為規(guī)整,每個(gè)tr下面就包含了一樣數(shù)量的td褥符,且龙誊,每個(gè)tr里的第二個(gè)td和第三個(gè)td以及倒數(shù)第三個(gè)td內(nèi)包含了我們所需要的ip,端口以及ip類型喷楣。

接下來(lái)就直接上代碼了趟大。

#設(shè)置ip池#
def get_ip():
    url = 'http://www.xicidaili.com/'
    
    headers = {    
                 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
                 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36',
                 'Cookie':'_free_proxy_session=BAh7B0kiD3Nlc3Npb25faWQGOgZFVEkiJTI3MGZjZTNkM2I2NWNhY2M5MWZkYmY0NDhkMGU4MTExBjsAVEkiEF9jc3JmX3Rva2VuBjsARkkiMXhjRldkci9BWGxZTHEzSittMGczMVBiYzhIcVdHTVlKaGFkamRQK1RZVXc9BjsARg%3D%3D--0f372ba6b8e9956e504653078cc02c497684f04e; Hm_lvt_0cf76c77469e965d2957f0553e6ecf59=1539521254,1539521263,1539567080,1539577822; Hm_lpvt_0cf76c77469e965d2957f0553e6ecf59=1539578075'
                 }
    
    html = requests.get(url,headers=headers)
    
    soup = BeautifulSoup(html.text)
    ips = soup.find_all('tr')
    result_ls = []
    
    for i in range(2,len(ips)):
        ip_info = ips[i]
        tds = ip_info.find_all('td')
        if len(tds) > 0:
            result_ls.append(tds[1].text+':'+tds[2].text)
        else:
            continue
    
    return result_ls

首先爬取頁(yè)面上所有的td,將里面的第一個(gè)與第二個(gè)td取出铣焊,并進(jìn)行拼接逊朽,然后再把結(jié)果存入到數(shù)組中,最后返回result_ls這個(gè)數(shù)組作為ip池曲伊。

def result(user_id,ip):
    #根據(jù)獲取的uid分別爬取用戶的股票信息#
    url2 ='https://xueqiu.com/v4/stock/portfolio/stocks.json'
        
    user_agent_list=[        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 "  
        "(KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1",  
        "Mozilla/5.0 (X11; CrOS i686 2268.111.0) AppleWebKit/536.11 "  
        "(KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11",  
        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.6 "  
        "(KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6",  
        "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.6 "  
        "(KHTML, like Gecko) Chrome/20.0.1090.0 Safari/536.6",  
        "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.1 "  
        "(KHTML, like Gecko) Chrome/19.77.34.5 Safari/537.1",  
        "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.5 "  
        "(KHTML, like Gecko) Chrome/19.0.1084.9 Safari/536.5",  
        "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 "  
        "(KHTML, like Gecko) Chrome/19.0.1084.36 Safari/536.5",  
        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 "  
        "(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",  
        "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.3 "  
        "(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",  
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_0) AppleWebKit/536.3 "  
        "(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",  
        "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 "  
        "(KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",  
        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 "  
        "(KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",  
        "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 "  
        "(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",  
        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 "  
        "(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",  
        "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.3 "  
        "(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",  
        "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 "  
        "(KHTML, like Gecko) Chrome/19.0.1061.0 Safari/536.3",  
        "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.24 "  
        "(KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24",  
        "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 "  
        "(KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24"]
    #設(shè)置參數(shù)#
    params = {
            'size':10000,
            'type':1,
            'pid':-1,
            'category':2,
            'uid':str(user_id)
            }
    #請(qǐng)求頭文件#
    headers2 = {
                'User-Agent':choice(user_agent_list),
                'Referer': 'https://xueqiu.com/u/'+str(user_id),
                'Connection': 'keep-alive',
                'cache-control':'no-cache',
                'Cookie':cookies
            }

    #設(shè)置代理#
    proxies = {'http':ip}
    #發(fā)送請(qǐng)求#
    response2 = requests.get(url2,headers=headers2,params=params,proxies=proxies)  
    time.sleep(5)
    print("響應(yīng)碼:"+str(response2.status_code) + '使用ip:'+ip)
    stockInfo = []
    if response2.status_code == 400:
        pass
    else:
        text2 = json.loads(response2.text)
    
        stockInfo = text2['stocks']
    
    return stockInfo

之后再?gòu)膇p池中選取一個(gè)ip作為參數(shù)傳入到代理中叽讳。

最后上完整代碼追他,有問(wèn)題或者改進(jìn)意見(jiàn)歡迎私信共同探討。

# -*- coding: utf-8 -*-
"""
Created on Sat Oct 13 18:33:35 2018
@author: Chen.Xiao
"""

import requests
import json
import pandas as pd
import numpy as np
import time
import ssl
from random import choice
from bs4 import BeautifulSoup
#防止ssl報(bào)錯(cuò)#
ssl._create_default_https_context = ssl._create_unverified_context
cookies = '_ga=GA1.2.1962796314.1538825138; device_id=516f0c95e19bf48ba37ca2f3e68811ed; __utmz=1.1538825152.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); s=fi17cigo1a; bid=5ab5c3e86b4b2c95f32bb04bd50357c3_jn791euz; aliyungf_tc=AQAAAMdnpDWO5QEA/c5CMbWjR/czsdk4; Hm_lvt_1db88642e346389874251b5a1eded6e3=1554460329; __utmc=1; xq_a_token=3450822dc3b6c0b631c3ba4768fcddac23c054d7; xq_a_token.sig=f1ZlcbP6BFkUA32I1cexLDa2KTk; xq_r_token=0de8d3b6155ce156310ff6d4e214d4532198ccec; xq_r_token.sig=8xgxBORSc2oawdjl5r3ksadcr9s; _gid=GA1.2.684281849.1555120269; u=231555120270276; _gat=1; __utma=1.1962796314.1538825138.1554640094.1555121068.23; __utmt=1; __utmb=1.2.10.1555121068; Hm_lpvt_1db88642e346389874251b5a1eded6e3=1555121080'
    
#設(shè)置ip池#
def get_ip():
    url = 'http://www.xicidaili.com/'
    
    headers = {    
                 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
                 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36',
                 'Cookie':'_free_proxy_session=BAh7B0kiD3Nlc3Npb25faWQGOgZFVEkiJTI3MGZjZTNkM2I2NWNhY2M5MWZkYmY0NDhkMGU4MTExBjsAVEkiEF9jc3JmX3Rva2VuBjsARkkiMXhjRldkci9BWGxZTHEzSittMGczMVBiYzhIcVdHTVlKaGFkamRQK1RZVXc9BjsARg%3D%3D--0f372ba6b8e9956e504653078cc02c497684f04e; Hm_lvt_0cf76c77469e965d2957f0553e6ecf59=1539521254,1539521263,1539567080,1539577822; Hm_lpvt_0cf76c77469e965d2957f0553e6ecf59=1539578075'
                 }
    
    html = requests.get(url,headers=headers)
    
    soup = BeautifulSoup(html.text)
    ips = soup.find_all('tr')
    result_ls = []
    
    for i in range(2,len(ips)):
        ip_info = ips[i]
        tds = ip_info.find_all('td')
        if len(tds) > 0:
            result_ls.append(tds[1].text+':'+tds[2].text)
        else:
            continue
    
    return result_ls
ips = get_ip()
print(ips)
#存儲(chǔ)股票id#
stocks = {
            'code':[],
            'stockName':[],
            'exchange':[]
       
        }
def get_uid():
    #設(shè)置url_1獲取投資者的id#
    url = 'https://xueqiu.com/recommend/user/industry.json?id=47&_=1555121107441'
    #設(shè)置請(qǐng)求頭文件#
    headers = {
             'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36',
             'Cookie': cookies,
             'Referer': 'https://xueqiu.com/people'
            }
    #獲取userid#
    user_id=[]
    #發(fā)送請(qǐng)求#
    print('開(kāi)始發(fā)送請(qǐng)求...')
    response = requests.get(url,headers=headers)
    print("響應(yīng)碼:"+str(response.status_code))
    if response.status_code == 400:
        pass
    #print(response.text)
    else:
        text = json.loads(response.text)
        users = text['industries'][0]['users']

        for i in range(len(users)):
            user_id.append(users[i]['id'])
        print(user_id)
    return user_id

def result(user_id,ip):
    #根據(jù)獲取的uid分別爬取用戶的股票信息#
    url2 ='https://xueqiu.com/v4/stock/portfolio/stocks.json'
        
    user_agent_list=[        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 "  
        "(KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1",  
        "Mozilla/5.0 (X11; CrOS i686 2268.111.0) AppleWebKit/536.11 "  
        "(KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11",  
        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.6 "  
        "(KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6",  
        "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.6 "  
        "(KHTML, like Gecko) Chrome/20.0.1090.0 Safari/536.6",  
        "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.1 "  
        "(KHTML, like Gecko) Chrome/19.77.34.5 Safari/537.1",  
        "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.5 "  
        "(KHTML, like Gecko) Chrome/19.0.1084.9 Safari/536.5",  
        "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 "  
        "(KHTML, like Gecko) Chrome/19.0.1084.36 Safari/536.5",  
        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 "  
        "(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",  
        "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.3 "  
        "(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",  
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_0) AppleWebKit/536.3 "  
        "(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",  
        "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 "  
        "(KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",  
        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 "  
        "(KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",  
        "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 "  
        "(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",  
        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 "  
        "(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",  
        "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.3 "  
        "(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",  
        "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 "  
        "(KHTML, like Gecko) Chrome/19.0.1061.0 Safari/536.3",  
        "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.24 "  
        "(KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24",  
        "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 "  
        "(KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24"]
    #設(shè)置參數(shù)#
    params = {
            'size':10000,
            'type':1,
            'pid':-1,
            'category':2,
            'uid':str(user_id)
            }
    #請(qǐng)求頭文件#
    headers2 = {
                'User-Agent':choice(user_agent_list),
                'Referer': 'https://xueqiu.com/u/'+str(user_id),
                'Connection': 'keep-alive',
                'cache-control':'no-cache',
                'Cookie':cookies
            }

    #設(shè)置代理#
    proxies = {'http':ip}
    #發(fā)送請(qǐng)求#
    response2 = requests.get(url2,headers=headers2,params=params,proxies=proxies)  
    time.sleep(5)
    print("響應(yīng)碼:"+str(response2.status_code) + '使用ip:'+ip)
    stockInfo = []
    if response2.status_code == 400:
        pass
    else:
        text2 = json.loads(response2.text)
    
        stockInfo = text2['stocks']
    
    return stockInfo

def main():
    uid = get_uid()
    for i in range(len(uid)):    
        print('采集第'+str(i)+'個(gè)數(shù)據(jù)')
        ip = choice(ips)
        user_stockInfo = result(uid[i],ip)
        for j in range(len(user_stockInfo)):
            stocks['code'].append(user_stockInfo[j]['code'])
            stocks['stockName'].append(user_stockInfo[j]['stockName'])
            stocks['exchange'].append(user_stockInfo[j]['exchange'])
    
    stocks_df = pd.DataFrame(stocks)
    stocks_df.to_csv('./stocksInfo.csv')


if __name__ =="__main__":
    #main()
    uid = get_uid()
    for i in range(len(uid)):    
        print('采集第'+str(i)+'個(gè)數(shù)據(jù)')
        ip = choice(ips)
        user_stockInfo = result(uid[i],ip)a
        for j in range(len(user_stockInfo)):
            stocks['code'].append(user_stockInfo[j]['code'])
            stocks['stockName'].append(user_stockInfo[j]['stockName'])
            stocks['exchange'].append(user_stockInfo[j]['exchange'])
    
    stocks_df = pd.DataFrame(stocks)
    stocks_df.to_csv('./stocksInfo.csv')


?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請(qǐng)聯(lián)系作者
  • 序言:七十年代末岛蚤,一起剝皮案震驚了整個(gè)濱河市邑狸,隨后出現(xiàn)的幾起案子,更是在濱河造成了極大的恐慌涤妒,老刑警劉巖单雾,帶你破解...
    沈念sama閱讀 206,126評(píng)論 6 481
  • 序言:濱河連續(xù)發(fā)生了三起死亡事件,死亡現(xiàn)場(chǎng)離奇詭異她紫,居然都是意外死亡硅堆,警方通過(guò)查閱死者的電腦和手機(jī),發(fā)現(xiàn)死者居然都...
    沈念sama閱讀 88,254評(píng)論 2 382
  • 文/潘曉璐 我一進(jìn)店門贿讹,熙熙樓的掌柜王于貴愁眉苦臉地迎上來(lái)渐逃,“玉大人,你說(shuō)我怎么就攤上這事围详∑庸裕” “怎么了?”我有些...
    開(kāi)封第一講書(shū)人閱讀 152,445評(píng)論 0 341
  • 文/不壞的土叔 我叫張陵助赞,是天一觀的道長(zhǎng)。 經(jīng)常有香客問(wèn)我袁勺,道長(zhǎng)雹食,這世上最難降的妖魔是什么? 我笑而不...
    開(kāi)封第一講書(shū)人閱讀 55,185評(píng)論 1 278
  • 正文 為了忘掉前任期丰,我火速辦了婚禮群叶,結(jié)果婚禮上,老公的妹妹穿的比我還像新娘钝荡。我一直安慰自己街立,他們只是感情好,可當(dāng)我...
    茶點(diǎn)故事閱讀 64,178評(píng)論 5 371
  • 文/花漫 我一把揭開(kāi)白布埠通。 她就那樣靜靜地躺著赎离,像睡著了一般。 火紅的嫁衣襯著肌膚如雪端辱。 梳的紋絲不亂的頭發(fā)上梁剔,一...
    開(kāi)封第一講書(shū)人閱讀 48,970評(píng)論 1 284
  • 那天,我揣著相機(jī)與錄音舞蔽,去河邊找鬼荣病。 笑死,一個(gè)胖子當(dāng)著我的面吹牛渗柿,可吹牛的內(nèi)容都是我干的个盆。 我是一名探鬼主播,決...
    沈念sama閱讀 38,276評(píng)論 3 399
  • 文/蒼蘭香墨 我猛地睜開(kāi)眼,長(zhǎng)吁一口氣:“原來(lái)是場(chǎng)噩夢(mèng)啊……” “哼颊亮!你這毒婦竟也來(lái)了鸡岗?” 一聲冷哼從身側(cè)響起,我...
    開(kāi)封第一講書(shū)人閱讀 36,927評(píng)論 0 259
  • 序言:老撾萬(wàn)榮一對(duì)情侶失蹤编兄,失蹤者是張志新(化名)和其女友劉穎轩性,沒(méi)想到半個(gè)月后,有當(dāng)?shù)厝嗽跇?shù)林里發(fā)現(xiàn)了一具尸體狠鸳,經(jīng)...
    沈念sama閱讀 43,400評(píng)論 1 300
  • 正文 獨(dú)居荒郊野嶺守林人離奇死亡揣苏,尸身上長(zhǎng)有42處帶血的膿包…… 初始之章·張勛 以下內(nèi)容為張勛視角 年9月15日...
    茶點(diǎn)故事閱讀 35,883評(píng)論 2 323
  • 正文 我和宋清朗相戀三年,在試婚紗的時(shí)候發(fā)現(xiàn)自己被綠了件舵。 大學(xué)時(shí)的朋友給我發(fā)了我未婚夫和他白月光在一起吃飯的照片卸察。...
    茶點(diǎn)故事閱讀 37,997評(píng)論 1 333
  • 序言:一個(gè)原本活蹦亂跳的男人離奇死亡,死狀恐怖铅祸,靈堂內(nèi)的尸體忽然破棺而出坑质,到底是詐尸還是另有隱情,我是刑警寧澤临梗,帶...
    沈念sama閱讀 33,646評(píng)論 4 322
  • 正文 年R本政府宣布涡扼,位于F島的核電站,受9級(jí)特大地震影響盟庞,放射性物質(zhì)發(fā)生泄漏吃沪。R本人自食惡果不足惜,卻給世界環(huán)境...
    茶點(diǎn)故事閱讀 39,213評(píng)論 3 307
  • 文/蒙蒙 一什猖、第九天 我趴在偏房一處隱蔽的房頂上張望票彪。 院中可真熱鬧,春花似錦不狮、人聲如沸降铸。這莊子的主人今日做“春日...
    開(kāi)封第一講書(shū)人閱讀 30,204評(píng)論 0 19
  • 文/蒼蘭香墨 我抬頭看了看天上的太陽(yáng)推掸。三九已至,卻和暖如春遂黍,著一層夾襖步出監(jiān)牢的瞬間终佛,已是汗流浹背。 一陣腳步聲響...
    開(kāi)封第一講書(shū)人閱讀 31,423評(píng)論 1 260
  • 我被黑心中介騙來(lái)泰國(guó)打工雾家, 沒(méi)想到剛下飛機(jī)就差點(diǎn)兒被人妖公主榨干…… 1. 我叫王不留铃彰,地道東北人。 一個(gè)月前我還...
    沈念sama閱讀 45,423評(píng)論 2 352
  • 正文 我出身青樓芯咧,卻偏偏與公主長(zhǎng)得像牙捉,于是被迫代替她去往敵國(guó)和親竹揍。 傳聞我的和親對(duì)象是個(gè)殘疾皇子,可洞房花燭夜當(dāng)晚...
    茶點(diǎn)故事閱讀 42,722評(píng)論 2 345

推薦閱讀更多精彩內(nèi)容