liepin.jpg

首先自我批評(píng)一下薪缆，上周出差沒(méi)有更新博客烛缔，本周繼續(xù)

上周突發(fā)奇想怒竿，想看看當(dāng)前數(shù)據(jù)分析崗位的薪水情況。所以抓取了獵聘網(wǎng)20170427關(guān)于數(shù)據(jù)分析的所有100頁(yè)內(nèi)容稳其。由于本人能力有限驶赏，整理筆記如下，以備后來(lái)翻閱既鞠。

數(shù)據(jù)抓取

數(shù)據(jù)抓取中遇到的困難

1. 本人爬蟲功底有限煤傍，看了幾頁(yè)就照葫蘆畫瓢，代碼能力不好
2. 獵聘網(wǎng)的安全防護(hù)阻止我100次循環(huán)連續(xù)抓取內(nèi)容嘱蛋，所以只能連續(xù)20幾頁(yè)的抓取
3. 本打算獲取技能要求的內(nèi)容蚯姆，但由于是動(dòng)態(tài)加載，再加上時(shí)間有限洒敏，就沒(méi)有過(guò)多的研究龄恋。所以在本章中沒(méi)有實(shí)現(xiàn)抓取技能要求的內(nèi)容。

爬蟲代碼

# -*- coding:utf-8 -*-
#__author__ = 'ecaoyng'

import urllib.request
import re
import csv
import time
import codecs

class LiePinCrawler:
    def __init__(self):
        self.user_agent = 'Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Mobile Safari/537.36'
        self.headers = {'User-Agent': self.user_agent}
        self.jobs=[]
        self.csv='C:/Users/ecaoyng/Desktop/work space/job.csv'

    # 獲取搜索結(jié)果的某一頁(yè)
    def getPage(self, pageIndex):
        try:
            url = 'https://www.liepin.com/zhaopin/?pubTime=&ckid=757c5403caae67a7&fromSearchBtn=2&compkind=&isAnalysis=&init=-1&searchType=1&dqs=&industryType=&jobKind=&sortFlag=15&industries=&salary=&compscale=&key=%E6%95%B0%E6%8D%AE%E5%88%86%E6%9E%90&clean_condition=&headckid=757c5403caae67a7&curPage='
            url=url + str(pageIndex)
            # print(url)
            request = urllib.request.Request(url,headers = self.headers)
            response = urllib.request.urlopen(request)
            pageCode = response.read().decode('utf-8')
            print (pageCode)
            print('='*60)
            return pageCode
        except urllib.request.URLError as e:
            if hasattr(e, "reason"):
                print(u"Errors during connect to LiePin:", e.reason)
                return None
    #獲取職位詳細(xì)描述
    def getDesPage(self, contentURL):
        contentURL='https://www.liepin.com/job/198467615.shtml?imscid=R000000075&ckid=4704ed8e99af8c70&headckid=4704ed8e99af8c70&pageNo=0&pageIdx1&totalIdx=1&sup=1'
        print('contentURL is %s' % contentURL)
        request = urllib.request.Request(contentURL, headers=self.headers)
        response = urllib.request.urlopen(request)
        desPage = response.read().decode('utf-8')
        print(desPage)
        return desPage


    #加載一頁(yè)并過(guò)濾需要的內(nèi)容
    def getItems(self,pageIndex):
        pageCode=self.getPage(pageIndex)
        if not pageCode:
            print('頁(yè)面加載失敗...')
            return None
        pattern = re.compile(
            '<div class="job-info">.*?<h3 title.*?>.*?<a href="(.*?)".*?>(.*?)</a>.*?</h3>.*?<span class="text-warning">(.*?)</span>.*?<a href.*?>(.*?)</a>.*?<span class.*?>(.*?)</span>.*?<span>(.*?)</span>.*?<a title=.*?>(.*?)</a>.*?<a class.*?>(.*?)</a>.*?<p class=.*?>(.*?)</p>',
            re.S)
        items = re.findall(pattern, pageCode)


        for item in items:
            # contentPage=self.getDesPage(item[0].strip())
            # print(contentPage)
            break
            temp=re.sub('\s+','',item[8].strip())
            temp=re.sub('</span>','',temp)
            words=re.split(r'<span>',temp)
            self.jobs.append([item[0].strip(),item[1].strip(),item[2].strip(),item[3].strip(),item[4].strip(),item[5].strip(),item[6].strip(),item[7].strip(),words])
            print(self.jobs)
        return self.jobs



    #每次讀取一頁(yè)內(nèi)容并寫入文件
    def writeCSV(self):

        # for i in self.jobs:
        #     print(i)

        with open(self.csv, 'a+',encoding='utf-8') as csvFile:
            # csvFile.write(codecs.BOM_UTF8)
            csvwriter = csv.writer(csvFile, dialect=("excel"))
            # csvwriter.writerow(['Jobs','Salary','City','Edu','Years','Company','Industry','Key words'])
            for row in self.jobs:
                print(row)
                csvwriter.writerow(row)

        self.jobs=[]


if __name__== '__main__':

    liepin=LiePinCrawler()
    for i in range(0,20): # 可以修改此處凶伙，每次抓取指定頁(yè)的內(nèi)容
        time.sleep(5)
        liepin.getItems(i)
        #由于獵聘封號(hào)郭毕，所以只能讀一頁(yè)寫一頁(yè)
        liepin.writeCSV()

抓取的文件

由于不知該如何上傳，所以無(wú)法展示供大家下載

抓取的文件放到excel中顯示

由于抓取的文件是uft-8格式函荣，放在excel中顯示的中文亂碼显押，但是在utraledit和notepad中可以正常打開扳肛。這是excel的問(wèn)題，可以不用理睬乘碑，但如果非想用excel打開的話挖息，解決方法如下.

1. 將excel后綴名改為txt 
2. 打開Excel-> Data->Get Data From Text
3. 打開txt，根據(jù)需求選擇并next

數(shù)據(jù)清洗

數(shù)據(jù)分析中占用時(shí)間最多的就是數(shù)據(jù)清洗蝉仇。
原始數(shù)據(jù)的特點(diǎn)：

1. 在年薪payment中會(huì)出現(xiàn)面議旋讹，保密等字樣
2. 在年薪payment中出現(xiàn)13-20萬(wàn)這樣的區(qū)間殖蚕，不利于分析
3. 會(huì)有重復(fù)數(shù)據(jù)
5. 在place中有一行數(shù)據(jù)為空
4. 在地域中會(huì)出現(xiàn)類似北京轿衔、上海、深圳 這樣的數(shù)據(jù)

針對(duì)如上的問(wèn)題睦疫，解決思路如下:

1. 將保密替換為面議方便處理
2. 將一個(gè)13-20萬(wàn)這樣的字段害驹，拆分成三個(gè)income_min, income_max并計(jì)算income_avg
3. 丟棄掉重復(fù)數(shù)據(jù)
4. 在替他條件都整理好的情況下，將一條北京蛤育、上海宛官、深圳這樣的row，復(fù)制成三條數(shù)據(jù)并刪除原有的行

下面來(lái)看下原始數(shù)據(jù)的情況

import pandas as pd
srcFile='C:\\Users\\ecaoyng\\Desktop\\work space\\job.csv'
#加入header=None是為了不把第一行數(shù)據(jù)方做header
liepin=pd.read_csv(srcFile,header=None)
liepin.columns=['jobs','payment','place','qualifications','experience','company','area','walfare']
# liepin.columns=['職位','年薪','工作地點(diǎn)','學(xué)歷','工作年限','公司','行業(yè)','公司福利']
liepin.head()

liepin.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3939 entries, 0 to 3938
Data columns (total 8 columns):
jobs              3939 non-null object
payment           3939 non-null object
place             3938 non-null object
qualifications    3939 non-null object
experience        3939 non-null object
company           3939 non-null object
area              3939 non-null object
walfare           3939 non-null object
dtypes: object(8)
memory usage: 246.3+ KB

#丟棄重復(fù)數(shù)據(jù)
liepin.drop_duplicates()

發(fā)現(xiàn)1646行的place為空瓦糕，經(jīng)查實(shí)應(yīng)該是成都底洗，填充上

#發(fā)現(xiàn)異常值在1646行
liepin.iloc[1646,2:3]='成都'
liepin.iloc[1646,:]

發(fā)現(xiàn)place中有類似上海-浦東區(qū)這樣的內(nèi)容，同意將其過(guò)濾為上海并添加一個(gè)城市字段

# 將city標(biāo)準(zhǔn)化
city=[]

for x in liepin.place:
    print(x)
    if '-' in x:
        x=x[0:x.index('-')]
    city.append(x)

liepin['City']=city
liepin.head(3)

按照之前所講會(huì)出現(xiàn)多地址的問(wèn)題

liepin.iloc[923,:]

Out[322]:
jobs                                                數(shù)據(jù)分析P6 p7 p8 p9
payment                                                      40-70萬(wàn)
place                                                     上海,北京,浙江省
qualifications                                                本科或以上
experience                                                   3年工作經(jīng)驗(yàn)
company                                                   國(guó)內(nèi)最大互聯(lián)網(wǎng)企業(yè)
area                                                 互聯(lián)網(wǎng)/移動(dòng)互聯(lián)網(wǎng)/電子商務(wù)
walfare           ['<iclass="icons24icons24-honesty"></i><em>該職位...
City                                                      上海,北京,浙江省
Name: 923, dtype: object

另外咕娄，本打算丟棄掉面議的row亥揖，但無(wú)奈太多，不能丟棄

liepin[liepin.payment=='面議'].info()

<class 'pandas.core.frame.DataFrame'>
Int64Index: 907 entries, 1 to 3938
Data columns (total 9 columns):
jobs              907 non-null object
payment           907 non-null object
place             907 non-null object
qualifications    907 non-null object
experience        907 non-null object
company           907 non-null object
area              907 non-null object
walfare           907 non-null object
City              907 non-null object
dtypes: object(9)
memory usage: 70.9+ KB

為了對(duì)薪資進(jìn)行處理圣勒，根據(jù)年薪字段费变，增加三個(gè)薪水字段，分別為最低年薪圣贸，最高年薪挚歧，平均年薪

#新建三個(gè)地段來(lái)描述薪資情況
import re

income_avg=[]
income_min=[]
income_max=[]
for i in liepin.payment:
    if i.strip()== '面議':
        income_avg.append('面議')
        income_min.append('面議')
        income_max.append('面議')
    elif(i.strip()=='保密'):
        # 數(shù)據(jù)中有一行的薪水是保密，將其改為面議
        income_avg.append('面議')
        income_min.append('面議')
        income_max.append('面議')
    else:
        pattern=r'(\d+)-(\d+).*'
        regex=re.compile(pattern)
        m=regex.match(i)
        income_min.append(m.group(1))
        income_max.append(m.group(2))
        int_min=int(m.group(1))
        int_max=int(m.group(2))
        
        income_avg.append(format((int_min+int_max)/2,'.2f'))
        
liepin['income_min']=income_min
liepin['income_max']=income_max
liepin['income_avg']=income_avg

增加字段后的表結(jié)構(gòu)如下

現(xiàn)在表的整體信息是:

liepin.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3939 entries, 0 to 3938
Data columns (total 12 columns):
jobs              3939 non-null object
payment           3939 non-null object
place             3939 non-null object
qualifications    3939 non-null object
experience        3939 non-null object
company           3939 non-null object
area              3939 non-null object
walfare           3939 non-null object
City              3939 non-null object
income_min        3939 non-null object
income_max        3939 non-null object
income_avg        3939 non-null object
dtypes: object(12)
memory usage: 369.4+ KB

對(duì)多地址行進(jìn)行復(fù)制

# 顯示多地址行
liepin[liepin.place.str.contains(',')].head(2)

#一共159條數(shù)據(jù),無(wú)法忽略
liepin[liepin.place.str.contains(',')].info()

#獲取一行dataframe一行數(shù)據(jù)
liepin.iloc[[826]]

# 將一行數(shù)據(jù)拆分成三行
for i in liepin[liepin.place.str.contains(',')].index:
    for city in str(liepin.iloc[i].place).split(','):

        row=pd.DataFrame([dict(jobs=liepin.iloc[i].jobs,
        payment=liepin.iloc[i].payment,
        place=city, 
        qualifications=liepin.iloc[i].qualifications,
        experience=liepin.iloc[i].experience,
        company=liepin.iloc[i].company,
        area=liepin.iloc[i].area,
        walfare=liepin.iloc[i].walfare,
        City=city,
        income_min=liepin.iloc[i].income_min, 
        income_max=liepin.iloc[i].income_max,
        income_avg=liepin.iloc[i].income_avg,
        )])
        liepin=liepin.append(row,ignore_index=True)
    
liepin.tail()

#丟棄重復(fù)數(shù)據(jù)
liepin.drop_duplicates()
# 查看多地址行吁峻，一共159行
liepin[liepin.place.str.contains(',')].index.size

#丟棄掉多地址字段
for i in liepin[liepin.place.str.contains(',')].index:
      liepin.drop(i,inplace=True)

liepin.info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 4208 entries, 0 to 4366
Data columns (total 12 columns):
City              4208 non-null object
area              4208 non-null object
company           4208 non-null object
experience        4208 non-null object
income_avg        4208 non-null object
income_max        4208 non-null object
income_min        4208 non-null object
jobs              4208 non-null object
payment           4208 non-null object
place             4208 non-null object
qualifications    4208 non-null object
walfare           4208 non-null object
dtypes: object(12)
memory usage: 427.4+ KB

#check發(fā)現(xiàn)已經(jīng)沒(méi)有多地址的行
liepin[liepin.place.str.contains(',')].index.size # 0

#將其寫入csv文件
targetDir='C:\\Users\\ecaoyng\\Downloads\\jobs_clean.csv'
liepin.to_csv(targetDir)

#數(shù)據(jù)清洗完成

將在下一篇中記錄分析過(guò)程滑负。
最后感謝 Tiger、DrFish 和成元