要獲取什么樣的數(shù)據(jù)?
我們要獲取的數(shù)據(jù),是指那些公開的,可以輕易地獲取地?cái)?shù)據(jù).如果你有完整的數(shù)據(jù)集,肯定是極好的,但一般都很難通過還算正當(dāng)?shù)姆绞捷p易獲取.單就本系列文章要研究的實(shí)時(shí)招聘信息來講,能獲取最近一個(gè)月的相關(guān)信息,已是足矣.
如何獲取數(shù)據(jù)?
爬蟲,也是可以的,作為一個(gè)備選方案.但是,我注意到拉勾網(wǎng)本身的數(shù)據(jù),是通過ajax請求更新的,所以批量獲取變得更加簡單.基于ajax請求來獲取數(shù)據(jù),方式有很多,這里我演示其中的自認(rèn)為較為簡單通用的一種: 使用 curl 模擬 ajax 請求獲取數(shù)據(jù).
注意,以下的步驟演示全部基于 Mac 版的 ** Google Chrome** 瀏覽器,其他瀏覽器部分操作可能會(huì)有些許差異.最后一步會(huì)給出 提取出的通用 curl 腳本,直接其實(shí)也是可以的,如果對步驟不是很關(guān)心.
1.找到目標(biāo)城市和目標(biāo)職位,然后按"最新排序",參考鏈接: http://www.lagou.com/jobs/list_iOS?px=new&city=北京#order
0-0.png
2.雙指擊/右擊 頁面,彈出快捷菜單,選擇"檢查",以進(jìn)入瀏覽器調(diào)試界面,切換到調(diào)試器的 network -> xhr 標(biāo)簽下.
0_1.png
3.cmd + R 刷新頁面,此時(shí)會(huì)捕捉到此頁面發(fā)出的xhr請求.找到 http://www.lagou.com/jobs/positionAjax.json 開頭的請求,并雙指擊/右擊,選擇 copy as cUrl.
這個(gè) curl代碼非常長,對于本次分析來說,最關(guān)鍵的是 末尾的 pn=1&kd=iOS,分別代表著頁面和職位,動(dòng)態(tài)設(shè)置,即可獲取更多職位的更多數(shù)據(jù)了,文章的其他篇幅,會(huì)單獨(dú)分析.
curl 'http://www.lagou.com/jobs/positionAjax.json?px=new&city=%E5%8C%97%E4%BA%AC&needAddtionalResult=false' -H 'Cookie: user_trace_token=20160522122749-8a4d6717-1fd5-11e6-963e-5254005c3644; LGUID=20160522122749-8a4d6cb3-1fd5-11e6-963e-5254005c3644; tencentSig=8513357824; LGMOID=20160818212815-33C56329AA2FB6D809D557FD6CC1DE3C; JSESSIONID=E0C5692414B160F3BEFFCFF8D240B693; _gat=1; PRE_UTM=; PRE_HOST=; PRE_SITE=; PRE_LAND=http%3A%2F%2Fwww.lagou.com%2Fjobs%2Flist_iOS%3Fpx%3Dnew%26city%3D%25E5%258C%2597%25E4%25BA%25AC; ctk=1472587366; Hm_lvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1470032014,1470032429,1471526897,1471752875; Hm_lpvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1472587368; LGSID=20160831035434-92fffed3-6eeb-11e6-8c02-5254005c3644; LGRID=20160831040247-b90762b6-6eec-11e6-a745-525400f775ce; _ga=GA1.2.1654849521.1463891270; SEARCH_ID=506c8097626343ab9b5a2c1139807a2c' -H 'Origin: http://www.lagou.com' -H 'X-Anit-Forge-Code: 0' -H 'Accept-Encoding: gzip, deflate' -H 'Accept-Language: zh-CN,zh;q=0.8,en;q=0.6' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36' -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Cache-Control: max-age=0' -H 'X-Requested-With: XMLHttpRequest' -H 'Connection: keep-alive' -H 'X-Anit-Forge-Token: None' -H 'Referer: http://www.lagou.com/jobs/list_iOS?px=new&city=%E5%8C%97%E4%BA%AC' --data 'first=true&pn=1&kd=iOS' --compressed
0_2.png
4.講上一步中的curl指令復(fù)制到終端,橋下回車鍵,即可看到輸出.
{"success":true,"requestId":null,"msg":null,"resubmitToken":null,"content":{"pageNo":1,"pageSize":15,"positionResult":{"totalCount":974,"resultSize":15,"locationInfo":{"city":"北京","district":null,"queryByGisCode":false,"businessZone":null,"locationCode":null},"queryAnalysisInfo":{"positionName":"ios","companyName":null,"usefulCompany":false,"industryName":null},"strategyProperty":{"name":"dm-csearch-newSimScorer","id":1},"result":[{"companyId":129801,"companyShortName":"言之有物科技","createTime":"2016-08-30 19:28:12","positionId":1857486,"positionAdvantage":"一線公司杈帐，技術(shù)驅(qū)動(dòng)典尾，免費(fèi)三餐疚颊，超期望回報(bào)","salary":"25k-50k","score":0,"workYear":"不限","education":"本科","city":"北京","positionName":"iOS高級研發(fā)工程師/Lead","companyLogo":"i/image/M00/43/4E/CgqKkVeDGsuAXz0gAAA4XeGAAHQ390.png","financeStage":"成長型(A輪)","industryField":"移動(dòng)互聯(lián)網(wǎng),電子商務(wù)","jobNature":"全職","approve":1,"companySize":"15-50人","district":null,"companyLabelList":["股票期權(quán)","扁平管理","美女多","領(lǐng)導(dǎo)好"],"adWord":0,"appShow":0,"deliver":0,"formatCreateTime":"19:28發(fā)布","gradeDescription":null,"companyFullName":"北京言之有物科技有限公司","businessZones":null,"imState":"today","lastLogin":1472556472000,"publisherId":5092848,"explain":null,"plus":null,"pcShow":0},{"companyId":133,"companyShortName":"獵豹移動(dòng)","createTime":"2016-08-30 19:09:34","positionId":2151896,"positionAdvantage":"明星產(chǎn)品 超贊年終獎(jiǎng) 靠譜領(lǐng)導(dǎo)","salary":"15k-30k","score":0,"workYear":"1-3年","education":"本科","city":"北京","positionName":"iOS","companyLogo":"image1/M00/39/70/CgYXBlWo3nqABJTsAADJ3hn5gmE062.jpg","financeStage":"上市公司","industryField":"移動(dòng)互聯(lián)網(wǎng),信息安全","jobNature":"全職","approve":1,"companySize":"500-2000人","district":"朝陽區(qū)","companyLabelList":["帶薪年假","美女前臺","超贊年終獎(jiǎng)","一公里工作圈"],"adWord":0,"appShow":0,"deliver":0,"formatCreateTime":"19:09發(fā)布","gradeDescription":null,"companyFullName":"北京金山網(wǎng)絡(luò)科技有限公司","businessZones":["姚家園","十里堡","高碑店"],"imState":"today","lastLogin":1472555392000,"publisherId":129969,"explain":null,"plus":null,"pcShow":0},{"companyId":107608,"companyShortName":"MUM計(jì)算機(jī)","createTime":"2016-08-30 19:03:24","positionId":1963945,"positionAdvantage":"幫助程序員赴美做IT，享受高薪高品質(zhì)生活","salary":"10k-20k","score":0,"workYear":"不限","education":"本科","city":"北京","positionName":"IOS程序員赴美項(xiàng)目推廣員","companyLogo":"i/image/M00/00/C2/CgqKkVZVHmSAWPtRAASUg0iUVuI932.jpg","financeStage":"初創(chuàng)型(不需要融資)","industryField":"教育","jobNature":"全職","approve":0,"companySize":"少于15人","district":"昌平區(qū)","companyLabelList":["赴美工作","美元薪水","告別996","技術(shù)前沿"],"adWord":0,"appShow":0,"deliver":0,"formatCreateTime":"19:03發(fā)布","gradeDescription":null,"companyFullName":"北京瑪赫西計(jì)算機(jī)教育咨詢有限公司","businessZones":null,"imState":"disabled","lastLogin":1472558059000,"publisherId":5179699,"explain":null,"plus":null,"pcShow":0},{"companyId":67576,"companyShortName":"車滿滿","createTime":"2016-08-30 18:47:30","positionId":2307877,"positionAdvantage":"期權(quán)","salary":"20k-25k","score":0,"workYear":"3-5年","education":"本科","city":"北京","positionName":"iOS高級開發(fā)工程師","companyLogo":"i/image/M00/01/47/Cgp3O1ZmYACABBpPAAGzVR5S-Ps906.png","financeStage":"成長型(A輪)","industryField":"移動(dòng)互聯(lián)網(wǎng)","jobNature":"全職","approve":1,"companySize":"50-150人","district":"朝陽區(qū)","companyLabelList":["股票期權(quán)","技能培訓(xùn)","彈性工作","定期體檢"],"adWord":0,"appShow":0,"deliver":0,"formatCreateTime":"18:47發(fā)布","gradeDescription":null,"companyFullName":"車滿滿（北京）信息技術(shù)有限公司","businessZones":["建外大街","CBD","國貿(mào)"],"imState":"today","lastLogin":1472566873000,"publisherId":2116322,"explain":null,"plus":null,"pcShow":0},{"companyId":1575,"companyShortName":"百度","createTime":"2016-08-30 18:30:05","positionId":2307765,"positionAdvantage":"BAT 薪酬福利好","salary":"15k-25k","score":0,"workYear":"3-5年","education":"本科","city":"北京","positionName":"iOS移動(dòng)開發(fā)","companyLogo":"image1/M00/00/06/CgYXBlTUWAWAOBXrAABGHHFb0q8748.jpg","financeStage":"上市公司","industryField":"移動(dòng)互聯(lián)網(wǎng),數(shù)據(jù)服務(wù)","jobNature":"全職","approve":1,"companySize":"2000人以上","district":null,"companyLabelList":["股票期權(quán)","彈性工作","五險(xiǎn)一金","免費(fèi)班車"],"adWord":0,"appShow":0,"deliver":0,"formatCreateTime":"18:30發(fā)布","gradeDescription":null,"companyFullName":"百度在線網(wǎng)絡(luò)技術(shù)（北京）有限公司","businessZones":null,"imState":"disabled","lastLogin":1472553001000,"publisherId":5705515,"explain":null,"plus":null,"pcShow":0},{"companyId":13321,"companyShortName":"FunPlus 趣加游戲","createTime":"2016-08-30 18:26:28","positionId":2240276,"positionAdvantage":"國際一線團(tuán)隊(duì)萎庭，無限的成長空間，任你發(fā)揮","salary":"18k-36k","score":0,"workYear":"5-10年","education":"本科","city":"北京","positionName":"iOS 視頻處理工程師/高級工程師","companyLogo":"image1/M00/00/1A/Cgo8PFTUWFWAKE5aAABwJ1mgAYw423.png","financeStage":"成長型(B輪)","industryField":"游戲","jobNature":"全職","approve":0,"companySize":"150-500人","district":"海淀區(qū)","companyLabelList":["績效獎(jiǎng)金","股票期權(quán)","專項(xiàng)獎(jiǎng)金","五險(xiǎn)一金"],"adWord":0,"appShow":0,"deliver":0,"formatCreateTime":"18:26發(fā)布","gradeDescription":null,"companyFullName":"北京趣加科技有限公司","businessZones":["中關(guān)村","知春路","雙榆樹"],"imState":"today","lastLogin":1472552889000,"publisherId":285309,"explain":null,"plus":null,"pcShow":0},{"companyId":15111,"companyShortName":"聯(lián)拓天際","createTime":"2016-08-30 18:22:12","positionId":2307696,"positionAdvantage":"與其在別處仰望忌傻，不如在這里并肩","salary":"15k-25k","score":0,"workYear":"3-5年","education":"本科","city":"北京","positionName":"iOS","companyLogo":"image1/M00/00/1D/Cgo8PFTUWGGAZQdjAADRNZVO9fc470.jpg","financeStage":"成熟型(不需要融資)","industryField":"電子商務(wù)","jobNature":"全職","approve":1,"companySize":"500-2000人","district":null,"companyLabelList":["五險(xiǎn)一金","午餐補(bǔ)助","定期體檢","技能培訓(xùn)"],"adWord":0,"appShow":0,"deliver":0,"formatCreateTime":"18:22發(fā)布","gradeDescription":null,"companyFullName":"北京聯(lián)拓天際電子商務(wù)有限公司","businessZones":null,"imState":"today","lastLogin":1472552392000,"publisherId":1595082,"explain":null,"plus":null,"pcShow":0},{"companyId":119049,"companyShortName":"優(yōu)久科技","createTime":"2016-08-30 18:15:29","positionId":1853231,"positionAdvantage":"良好的工作環(huán)境屁置、成長平臺和工作伙伴","salary":"10k-18k","score":0,"workYear":"1-3年","education":"本科","city":"北京","positionName":"iOS","companyLogo":"i/image/M00/16/74/CgqKkVbvnVuAeC-YAAA_YSPyb5A166.jpg","financeStage":"初創(chuàng)型(天使輪)","industryField":"移動(dòng)互聯(lián)網(wǎng)","jobNature":"全職","approve":0,"companySize":"少于15人","district":"海淀區(qū)","companyLabelList":["交通補(bǔ)助","通訊津貼","午餐補(bǔ)助"],"adWord":0,"appShow":0,"deliver":0,"formatCreateTime":"18:15發(fā)布","gradeDescription":null,"companyFullName":"北京優(yōu)久科技有限責(zé)任公司","businessZones":["中關(guān)村","知春路","人民大學(xué)"],"imState":"today","lastLogin":1472552013000,"publisherId":4427723,"explain":null,"plus":null,"pcShow":0},{"companyId":41878,"companyShortName":"商詢科技","createTime":"2016-08-30 18:14:06","positionId":2278393,"positionAdvantage":"微軟創(chuàng)業(yè)團(tuán)隊(duì)，工程師文化朗若！","salary":"10k-15k","score":0,"workYear":"1-3年","education":"本科","city":"北京","positionName":"iOS開發(fā)","companyLogo":"i/image/M00/24/22/Cgp3O1cZmpWAGslpAAA9MdgVNWU645.jpg","financeStage":"成長型(A輪)","industryField":"企業(yè)服務(wù),數(shù)據(jù)服務(wù)","jobNature":"全職","approve":1,"companySize":"15-50人","district":"朝陽區(qū)","companyLabelList":["股票期權(quán)","人脈資源","辦公環(huán)境好","國際化團(tuán)隊(duì)"],"adWord":0,"appShow":0,"deliver":0,"formatCreateTime":"18:14發(fā)布","gradeDescription":null,"companyFullName":"北京商詢科技有限公司","businessZones":["姚家園"],"imState":"today","lastLogin":1472554153000,"publisherId":803257,"explain":null,"plus":null,"pcShow":0},{"companyId":5832,"companyShortName":"新浪微博","createTime":"2016-08-30 18:02:30","positionId":254885,"positionAdvantage":"億級別DAU恼五，微博重點(diǎn)項(xiàng)目組","salary":"20k-40k","score":0,"workYear":"1-3年","education":"本科","city":"北京","positionName":"新浪微博iOS客戶端研發(fā)工程師","companyLogo":"image1/M00/00/0D/CgYXBlTUWCCAdkhOAABNgyvZQag818.jpg","financeStage":"上市公司","industryField":"移動(dòng)互聯(lián)網(wǎng)","jobNature":"全職","approve":0,"companySize":"2000人以上","district":"海淀區(qū)","companyLabelList":["年底雙薪","專項(xiàng)獎(jiǎng)金","股票期權(quán)","五險(xiǎn)一金"],"adWord":0,"appShow":0,"deliver":0,"formatCreateTime":"18:02發(fā)布","gradeDescription":null,"companyFullName":"微夢創(chuàng)科網(wǎng)絡(luò)科技（中國）有限公司","businessZones":["西北旺","馬連洼","上地"],"imState":"disabled","lastLogin":1472556144000,"publisherId":561302,"explain":null,"plus":null,"pcShow":0},{"companyId":48321,"companyShortName":"合廣眾","createTime":"2016-08-30 18:00:40","positionId":2263615,"positionAdvantage":"老板nice","salary":"10k-20k","score":0,"workYear":"3-5年","education":"本科","city":"北京","positionName":"iOS開發(fā)工程師","companyLogo":"i/image/M00/01/D6/CgqKkVZ496GAYypzAAAKATKLXuY379.png","financeStage":"初創(chuàng)型(天使輪)","industryField":"移動(dòng)互聯(lián)網(wǎng)","jobNature":"全職","approve":0,"companySize":"50-150人","district":"海淀區(qū)","companyLabelList":["節(jié)日禮物","帶薪年假","績效獎(jiǎng)金","崗位晉升"],"adWord":0,"appShow":0,"deliver":0,"formatCreateTime":"18:00發(fā)布","gradeDescription":null,"companyFullName":"北京合廣眾文化發(fā)展有限公司","businessZones":["八里莊","定慧寺","四季青"],"imState":"today","lastLogin":1472550077000,"publisherId":3608518,"explain":null,"plus":null,"pcShow":0},{"companyId":38239,"companyShortName":"Keep","createTime":"2016-08-30 17:52:25","positionId":2076872,"positionAdvantage":"福利健全、北京工作居住證哭懈、C輪","salary":"25k-35k","score":0,"workYear":"5-10年","education":"本科","city":"北京","positionName":"iOS開發(fā)工程師","companyLogo":"image1/M00/0A/40/CgYXBlTun9KASqKdAAAs36QVurU409.png","financeStage":"成熟型(C輪)","industryField":"社交網(wǎng)絡(luò),文化娛樂","jobNature":"全職","approve":1,"companySize":"150-500人","district":null,"companyLabelList":["節(jié)日禮物","年度旅游","定期體檢","五險(xiǎn)一金"],"adWord":0,"appShow":0,"deliver":0,"formatCreateTime":"17:52發(fā)布","gradeDescription":null,"companyFullName":"北京卡路里科技有限公司","businessZones":null,"imState":"today","lastLogin":1472550738000,"publisherId":3425178,"explain":null,"plus":null,"pcShow":0},{"companyId":179,"companyShortName":"她理財(cái)","createTime":"2016-08-30 17:52:02","positionId":982402,"positionAdvantage":"五險(xiǎn)一金 績效獎(jiǎng)金 年底15薪 帶薪年假","salary":"15k-25k","score":0,"workYear":"1-3年","education":"本科","city":"北京","positionName":"高級iOS開發(fā)工程師","companyLogo":"image1/M00/0C/F2/CgYXBlT2mG2AOPevAAB_09mD2Ko247.png","financeStage":"成長型(A輪)","industryField":"電子商務(wù),金融","jobNature":"全職","approve":1,"companySize":"50-150人","district":"朝陽區(qū)","companyLabelList":["年底雙薪","節(jié)日禮物","技能培訓(xùn)","績效獎(jiǎng)金"],"adWord":0,"appShow":0,"deliver":0,"formatCreateTime":"17:52發(fā)布","gradeDescription":null,"companyFullName":"北京新工場投資顧問有限公司","businessZones":["大望路","華貿(mào)","百子灣"],"imState":"today","lastLogin":1472557005000,"publisherId":97147,"explain":null,"plus":null,"pcShow":0},{"companyId":11053,"companyShortName":"中科三方","createTime":"2016-08-30 17:33:13","positionId":2307276,"positionAdvantage":"留用機(jī)會(huì),戶口指標(biāo)","salary":"2k-4k","score":0,"workYear":"應(yīng)屆畢業(yè)生","education":"本科","city":"北京","positionName":"iOS實(shí)習(xí)生","companyLogo":"image1/M00/00/16/CgYXBlTUWEWAXnWbAACvz96W4qA927.jpg","financeStage":"成長型(不需要融資)","industryField":"移動(dòng)互聯(lián)網(wǎng)","jobNature":"實(shí)習(xí)","approve":0,"companySize":"150-500人","district":"海淀區(qū)","companyLabelList":null,"adWord":0,"appShow":0,"deliver":0,"formatCreateTime":"17:33發(fā)布","gradeDescription":null,"companyFullName":"北京中科三方網(wǎng)絡(luò)技術(shù)有限公司","businessZones":["中關(guān)村","知春路","雙榆樹"],"imState":"today","lastLogin":1472549621000,"publisherId":141237,"explain":null,"plus":null,"pcShow":0},{"companyId":116183,"companyShortName":"情非得已","createTime":"2016-08-30 17:28:11","positionId":1786957,"positionAdvantage":"五險(xiǎn)一金灾馒、無限小吃、Mac辦公遣总、定期體檢","salary":"8k-15k","score":0,"workYear":"1-3年","education":"不限","city":"北京","positionName":"android&iOS測試工程師","companyLogo":"i/image/M00/1C/58/CgqKkVcB1QyAJM2-AAA4t6tVzs8439.jpg","financeStage":"初創(chuàng)型(天使輪)","industryField":"移動(dòng)互聯(lián)網(wǎng),企業(yè)服務(wù)","jobNature":"全職","approve":0,"companySize":"15-50人","district":"朝陽區(qū)","companyLabelList":["定期體檢","年度旅游","領(lǐng)導(dǎo)好","扁平管理"],"adWord":0,"appShow":0,"deliver":0,"formatCreateTime":"17:28發(fā)布","gradeDescription":null,"companyFullName":"情非得已（北京）科技有限公司","businessZones":["建外大街","國貿(mào)","CBD"],"imState":"today","lastLogin":1472553855000,"publisherId":4170237,"explain":null,"plus":null,"pcShow":0}]}},"code":0}
可以看到,與網(wǎng)站的第一頁獲取的實(shí)際數(shù)據(jù)是完全對應(yīng)的.
如何將數(shù)據(jù)保存為文件?
將curl的結(jié)果,直接保存為文件,才方便進(jìn)一步處理,方法就是使用重定向符 >,以下代碼,講curl的結(jié)果,不是在控制器輸出,而是保存到指定文件 1.json
curl 'http://www.lagou.com/jobs/positionAjax.json?px=new&city=%E5%8C%97%E4%BA%AC&needAddtionalResult=false' -H 'Cookie: user_trace_token=20160522122749-8a4d6717-1fd5-11e6-963e-5254005c3644; LGUID=20160522122749-8a4d6cb3-1fd5-11e6-963e-5254005c3644; tencentSig=8513357824; LGMOID=20160818212815-33C56329AA2FB6D809D557FD6CC1DE3C; JSESSIONID=E0C5692414B160F3BEFFCFF8D240B693; _gat=1; PRE_UTM=; PRE_HOST=; PRE_SITE=; PRE_LAND=http%3A%2F%2Fwww.lagou.com%2Fjobs%2Flist_iOS%3Fpx%3Dnew%26city%3D%25E5%258C%2597%25E4%25BA%25AC; ctk=1472587366; Hm_lvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1470032014,1470032429,1471526897,1471752875; Hm_lpvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1472587368; LGSID=20160831035434-92fffed3-6eeb-11e6-8c02-5254005c3644; LGRID=20160831040247-b90762b6-6eec-11e6-a745-525400f775ce; _ga=GA1.2.1654849521.1463891270; SEARCH_ID=506c8097626343ab9b5a2c1139807a2c' -H 'Origin: http://www.lagou.com' -H 'X-Anit-Forge-Code: 0' -H 'Accept-Encoding: gzip, deflate' -H 'Accept-Language: zh-CN,zh;q=0.8,en;q=0.6' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36' -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Cache-Control: max-age=0' -H 'X-Requested-With: XMLHttpRequest' -H 'Connection: keep-alive' -H 'X-Anit-Forge-Token: None' -H 'Referer: http://www.lagou.com/jobs/list_iOS?px=new&city=%E5%8C%97%E4%BA%AC' --data 'first=true&pn=1&kd=iOS' --compressed > 1.json
如何獲取其他職位的數(shù)據(jù)?
此處需要一點(diǎn)更深入些的shell語法,簡單說,需要一個(gè)for in 循環(huán)來遍歷一組給定的職位,動(dòng)態(tài)更改 前面curl腳本中的 末尾的kd屬性的值,并寫入職位對應(yīng)的文件中,注意 末尾 --data后的 單引號對,要改成雙引導(dǎo)對,否則無法應(yīng)用變量.完整代碼如下,職位數(shù)組,可按需自行添加:
for kd in "Java" "PHP" "C" "C++" "Android" "iOS"
do
curl 'http://www.lagou.com/jobs/positionAjax.json?px=new&city=%E5%8C%97%E4%BA%AC&needAddtionalResult=false' -H 'Cookie: user_trace_token=20160522122749-8a4d6717-1fd5-11e6-963e-5254005c3644; LGUID=20160522122749-8a4d6cb3-1fd5-11e6-963e-5254005c3644; tencentSig=8513357824; LGMOID=20160818212815-33C56329AA2FB6D809D557FD6CC1DE3C; JSESSIONID=E0C5692414B160F3BEFFCFF8D240B693; _gat=1; PRE_UTM=; PRE_HOST=; PRE_SITE=; PRE_LAND=http%3A%2F%2Fwww.lagou.com%2Fjobs%2Flist_iOS%3Fpx%3Dnew%26city%3D%25E5%258C%2597%25E4%25BA%25AC; ctk=1472587366; Hm_lvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1470032014,1470032429,1471526897,1471752875; Hm_lpvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1472587368; LGSID=20160831035434-92fffed3-6eeb-11e6-8c02-5254005c3644; LGRID=20160831040247-b90762b6-6eec-11e6-a745-525400f775ce; _ga=GA1.2.1654849521.1463891270; SEARCH_ID=506c8097626343ab9b5a2c1139807a2c' -H 'Origin: http://www.lagou.com' -H 'X-Anit-Forge-Code: 0' -H 'Accept-Encoding: gzip, deflate' -H 'Accept-Language: zh-CN,zh;q=0.8,en;q=0.6' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36' -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Cache-Control: max-age=0' -H 'X-Requested-With: XMLHttpRequest' -H 'Connection: keep-alive' -H 'X-Anit-Forge-Token: None' -H 'Referer: http://www.lagou.com/jobs/list_iOS?px=new&city=%E5%8C%97%E4%BA%AC' --data "first=true&pn=1&kd=$kd" --compressed > $kd.json
done
如何批量獲取?
curl 腳本,現(xiàn)在是每次只可以獲取單頁,要想獲取多頁,加個(gè)for循環(huán)就可以了.經(jīng)過觀察,拉勾有效數(shù)據(jù)大概最多在100頁左右,所以寫個(gè)1~100的循環(huán),并以 $kd_$pn.json 的格式保存:
for (( pn=1; pn<=100; pn=pn+1 )); do
for kd in "Java" "PHP" "C" "C++" "Android" "iOS"; do
curl 'http://www.lagou.com/jobs/positionAjax.json?px=new&city=%E5%8C%97%E4%BA%AC&needAddtionalResult=false' -H 'Cookie: user_trace_token=20160522122749-8a4d6717-1fd5-11e6-963e-5254005c3644; LGUID=20160522122749-8a4d6cb3-1fd5-11e6-963e-5254005c3644; tencentSig=8513357824; LGMOID=20160818212815-33C56329AA2FB6D809D557FD6CC1DE3C; JSESSIONID=E0C5692414B160F3BEFFCFF8D240B693; _gat=1; PRE_UTM=; PRE_HOST=; PRE_SITE=; PRE_LAND=http%3A%2F%2Fwww.lagou.com%2Fjobs%2Flist_iOS%3Fpx%3Dnew%26city%3D%25E5%258C%2597%25E4%25BA%25AC; ctk=1472587366; Hm_lvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1470032014,1470032429,1471526897,1471752875; Hm_lpvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1472587368; LGSID=20160831035434-92fffed3-6eeb-11e6-8c02-5254005c3644; LGRID=20160831040247-b90762b6-6eec-11e6-a745-525400f775ce; _ga=GA1.2.1654849521.1463891270; SEARCH_ID=506c8097626343ab9b5a2c1139807a2c' -H 'Origin: http://www.lagou.com' -H 'X-Anit-Forge-Code: 0' -H 'Accept-Encoding: gzip, deflate' -H 'Accept-Language: zh-CN,zh;q=0.8,en;q=0.6' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36' -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Cache-Control: max-age=0' -H 'X-Requested-With: XMLHttpRequest' -H 'Connection: keep-alive' -H 'X-Anit-Forge-Token: None' -H 'Referer: http://www.lagou.com/jobs/list_iOS?px=new&city=%E5%8C%97%E4%BA%AC' --data "first=true&pn=$pn&kd=$kd" --compressed > $kd\_$pn.json
done
done
如何提高獲取速度?
如果你運(yùn)行了上面的腳本,如你所見,似乎有點(diǎn)太慢,因?yàn)閏url請求是同步執(zhí)行的,必須一條下載完成后,才會(huì)繼續(xù)執(zhí)行下面的代碼.可以借助 & 符 異步同時(shí)獲取多個(gè)請求,來提高速度.另外需要注意的一點(diǎn)是:一個(gè)電腦,能同時(shí)創(chuàng)建的 curl 鏈接是有限的,為了避免不必要的中斷,加了個(gè)極短的sleep,改進(jìn)后的代碼如下:
注意: 此處代碼,可能會(huì)導(dǎo)致您的ip被lagou封閉,如果不是太趕時(shí)間的話,慎用;當(dāng)然,你可以多換幾個(gè)ip.
for (( pn=1; pn<=100; pn=pn+1 )); do
for kd in "Java" "PHP" "C" "C++" "Android" "iOS"; do
curl 'http://www.lagou.com/jobs/positionAjax.json?px=new&city=%E5%8C%97%E4%BA%AC&needAddtionalResult=false' -H 'Cookie: user_trace_token=20160522122749-8a4d6717-1fd5-11e6-963e-5254005c3644; LGUID=20160522122749-8a4d6cb3-1fd5-11e6-963e-5254005c3644; tencentSig=8513357824; LGMOID=20160818212815-33C56329AA2FB6D809D557FD6CC1DE3C; JSESSIONID=E0C5692414B160F3BEFFCFF8D240B693; _gat=1; PRE_UTM=; PRE_HOST=; PRE_SITE=; PRE_LAND=http%3A%2F%2Fwww.lagou.com%2Fjobs%2Flist_iOS%3Fpx%3Dnew%26city%3D%25E5%258C%2597%25E4%25BA%25AC; ctk=1472587366; Hm_lvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1470032014,1470032429,1471526897,1471752875; Hm_lpvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1472587368; LGSID=20160831035434-92fffed3-6eeb-11e6-8c02-5254005c3644; LGRID=20160831040247-b90762b6-6eec-11e6-a745-525400f775ce; _ga=GA1.2.1654849521.1463891270; SEARCH_ID=506c8097626343ab9b5a2c1139807a2c' -H 'Origin: http://www.lagou.com' -H 'X-Anit-Forge-Code: 0' -H 'Accept-Encoding: gzip, deflate' -H 'Accept-Language: zh-CN,zh;q=0.8,en;q=0.6' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36' -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Cache-Control: max-age=0' -H 'X-Requested-With: XMLHttpRequest' -H 'Connection: keep-alive' -H 'X-Anit-Forge-Token: None' -H 'Referer: http://www.lagou.com/jobs/list_iOS?px=new&city=%E5%8C%97%E4%BA%AC' --data "first=true&pn=$pn&kd=$kd" --compressed > $kd\_$pn.json &
sleep 0.02
done
done
注意: 如果一直卡住不動(dòng),可以 ctrl + c 退出;如果總是異常腳本中斷,可以嘗試將 sleep 后的數(shù)值調(diào)大.
一個(gè)更完整的腳本
此處,單獨(dú)將數(shù)據(jù)放到 jobs目錄,以便于組織目錄結(jié)構(gòu),完整數(shù)據(jù)可異步文首的github項(xiàng)目中下載:
mkdir jobs
for (( pn=1; pn<=100; pn=pn+1 )); do
for kd in "Java" "PHP" "C" "C++" "Android" "iOS"; do
curl 'http://www.lagou.com/jobs/positionAjax.json?px=new&city=%E5%8C%97%E4%BA%AC&needAddtionalResult=false' -H 'Cookie: user_trace_token=20160522122749-8a4d6717-1fd5-11e6-963e-5254005c3644; LGUID=20160522122749-8a4d6cb3-1fd5-11e6-963e-5254005c3644; tencentSig=8513357824; LGMOID=20160818212815-33C56329AA2FB6D809D557FD6CC1DE3C; JSESSIONID=E0C5692414B160F3BEFFCFF8D240B693; _gat=1; PRE_UTM=; PRE_HOST=; PRE_SITE=; PRE_LAND=http%3A%2F%2Fwww.lagou.com%2Fjobs%2Flist_iOS%3Fpx%3Dnew%26city%3D%25E5%258C%2597%25E4%25BA%25AC; ctk=1472587366; Hm_lvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1470032014,1470032429,1471526897,1471752875; Hm_lpvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1472587368; LGSID=20160831035434-92fffed3-6eeb-11e6-8c02-5254005c3644; LGRID=20160831040247-b90762b6-6eec-11e6-a745-525400f775ce; _ga=GA1.2.1654849521.1463891270; SEARCH_ID=506c8097626343ab9b5a2c1139807a2c' -H 'Origin: http://www.lagou.com' -H 'X-Anit-Forge-Code: 0' -H 'Accept-Encoding: gzip, deflate' -H 'Accept-Language: zh-CN,zh;q=0.8,en;q=0.6' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36' -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Cache-Control: max-age=0' -H 'X-Requested-With: XMLHttpRequest' -H 'Connection: keep-alive' -H 'X-Anit-Forge-Token: None' -H 'Referer: http://www.lagou.com/jobs/list_iOS?px=new&city=%E5%8C%97%E4%BA%AC' --data "first=true&pn=$pn&kd=$kd" --compressed > jobs/$kd\_$pn.json &
sleep 0.02
done
done
另外,你可能會(huì)發(fā)現(xiàn),部分職位并沒有100頁的有效數(shù)據(jù),那是否需要額外處理這些數(shù)據(jù)呢?當(dāng)然是沒有的.Spark等大數(shù)據(jù)分析工具的一個(gè)基本功能就是適度數(shù)據(jù)集容錯(cuò).部分異常數(shù)據(jù),一般是不會(huì)影響數(shù)據(jù)本身的導(dǎo)入的.導(dǎo)入后,直接分析即可.這都是后話,此系列后面的文章會(huì)單獨(dú)講述的.
