環(huán)境

• ubuntu 14.04 LTS
• Scrapy 1.4.0

爬取鏈接

• http://search.51job.com/list/030200,000000,0000,00,9,99,python,2,1.html?lang=c&stype=1&postchannel=0000&workyear=99&cotype=99&degreefrom=99&jobterm=99&companysize=99&lonlat=0%2C0&radius=-1&ord_field=0&confirmdate=9&fromType=&dibiaoid=0&address=&line=&specialarea=00&from=&welfare=

爬取步驟

1. 在瀏覽器中打開爬取鏈接, 并進入審查元素模式(F12), 定位到要爬取的標(biāo)題浴讯。

1.png

2. 在終端中輸入

scrapy shell "http://search.51job.com/list/030200,000000,0000,00,9,99,python,2,1.html?lang=c&stype=1&postchannel=0000&workyear=99&cotype=99&degreefrom=99&jobterm=99&companysize=99&lonlat=0%2C0&radius=-1&ord_field=0&confirmdate=9&fromType=&dibiaoid=0&address=&line=&specialarea=00&from=&welfare="

3. 使用Scrapy選擇器(Selectors)獲取崗位標(biāo)題

# 輸入, 由步驟1中知道css位置為 p > a
response.css('p a::attr(title)').extract_first()
# 輸出
u'Python\u5f00\u53d1\u5de5\u7a0b\u5e08'

4. 獲取公司名稱

# 輸入
response.css('span.t2 a::attr(title)').extract_first()
# 輸出
u'\u4e0a\u6d77\u65b0\u5de5\u5f0f\u7f51\u7edc\u79d1\u6280\u6709\u9650\u516c\u53f8'

5. 獲取其它信息類似，這里不多說，Selectors使用方法請參考
   https://doc.scrapy.org/en/1.4/topics/selectors.html

編寫整體代碼

• 因為每個崗位的css結(jié)構(gòu)為div.dw_table div.el, 所以在代碼中要把它遍歷出來

2.png

• 整體代碼fiveOneJob_spider.py

#coding:utf-8

"""
author: ilyq69
date: 20170828
"""

import scrapy


class FiveOneJobSpider(scrapy.Spider):

    name = 'fiveOnejob'

    start_urls= [
        'http://search.51job.com/list/030200,000000,0000,00,9,99,python,2,1.html?lang=c&stype=1&postchannel=0000&workyear=99&cotype=99&degreefrom=99&jobterm=99&companysize=99&lonlat=0%2C0&radius=-1&ord_field=0&confirmdate=9&fromType=&dibiaoid=0&address=&line=&specialarea=00&from=&welfare='
    ]

    def parse(self, response):
        '''
        :param response:
        :return:
        '''
        for item in response.css('div.dw_table div.el'):
            yield {
                "title": item.css('p a::attr(title)').extract_first(),
                "link": item.css('p a::attr(href)').extract_first(),
                "company": item.css('span.t2 a::attr(title)').extract_first(),
                "city": item.css('span.t3 ::text').extract_first(),
                "salary": item.css('span.t4 ::text').extract_first(),
                "createTime": item.css('span.t5 ::text').extract_first()
            }

        next_page = response.css('div.p_in ul li.bk a::attr(href)').extract()
        if next_page is not None:
            if len(next_page) > 1:
                yield response.follow(next_page[1], callback=self.parse)
            else:
                yield response.follow(next_page[0], callback=self.parse)

運行測試

• 運行

scrapy runspider fiveOneJob_spider.py -o job.json

• 查看結(jié)果, 打開job.json

參考

• https://doc.scrapy.org/en/1.4/index.html

最后

• 代碼僅供學(xué)習(xí)使用宿百，侵刪