非結(jié)構(gòu)化數(shù)據(jù)抓取流程

1.抓取網(wǎng)絡(luò)數(shù)據(jù)包

2堵幽、F12抓包,抓取到j(luò)son地址 和 查詢參數(shù)(QueryString)
url = 'https://image.so.com/zjl?ch=beauty&t1=595&src=banner_beauty&sn={}&listtype=new&temp=1'.format(sn)
ch: beauty
t1: 595
src: banner_beauty
sn: 90
listtype: new
temp: 1

項目實現(xiàn)

1.創(chuàng)建爬蟲項目和爬蟲文件

scrapy startproject So
cd So
scrapy genspider so image.so.com

2.響應(yīng)對象屬性及方法

?# 屬性
1滑频、response.text ：獲取響應(yīng)內(nèi)容 - 字符串
2橄教、response.body ：獲取bytes數(shù)據(jù)類型
3、response.xpath('')
?
? # response.xpath('')調(diào)用方法
1、結(jié)果 ：列表,元素為選擇器對象
? # <selector xpath='//article' data=''>
2、.extract() ：提取文本內(nèi)容,將列表中所有元素序列化為Unicode字符串
3、.extract_first() ：提取列表中第1個文本內(nèi)容
4叙量、.get() ： 提取列表中第1個文本內(nèi)容

代碼實現(xiàn)

1.定義要爬取的數(shù)據(jù)結(jié)構(gòu)(items.py)

# -*- coding: utf-8 -*-

# Define here the models for your scraped items
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/items.html

import scrapy


class SoItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    img_link = scrapy.Field()

2.爬蟲文件實現(xiàn)圖片鏈接爬取，把鏈接yield到項目管道

# -*- coding: utf-8 -*-
import scrapy
import json
from ..items import SoItem

class SoSpider(scrapy.Spider):
    name = 'so'
    allowed_domains = ['image.so.com']

    # 重寫start_requests()方法九串，把所有URL地址都交給調(diào)度器
    def start_requests(self):
        url = "https://image.so.com/zjl?ch=beauty&t1=595&src=banner_beauty&sn={}&listtype=new&temp=1"
        #生成5頁地址绞佩，交給調(diào)度器
        for i in range(5):
            sn = i * 30
            full_url = url.format(sn)
            # 交給調(diào)度器
            yield scrapy.Request(
                url = full_url ,
                callback=self.parse_image
            )

    def parse_image(self, response):
        html = json.loads(response.text)
        #提取圖片鏈接
        for img in html["list"]:
            item = SoItem()
            item["img_link"] = img["qhimg_url"]
            yield item

3.pipelines.py

# -*- coding: utf-8 -*-

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html

#導(dǎo)入scrapy的圖片管道類
from scrapy.pipelines.images import ImagesPipeline
import scrapy

#1.繼承 ImagesPipeline
#2.重寫類內(nèi)方法
class SoPipeline(ImagesPipeline):
    def get_media_requests(self, item, info):
        #把圖片鏈接發(fā)給調(diào)度器
        yield scrapy.Request(url = item['img_link'])

4.settings.py

定義圖片存儲路徑：IMAGES_STORE = 'D:\node\nd\spider\day09\photo\'
打開CONCURRENT_REQUESTS = 10寺鸥，最大并發(fā)量,注意以下沒注釋的代碼。

# -*- coding: utf-8 -*-

# Scrapy settings for So project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
#     https://docs.scrapy.org/en/latest/topics/settings.html
#     https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
#     https://docs.scrapy.org/en/latest/topics/spider-middleware.html

BOT_NAME = 'So'

SPIDER_MODULES = ['So.spiders']
NEWSPIDER_MODULE = 'So.spiders'


# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'So (+http://www.yourdomain.com)'

# Obey robots.txt rules
ROBOTSTXT_OBEY = False

# Configure maximum concurrent requests performed by Scrapy (default: 16)
#最大并發(fā)量品山，默認16
CONCURRENT_REQUESTS = 10

# Configure a delay for requests for the same website (default: 0)
# See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
#DOWNLOAD_DELAY = 3
# The download delay setting will honor only one of:
#CONCURRENT_REQUESTS_PER_DOMAIN = 16
#CONCURRENT_REQUESTS_PER_IP = 16

# Disable cookies (enabled by default)
#COOKIES_ENABLED = False

# Disable Telnet Console (enabled by default)
#TELNETCONSOLE_ENABLED = False

# Override the default request headers:
DEFAULT_REQUEST_HEADERS = {
  'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
  'Accept-Language': 'en',
  'User-Agent':'Mozilla/5.0'
}

# Enable or disable spider middlewares
# See https://docs.scrapy.org/en/latest/topics/spider-middleware.html
#SPIDER_MIDDLEWARES = {
#    'So.middlewares.SoSpiderMiddleware': 543,
#}

# Enable or disable downloader middlewares
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
#DOWNLOADER_MIDDLEWARES = {
#    'So.middlewares.SoDownloaderMiddleware': 543,
#}

# Enable or disable extensions
# See https://docs.scrapy.org/en/latest/topics/extensions.html
#EXTENSIONS = {
#    'scrapy.extensions.telnet.TelnetConsole': None,
#}

# Configure item pipelines
# See https://docs.scrapy.org/en/latest/topics/item-pipeline.html
ITEM_PIPELINES = {
   'So.pipelines.SoPipeline': 300,
}

# Enable and configure the AutoThrottle extension (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/autothrottle.html
#AUTOTHROTTLE_ENABLED = True
# The initial download delay
#AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
#AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
#AUTOTHROTTLE_DEBUG = False

# Enable and configure HTTP caching (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
#HTTPCACHE_ENABLED = True
#HTTPCACHE_EXPIRATION_SECS = 0
#HTTPCACHE_DIR = 'httpcache'
#HTTPCACHE_IGNORE_HTTP_CODES = []
#HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'

IMAGES_STORE = 'D:\\node\\nd\\spider\\day09\\photo\\'

5.爬取結(jié)束后胆建，會在目錄下生成full文件，里面有我想爬取的150文件肘交，