scrapy——settings設(shè)置解讀
# -*- coding: utf-8 -*-
# Scrapy settings for demo project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
# http://doc.scrapy.org/en/latest/topics/settings.html
# http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html
# http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html
BOT_NAME = 'demo' #Scrapy項(xiàng)目的名字,這將用來構(gòu)造默認(rèn) User-Agent,同時(shí)也用來log,當(dāng)您使用 startproject 命令創(chuàng)建項(xiàng)目時(shí)其也被自動賦值滑肉。
SPIDER_MODULES = ['demo.spiders'] #Scrapy搜索spider的模塊列表 默認(rèn): [xxx.spiders]
NEWSPIDER_MODULE = 'demo1.spiders' #使用 genspider 命令創(chuàng)建新spider的模塊迫像。默認(rèn): 'xxx.spiders'
#爬取的默認(rèn)User-Agent，除非被覆蓋
#USER_AGENT = 'demo1 (+http://www.yourdomain.com)'
#如果啟用,Scrapy將會采用 robots.txt策略
ROBOTSTXT_OBEY = True
#Scrapy downloader 并發(fā)請求(concurrent requests)的最大值,默認(rèn): 16
#CONCURRENT_REQUESTS = 32
#為同一網(wǎng)站的請求配置延遲（默認(rèn)值：0）
# See http://scrapy.readthedocs.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
#DOWNLOAD_DELAY = 3 下載器在下載同一個(gè)網(wǎng)站下一個(gè)頁面前需要等待的時(shí)間,該選項(xiàng)可以用來限制爬取速度,減輕服務(wù)器壓力橱鹏。同時(shí)也支持小數(shù):0.25 以秒為單位
#下載延遲設(shè)置只有一個(gè)有效
#CONCURRENT_REQUESTS_PER_DOMAIN = 16 對單個(gè)網(wǎng)站進(jìn)行并發(fā)請求的最大值卵皂。
#CONCURRENT_REQUESTS_PER_IP = 16 對單個(gè)IP進(jìn)行并發(fā)請求的最大值心褐。如果非0,則忽略 CONCURRENT_REQUESTS_PER_DOMAIN 設(shè)定,使用該設(shè)定蜈七。 也就是說,并發(fā)限制將針對IP,而不是網(wǎng)站乱凿。該設(shè)定也影響 DOWNLOAD_DELAY: 如果 CONCURRENT_REQUESTS_PER_IP 非0,下載延遲應(yīng)用在IP而不是網(wǎng)站上。
#禁用Cookie（默認(rèn)情況下啟用）
#COOKIES_ENABLED = False
#禁用Telnet控制臺（默認(rèn)啟用）
#TELNETCONSOLE_ENABLED = False
#覆蓋默認(rèn)請求標(biāo)頭：
#DEFAULT_REQUEST_HEADERS = {
# 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
# 'Accept-Language': 'en',
#}
#啟用或禁用蜘蛛中間件
# See http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html
#SPIDER_MIDDLEWARES = {
# 'demo1.middlewares.Demo1SpiderMiddleware': 543,
#}
#啟用或禁用下載器中間件
# See http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html
#DOWNLOADER_MIDDLEWARES = {
# 'demo1.middlewares.MyCustomDownloaderMiddleware': 543,
#}
#啟用或禁用擴(kuò)展程序
# See http://scrapy.readthedocs.org/en/latest/topics/extensions.html
#EXTENSIONS = {
# 'scrapy.extensions.telnet.TelnetConsole': None,
#}
#配置項(xiàng)目管道
# See http://scrapy.readthedocs.org/en/latest/topics/item-pipeline.html
#ITEM_PIPELINES = {
# 'demo1.pipelines.Demo1Pipeline': 300,
#}
#啟用和配置AutoThrottle擴(kuò)展（默認(rèn)情況下禁用）
# See http://doc.scrapy.org/en/latest/topics/autothrottle.html
#AUTOTHROTTLE_ENABLED = True
#初始下載延遲
#AUTOTHROTTLE_START_DELAY = 5
#在高延遲的情況下設(shè)置的最大下載延遲
#AUTOTHROTTLE_MAX_DELAY = 60
#Scrapy請求的平均數(shù)量應(yīng)該并行發(fā)送每個(gè)遠(yuǎn)程服務(wù)器
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
#啟用顯示所收到的每個(gè)響應(yīng)的調(diào)節(jié)統(tǒng)計(jì)信息：
#AUTOTHROTTLE_DEBUG = False
#啟用和配置HTTP緩存（默認(rèn)情況下禁用）
# See http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
#HTTPCACHE_ENABLED = True
#HTTPCACHE_EXPIRATION_SECS = 0
#HTTPCACHE_DIR = 'httpcache'
#HTTPCACHE_IGNORE_HTTP_CODES = []
#HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'
解釋幾個(gè)參數(shù)：
ROBOTSTXT_OBEY = True 是否遵守robots.txt
CONCURRENT_REQUESTS = 16 開啟線程數(shù)量础倍，默認(rèn)16
AUTOTHROTTLE_START_DELAY = 3: 開始下載時(shí)限速并延遲時(shí)間
AUTOTHROTTLE_MAX_DELAY = 60高并發(fā)請求時(shí)最大延遲時(shí)間
最底下的幾個(gè)：是否啟用在本地緩存童社，如果開啟會優(yōu)先讀取本地緩存，從而加快爬取速度著隆，視情況而定
HTTPCACHE_ENABLED = True
HTTPCACHE_EXPIRATION_SECS = 0
HTTPCACHE_DIR = 'httpcache'
HTTPCACHE_IGNORE_HTTP_CODES = []
HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'
以上幾個(gè)可以視項(xiàng)目需要開啟，但是有兩個(gè)參數(shù)最好每次都開啟呀癣，而每次都是項(xiàng)目文件手動開啟不免有些麻煩美浦，最好是項(xiàng)目創(chuàng)建后就自動開啟
DEFAULT_REQUEST_HEADERS = {
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Language': 'en',
}
