安裝
pip install scrapyd
pip install scrapydweb
配置 scrapyd
修改配置文件
以virtualenv目錄為例，打開安裝目錄下default_scrapyd.conf文件進行編輯
> vim ~/.virtualenvs/scrapyd/lib/python3.6/site-packages/scrapyd/default_scrapyd.conf
>
[scrapyd]
eggs_dir = /home/scrapyd/eggs
logs_dir = /home/scrapyd/logs
items_dir = /home/scrapyd/items
jobs_to_keep = 5
dbs_dir = /home/scrapyd/dbs
max_proc = 0
max_proc_per_cpu = 4
finished_to_keep = 100
poll_interval = 5.0
bind_address = 0.0.0.0
http_port = 6800
...
- 需要遠程訪問，修改bind_address=0.0.0.0
- eggs_dir/logs_dir/items_dir/dbs_dir 等目錄改成工作目錄
- 設置 username 和password 不為空该抒，來啟用basicAuth窿祥。
我在virtualenv 里面安裝scrapyd，啟動后一直無法正確讀取配置文件庇麦。重裝scrapyd计技，并嘗試刪除其他路徑的可能的配置文件后正常。
scrapyd 啟動時嘗試尋找的scrapyd.conf 路徑包括：
- /etc/scrapyd/scrapyd.conf
- /etc/scrapyd/conf.d/*
- scrapyd.conf
- ~/.scrapyd.conf
運行scrapyd
> nohup scrapyd>spider.log &
<回車>
配置 scrapydweb
- 設置 SCRAPYDWEB_BIND =0.0.0.0 來允許外網(wǎng)IP訪問
- 設置 ENABLE_AUTH = True 啟用basicAuth
- 設置 username 和password
############################## QUICK SETUP start ##############################
############################## 快速設置 開始 ###################################
# Setting SCRAPYDWEB_BIND to '0.0.0.0' or IP-OF-THE-CURRENT-HOST would make
# ScrapydWeb server visible externally; Otherwise, set it to '127.0.0.1'.
# The default is '0.0.0.0'.
SCRAPYDWEB_BIND = '0.0.0.0'
# Accept connections on the specified port, the default is 5000.
SCRAPYDWEB_PORT = 5000
# The default is False, set it to True to enable basic auth for the web UI.
ENABLE_AUTH = True
# In order to enable basic auth, both USERNAME and PASSWORD should be non-empty strings.
USERNAME = 'username-for-scrapydweb-login'
PASSWORD = 'xxxxxxxx'
綁定scrapyd
同一個配置文件山橄，找到SCRAPYD_SERVERS 項目垮媒。添加scrapyd 運行的host,port,username,password。
# - the string format: username:password@ip:port#group
# - The default port would be 6800 if not provided,
# - Both basic auth and group are optional.
# - e.g. '127.0.0.1:6800' or 'username:password@localhost:6801#group'
# - the tuple format: (username, password, ip, port, group)
# - When the username, password, or group is too complicated (e.g. contains ':@#'),
# - or if ScrapydWeb fails to parse the string format passed in,
# - it's recommended to pass in a tuple of 5 elements.
# - e.g. ('', '', '127.0.0.1', '6800', '') or ('username', 'password', 'localhost', '6801', 'group')
SCRAPYD_SERVERS = [
#'127.0.0.1:6800',
# 'username:password@localhost:6801#group',
('username', 'password', 'localhost', '6800', ''),
]
