服務(wù)端 scrapyd
1 環(huán)境安裝
sudo apt update -y
sudo apt install -y build-essential libssl-dev libffi-dev libxml2 libxml2-dev libxslt1-dev zlib1g-dev python-dev
sudo pip install -y scrapy redis python_redis pymongo scrapyd
2 配置
find / -name scrapyd
bind 0.0.0.0
3 運(yùn)行
scrapyd
客戶端 scrapyd-client
1 web端查看
http://localhost:6800
2 打包上傳
scrapyd-deploy
3 調(diào)度
curl http://localhost:6800/schedule.json -d project=zhihuuser -d spider=zhihu
4 取消
curl http:/localhost:6800/cancel.json -d project=zhihuuser -d job=51e8b2c0602011e9b97c000c2989fdcb
