安裝

服務(wù)
pip install scrapyd
使用命令行工具
python3 -m pip install scrapyd-client
python連接包
python3 -m pip install python-scrapyd-api
找到python文件路徑供置，設(shè)置scrapyd可以直接執(zhí)行

image.png

啟動服務(wù)

scrapyd

image.png

image.png

image.png

<p><code>curl http://localhost:6800/schedule.json -d project=default -d spider=somespider</code></p>
<p><code> curl http://localhost:6800/cancel.json -d project=myproject -d job=6487ec79947edab326d6db28a2d86511e8247444 </code></p>
<p><code> curl http://localhost:6800/listprojects.json </code></p>
<p><code> curl http://localhost:6800/listversions.json?project=myproject </code></p>
<p><code> curl http://localhost:6800/listspiders.json?project=myproject </code></p><p><code>  curl http://localhost:6800/listjobs.json?project=myproject </code></p>
<p><code> curl http://localhost:6800/delproject.json -d project=myproject </code></p>
<p><code> curl http://localhost:6800/delversion.json -d project=myproject -d version=r99 </code></p>

修改完成如下：

image.png

image.png

image.png

image.png

image.png

image.png

image.png

image.png

[root@localhost sinanews]# curl http://localhost:6800/listprojects.json
{"node_name": "localhost.localdomain", "status": "ok", "projects": ["sinanews"]}

list version

[root@localhost sinanews]# curl http://localhost:6800/listversions.json?project=sinanews
{"node_name": "localhost.localdomain", "status": "ok", "versions": ["1"]}

再deploy一個版本2遇骑，然后list version

[root@localhost sinanews]# curl http://localhost:6800/listversions.json?project=sinanews
{"node_name": "localhost.localdomain", "status": "ok", "versions": ["1", "2"]}

list spiders

[root@localhost sinanews]# curl http://localhost:6800/listspiders.json?project=sinanews
{"node_name": "localhost.localdomain", "status": "ok", "spiders": ["sina"]}

運行爬蟲

[root@localhost sinanews]# curl http://localhost:6800/schedule.json -d project=sinanews -d spider=sina
{"node_name": "localhost.localdomain", "status": "ok", "jobid": "2157910a9ef811e995c020040fe78714"}

取消任務(wù)

[root@localhost sinanews]# curl http://localhost:6800/cancel.json -d project=sinanews -d job=2157910a9ef811e995c020040fe78714
{"node_name": "localhost.localdomain", "status": "ok", "prevstate": null}

刪除項目

curl http://localhost:6800/delproject.json -d project=myproject

刪掉指定版本

curl http://localhost:6800/delversion.json -d project=myproject -d version=r99

日志文件存放
日志目錄/項目名稱/爬蟲名稱/任務(wù)ID.log,存儲個數(shù)根據(jù)配置文檔來定

image.png

image.png

使用scrapyd_api

調(diào)度

from scrapyd_api import ScrapydAPI
scrapyd = ScrapydAPI('http://localhost:6800')
scrapyd.schedule(project_name, spider_name)

源碼修改，方便使用cancel

#scrapyd.webservice.py
class SpiderId(WsResource):

    def render_POST(self, txrequest):
        args = native_stringify_dict(copy(txrequest.args), keys_only=False)
        project = args['project'][0]
        spider = args['spider'][0]
        spiders = self.root.launcher.processes.values()
        running = [(s.job,s.start_time.isoformat(' '))
                   for s in spiders if (s.project == project and s.spider == spider)]
        # queue = self.root.poller.queues[project]
        # pending = [(x["_job"],) for x in queue.list() if x["name"] == spider]
        finished = [(s.job,s.start_time.isoformat(' ')) for s in self.root.launcher.finished
            if (s.project == project and s.spider == spider)]
        alist = running + finished
        if len(alist) == 0:
            return {"node_name": self.root.nodename, "status": "error", "message": 'no such project or spider'}
        last_id = max(alist,key=lambda a:a[0])
        return {"node_name": self.root.nodename, "status": "ok", 'id': last_id[0]}

#scrapyd.default_scrapyd.conf
spiderid.json     = scrapyd.webservice.SpiderId

#scrapyd.website.py
<p><code> curl http://localhost:6800/cancel.json -d project=myproject -d job=6487ec79947edab326d6db28a2d86511e8247444 </code></p>
<p><code> curl http://localhost:6800/listprojects.json </code></p>
<p><b><code> curl http://localhost:6800/spiderid.json -d project=myproject -d spider=spider</b></code></p>
<p><code> curl http://localhost:6800/listversions.json?project=myproject </code></p>
<p><code> curl http://localhost:6800/listspiders.json?project=myproject </code></p><p><code>  curl http://localhost:6800/listjobs.json?project=myproject </code></p>
<p><code> curl http://localhost:6800/delproject.json -d project=myproject </code></p>
<p><code> curl http://localhost:6800/delversion.json -d project=myproject -d version=r99 </code></p>

scrapyd-api代碼修改

#contants.py
SPIDERID_ENDPOINT = 'spiderid'
DEFAULT_ENDPOINTS = {
    ADD_VERSION_ENDPOINT: '/addversion.json',
    CANCEL_ENDPOINT: '/cancel.json',
    DELETE_PROJECT_ENDPOINT: '/delproject.json',
    DELETE_VERSION_ENDPOINT: '/delversion.json',
    LIST_JOBS_ENDPOINT: '/listjobs.json',
    LIST_PROJECTS_ENDPOINT: '/listprojects.json',
    LIST_SPIDERS_ENDPOINT: '/listspiders.json',
    LIST_VERSIONS_ENDPOINT: '/listversions.json',
    SCHEDULE_ENDPOINT: '/schedule.json',
    SPIDERID_ENDPOINT: '/spiderid.json',
}

wrapper.py

def spiderid(self, project, spider):
        """
        """
        url = self._build_url(constants.SPIDERID_ENDPOINT)
        params = {'project': project, 'spider': spider}
        json = self.client.post(url, data=params)
        return json['id']