當(dāng)標(biāo)準(zhǔn)scrapyd 下載 pipeline 無法滿足需求時(shí)，可以自定義pipeline奔誓。
僅舉例文件下載和圖片下載pipeline斤吐。
擴(kuò)展文件(圖片) FilesPipeline (ImagesPipeline)僅需重寫以下兩個(gè)方法：
get_media_request(self, item, info) # 返回一個(gè)Request對(duì)象
# 當(dāng)上面的Requsts下載完成后回調(diào)這個(gè)方法，然后填充files或images字段
item_completed(self, results, item, info)
舉例：
pipelines.py
import scrapyd
from scrapy.pipelines.images import ImagesPipeline
from scrapy.exceptions import DropItem
class MyImagePipeline(ImagesPipeline):
def get_media_requests(self, item, info):
for image_url in item['image_urls']:
yield scrapy.Request(image_url)
def item_completed(self, request, item, info):
image_paths = [x['path'] for ok, x in request if ok]
if not image_paths:
raise DropItem("item contains no images")
item['image_paths'] = image_paths
return item
