更多干活分享可訪問博主個(gè)人網(wǎng)站
https://www.fzg5.com/blog/
scrapy_projects
可以作為 scrapy 學(xué)習(xí)項(xiàng)目
項(xiàng)目一
爬取一生必須知道的50幅中國名畫，每一幅你都不容錯(cuò)過 這篇文章中的50幅名畫
- items創(chuàng)建
painter = scrapy.Field()
pic_name = scrapy.Field()
picture = scrapy.Field()
- scrapy配置
allowed_domains = ['sohu.com']
start_urls = ['http://www.sohu.com/a/157709282_661623']
def parse(self, response):
pic_list = response.xpath('//article[@class="article"]/p')
items = []
for pic in pic_list[2:]:
if len(pic.extract().split('／'))>1:
item = FamouspicspiderItem()
item['painter'] = pic.xpath('span/text()')[0].extract().split('／')[1]
item['pic_name'] = pic.extract().split('／')[0].split('、')[1]
items.append(item)
if pic.xpath('img/@src').extract():
items[-1]['picture'] = pic.xpath('img/@src').extract()[0]
return items
- pipelines自定義存儲
with open(picPath, 'wb') as fp:
response = urlopen(item['picture'])
fp.write(response.read())
4.修改配置文件,注冊自定義存儲文件
# Configure item pipelines
# See https://doc.scrapy.org/en/latest/topics/item-pipeline.html
ITEM_PIPELINES = {
'famousPicSpider.pipelines.FamouspicspiderPipeline': 300,
}
5虎敦、scrapy crawl famousPic
GitHub
https://github.com/shellingshord/scrapy_projects#%E9%A1%B9%E7%9B%AE%E4%B8%80
更多干活分享可訪問博主個(gè)人網(wǎng)站
https://www.fzg5.com/blog/
