所謂"元數(shù)據(jù)",就是把自身的信息傳遞給其他組件.
scrapy提供的item類似于Python的字典,但是比字典的功能更為便捷,用來(lái)封裝數(shù)據(jù).用法如下:
import scrapy
class Bookitem(scrapy.Item):
# 相當(dāng)于定義字典的鍵
name=scrapy.Field()
price=scrapy.Field()
#實(shí)例化的時(shí)候,同時(shí)賦值
book1=Bookitem(name='The story',price='22.06元')
print(book1)
>>>{'name': 'The story', 'price': '22.06元'}
#也可以先實(shí)例化,以后再賦值(scrapy里面一般都是這樣處理)
book2=Bookitem()
print(book2)
>>>{}
book2['name']='How to learn Python'
book2['price']='66.60元'
print(book2)
#字典的方法也適用的:
print(book1.get('price','65.00元'))
>>>22.06元
#復(fù)習(xí)一下字典的用法:
print(book1)
>>>{'name': 'The story', 'price': '22.06元'}
print(list(book1))
>>>['price', 'name']
print(list(book1.keys()))
>>>['price', 'name']
print(list(book1.values()))
>>>['22.06元', 'The story']
print(list(book1.items()))
>>>[('price', '22.06元'), ('name', 'The story')]
現(xiàn)在修改原來(lái)爬取books.toscrape.com的實(shí)例:
- items.py這么寫(xiě):
import scrapy
class BookItem(scrapy.Item):
name=scrapy.Field()
price=scrapy.Field()
- spiders這么寫(xiě):
# -*- coding: utf-8 -*-
import scrapy
from Book.items import BookItem
class BooksSpider(scrapy.Spider):
name = 'books'
allowed_domains = ['books.toscrape.com']
#start_urls = ['http://books.toscrape.com/']
def start_requests(self):
yield scrapy.Request(
'http://books.toscrape.com',
callback=self.parse_book,
headers={'User-Agent':'Mozilla/5.0'},
dont_filter=True
)
def parse_book(self, response):
path=response.xpath('//li[@class="col-xs-6 col-sm-4 col-md-3 col-lg-3"]/article')
for book in path:
book_info=BookItem()
book_info['name']=book.xpath('./h3/a/text()').extract()
book_info['price']=book.xpath('./div[2]/p[1]/text()').extract()
yield book_info
next_page=response.xpath('//li[@class="next"]/a/@href').extract_first()
if next_page:
next_page=response.urljoin(next_page)
yield scrapy.Request(next_page,callback=self.parse_book)
#scrapy crawl books -o first_scrapy.csv
#忽略表頭,在屏幕上輸出
#sed -n '2,$p' first_scrapy.csv|cat -n
關(guān)于scrapy.Field()字段的元數(shù)據(jù),我是這么理解,通常來(lái)說(shuō),是用來(lái)定義鍵的,但是通過(guò)元數(shù)據(jù),可以進(jìn)行直接賦值處理:
import scrapy
class ExampleItem(scrapy.Item):
x=scrapy.Field(a='hello',b=[1,2,3])
y=scrapy.Field(a=lambda x:x**2)
e=ExampleItem(x=100,y=200)
print(e.fields)
>>>{'y': {'a': <function ExampleItem.<lambda> at 0x0000000000DAB840>}, 'x': {'b': [1, 2, 3], 'a': 'hello'}}
print(e.get('x'))
>>>100
print(issubclass(scrapy.Field,dict))
>>>True
書(shū)上的一個(gè)實(shí)例:
book['authors']=['李雷','韓梅梅','吉姆']
想實(shí)現(xiàn)這樣的效果:
1.'李雷|韓梅梅|吉姆'
2.'李雷,韓梅梅,吉姆'
3."['李雷','韓梅梅','吉姆']"
只需這樣定義author字段:
authors=scrapy.Field(serializer=lambda x:'|'.join(x))
底層源碼的實(shí)現(xiàn),可以再參考書(shū)本.
