scrapy作為一個(gè)強(qiáng)大的爬蟲框架痛侍，就不多作介紹曹步。
今天剛?cè)腴Tscrapy宪彩，所以做個(gè)簡(jiǎn)單的使用
Scrapy中文文檔
Ubuntu安裝scrapy
在Ubuntu下安裝scrapy，需要的命令為：
pip install zope.interface
pip install twisted
pip install pyopenssl
pip install scrapy
開始
1. 使用scrapy創(chuàng)建文件夾命令：
scrapy startproject doubanTest
命令執(zhí)行完系統(tǒng)將在當(dāng)前目錄下生成doubanTest文件
2. 看看最后的代碼文件目錄（開始時(shí)創(chuàng)建不存在douban_spider.py讲婚，main.py文件）
3. 網(wǎng)頁(yè)提取的元素需要在items.py定義
# -*- coding: utf-8 -*-
# Define here the models for your scraped items
#
# See documentation in:
# http://doc.scrapy.org/en/latest/topics/items.html
import scrapy
class DoubantestItem(scrapy.Item):
# define the fields for your item here like:
# name = scrapy.Field()
title = scrapy.Field() # 抓取電影標(biāo)題
content = scrapy.Field() # 抓取電影簡(jiǎn)介
比如我在代碼中定義了抓取電影的標(biāo)題和電影的簡(jiǎn)介
關(guān)于item的使用毯焕，官方文檔這么描述:
Item用法類似字典dict
4. 在settings.py定義一些防反爬元素
# -*- coding: utf-8 -*-
# Scrapy settings for doubanTest project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
# http://doc.scrapy.org/en/latest/topics/settings.html
# http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html
# http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html
BOT_NAME = 'doubanTest'
SPIDER_MODULES = ['doubanTest.spiders']
NEWSPIDER_MODULE = 'doubanTest.spiders'
# Crawl responsibly by identifying yourself (and your website) on the user-agent
USER_AGENT = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.96 Safari/537.36'
FEED_URI = 'doubanmovie.csv' # 將信息存儲(chǔ)到doubanmovie.csv文件中
FEED_FORMAT = 'CSV' # 存儲(chǔ)的文件類型
5. 在spiders下創(chuàng)建douban_spider.py，編寫主要代碼
#-*_coding:utf8-*-
import scrapy
from scrapy.spider import Spider
from scrapy.http import Request
from scrapy.selector import Selector
from doubanTest.items import DoubantestItem
class doubanSpider(Spider):
name="doubanSpider" # 爬蟲程序名稱
start_urls = ['https://movie.douban.com/top250'] # URL訪問列表
def parse(self, response):
# print response.body
item = DoubantestItem() # 定義一個(gè)Item對(duì)象
selector = Selector(response)
Movies = selector.xpath('//div[@class="info"]')
for each in Movies:
title = each.xpath('div[@class="hd"]/a/span/text()').extract() # 獲取電影標(biāo)題
full_title = "".join(title)
content = each.xpath('div[@class="bd"]/p/text()').extract() # 獲取電影簡(jiǎn)介
# item類似dict, 它的key名稱必須與items.py里的名稱一致
item['title'] = full_title
item['content'] = ';'.join(content).strip()
yield item
6. 在doubanTest目錄下創(chuàng)建一個(gè)main.py
編寫這兩句代碼，就可以使用pycharm運(yùn)行了
#-*_coding:utf8-*-
from scrapy import cmdline
cmdline.execute("scrapy crawl doubanSpider".split())
運(yùn)行main.py得到的結(jié)果
ok,就當(dāng)作scrapy入門例子吧
