Scrapy 的環(huán)境搭建
找到python3
> which python3
/Users/macroot/virtualenvs/article_spider/bin/python3
生成虛擬環(huán)境
virtualenv --python=/Users/macroot/virtualenvs/article_spider/bin/python3 article_spider
進(jìn)入文件夾
/Users/macroot [macroot@macroots-MacBook-Pro] [0:02]
> cd imooc
生成爬蟲項(xiàng)目
/Users/macroot/imooc [macroot@macroots-MacBook-Pro] [0:02]
> scrapy startproject ArticleSpider
在項(xiàng)目外面創(chuàng)建spider是錯(cuò)誤的弄跌。刪掉
/Users/macroot/imooc [macroot@macroots-MacBook-Pro] [0:12]
> scrapy genspider jobbole blog.jobbole.com
Created spider 'jobbole' using template 'basic'
(article_spider)
/Users/macroot/imooc [macroot@macroots-MacBook-Pro] [0:13]
> ls
ArticleSpider BingSearch html_start jobbole.py py3rex
(article_spider)
/Users/macroot/imooc [macroot@macroots-MacBook-Pro] [0:13]
> rm -rf jobbole.py
進(jìn)入目錄去創(chuàng)建spider，scrapy會自己放到spider目錄下。
/Users/macroot/imooc [macroot@macroots-MacBook-Pro] [0:14]
> ls
ArticleSpider BingSearch html_start py3rex
(article_spider)
/Users/macroot/imooc [macroot@macroots-MacBook-Pro] [0:14]
> cd ArticleSpider
(article_spider)
/Users/macroot/imooc/ArticleSpider [macroot@macroots-MacBook-Pro] [0:14]
> scrapy genspider jobbole blog.jobbole.com
Created spider 'jobbole' using template 'basic' in module:
ArticleSpider.spiders.jobbole
(article_spider)
