Scrapy筆記
pip 指定源安裝模塊
pip install -i https://pypi.douban.com/simple/ 模塊名
創(chuàng)建Scrapy項(xiàng)目
scrapy startproject 項(xiàng)目名Spider
創(chuàng)建Scrapy爬蟲程序
scrapy genspider 爬蟲名稱 爬蟲網(wǎng)站
啟動Scrapy爬蟲
scrapy crawl 爬蟲名
在Pycharm中添加main.py運(yùn)行調(diào)試
import os
import sys
from scrapy.cmdline import execute
sys.path.append(os.path.dirname(os.path.abspath(__file__)))
execute(["scrapy","crawl","爬蟲名稱"])
Scrapy settings.py配置文件
# Obey robots.txt rules
ROBOTSTXT_OBEY = False
命令行模式
scrapy shell 網(wǎng)站網(wǎng)址
