centos7分布式部署pyspider

1.搭建環(huán)境：

系統(tǒng)版本：Linux centos-linux.shared 3.10.0-123.el7.x86_64 #1 SMP Mon Jun 30 12:09:22 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

python版本：Python 3.5.1

1.1.搭建python3環(huán)境：

本人在嘗試過后選擇集成環(huán)境Anaconda

1.1.1.編譯

# 下載依賴
yum install -y ncurses-devel openssl openssl-devel zlib-devel gcc make glibc-devel libffi-devel glibc-static glibc-utils sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-deve
# 下載python版本
wget https://www.python.org/ftp/python/3.5.1/Python-3.5.1.tgz
# 或者使用國(guó)內(nèi)源
wget http://mirrors.sohu.com/python/3.5.1/Python-3.5.1.tgz
mv Python-3.5.1.tgz /usr/local/src;cd /usr/local/src
# 解壓
tar -zxf Python-3.5.1.tgz;cd Python-3.5.1
# 編譯安裝
./configure --prefix=/usr/local/python3.5 --enable-shared
make && make install
# 建立軟鏈接
ln -s /usr/local/python3.5/bin/python3 /usr/bin/python3
echo "/usr/local/python3.5/lib" > /etc/ld.so.conf.d/python3.5.conf
ldconfig
# 驗(yàn)證python3
python3
# Python 3.5.1 (default, Oct  9 2016, 11:44:24)
# [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux
# Type "help", "copyright", "credits" or "license" for more information.
# >>>
# pip
/usr/local/python3.5/bin/pip3 install --upgrade pip
ln -s /usr/local/python3.5/bin/pip /usr/bin/pip
# 本人在安裝時(shí)出現(xiàn)問題 將pip重裝
wget https://bootstrap.pypa.io/get-pip.py --no-check-certificate
python get-pip.py

1.1.2.集成環(huán)境anaconda

# 集成環(huán)境anaconda(推薦)
wget https://repo.continuum.io/archive/Anaconda3-4.2.0-Linux-x86_64.sh
# 直接安裝即可
./Anaconda3-4.2.0-Linux-x86_64.sh
# 若出錯(cuò)裂七，可能是解壓失敗
yum install bzip2

1.2.安裝mariaDB

# 安裝
yum -y install mariadb mariadb-server
# 啟動(dòng)
systemctl start mariadb
# 設(shè)置為開機(jī)啟動(dòng)
systemctl enable mariadb
# 配置密碼 默認(rèn)為空
mysql_secure_installation
# 登錄
mysql -u root -p
# 創(chuàng)建一個(gè)用戶 自己設(shè)定賬戶密碼
CREATE USER 'user_name'@'localhost' IDENTIFIED BY 'user_pass';
GRANT ALL PRIVILEGES ON *.* TO 'user_name'@'localhost' WITH GRANT OPTION;
CREATE USER 'user_name'@'%' IDENTIFIED BY 'user_pass';
GRANT ALL PRIVILEGES ON *.* TO 'user_name'@'%' WITH GRANT OPTION;

1.3.安裝pyspider

本人使用Anaconda

# 搭建虛擬環(huán)境sbird python版本3.*
conda create -n sbird python=3*
# 進(jìn)入環(huán)境
source activate sbird
# 安裝pyspider
pip install pyspider
# 報(bào)錯(cuò) 
# it does not exist.  The exported locale is "en_US.UTF-8" but it is not supported
# 執(zhí)行 可寫入.bashrc
export LC_ALL=en_US.utf-8
export LANG=en_US.utf-8
#ImportError: pycurl: libcurl link-time version (7.29.0) is older than compile-time version (7.49.0)
conda install pycurl
# 退出
source deactivate sbird
# 若在虛擬機(jī)內(nèi) 出現(xiàn)無(wú)法訪問localhost:5000 可關(guān)閉防火墻
systemctl stop firewalld.service
#########直接運(yùn)行源碼==============
mkdir git;cd git
# 下載
git clone https://github.com/binux/pyspider.git
# 安裝
/root/anaconda3/envs/sbird/bin/python  /root/git/pyspider/run.py

其他方法

# 搭建虛擬環(huán)境
pip install virtualenv
mkdir python;cd python
# 創(chuàng)建虛擬環(huán)境pyenv3
virtualenv -p /usr/bin/python3 pyenv3
# 進(jìn)入虛擬環(huán)境 激活環(huán)境
cd pyenv3/
source ./bin/activate
pip install pyspider
# 若pycurl報(bào)錯(cuò) 
yum install libcurl-devel
# 繼續(xù)
pip install pyspider
# 關(guān)閉
deactivate

本人推薦用anaconda方式安裝

若pyspider運(yùn)行過程中出現(xiàn)錯(cuò)誤，參考anaconda安裝部分，至此织阅，訪問localhost:5000可看到頁(yè)面。

1.4.安裝Supervisor

# 安裝
yum install supervisor -y
# 若無(wú)法檢索 則添加阿里的epel源
vim /etc/yum.repos.d/epel.repo
# 添加以下內(nèi)容
[epel]
name=Extra Packages for Enterprise Linux 7 - $basearch
baseurl=http://mirrors.aliyun.com/epel/7/$basearch
http://mirrors.aliyuncs.com/epel/7/$basearch
failovermethod=priority
enabled=1
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7

[epel-debuginfo]
name=Extra Packages for Enterprise Linux 7 - $basearch - Debug
baseurl=http://mirrors.aliyun.com/epel/7/$basearch/debug
http://mirrors.aliyuncs.com/epel/7/$basearch/debug
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=0

[epel-source]
name=Extra Packages for Enterprise Linux 7 - $basearch - Source
baseurl=http://mirrors.aliyun.com/epel/7/SRPMS
http://mirrors.aliyuncs.com/epel/7/SRPMS
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=0
# 安裝
yum install supervisor -y
# 測(cè)試是否安裝成功
echo_supervisord_conf

1.4.1.Supervisor用法

supervisord     #supervisor的服務(wù)器端部分 啟動(dòng)
supervisorctl   #啟動(dòng)supervisor的命令行窗口
# 假設(shè)創(chuàng)建進(jìn)程pyspider01
vim /etc/supervisord.d/pyspider01.ini
# 寫入以下內(nèi)容
[program:pyspider01]

command      = /root/anaconda3/envs/sbird/bin/python  /root/git/pyspider/run.py
directory    = /root/git/pyspider
user         = root
process_name = %(program_name)s
autostart    = true
autorestart  = true
startsecs    = 3

redirect_stderr         = true
stdout_logfile_maxbytes = 500MB
stdout_logfile_backups  = 10
stdout_logfile          = /pyspider/supervisor/pyspider01.log
# 重載
supervisorctl reload
# 啟動(dòng)
supervisorctl start pyspider01
# 也可這樣啟動(dòng)
supervisord -c /etc/supervisord.conf
# 查看狀態(tài)
supervisorctl status
# output 
pyspider01                       RUNNING   pid 4026, uptime 0:02:40
# 關(guān)閉
supervisorctl shutdown

1.5.安裝redis

# 消息隊(duì)列采用redis
mkdir download;cd download
wget http://download.redis.io/releases/redis-3.2.4.tar.gz
tar xzf redis-3.2.4.tar.gz
cd redis-3.2.4
make
# 或者直接yum安裝
yum -y install redis
# 啟動(dòng)
systemctl start redis.service
# 重啟
systemctl restart redis.service
# 停止
systemctl stop redis.service
# 查看狀態(tài)
systemctl status redis.service
# 更改文件/etc/redis.conf
vim /etc/redis.conf
# 更改內(nèi)容
daemonize no 改為 daemonize yes
bind 127.0.0.1 改為 bind 10.211.55.22(當(dāng)前服務(wù)器ip)
# 重啟redis
systemctl restart redis.service

1.6.關(guān)于自啟動(dòng)

# Supervisor添加到自啟動(dòng)服務(wù)
systemctl enable supervisord.service
# redis添加到自啟動(dòng)服務(wù)
systemctl enable redis.service
# 關(guān)閉防火墻自啟動(dòng)
systemctl disable firewalld.service

至此嗅钻，pyspider單個(gè)服務(wù)器運(yùn)行環(huán)境搭建且部署完畢灼擂，啟動(dòng)localhost:5000進(jìn)入web界面。

也可編寫腳本運(yùn)行峻贮，在/pyspider/supervisor/pyspider01.log查看運(yùn)行狀態(tài)。

2.分布式部署

剛才配置的服務(wù)器碉纺，將其命名為centos01，按照這樣的配置舱呻，再分別部署兩臺(tái)centos02、centos03茬高。

如下：

服務(wù)器名稱	ip	說(shuō)明
centos01	10.211.55.22	redis,mariaDB, scheduler
centos02	10.211.55.23	fetcher, processor, result_worker,phantomjs
centos03	10.211.55.24	fetcher, processor,,result_worker,webui

2.1.centos01

進(jìn)入服務(wù)器centos01慨亲，經(jīng)過第一步，基本環(huán)境已經(jīng)搭好蛉签，首先編輯配置文件/pyspider/config.json

{
  "taskdb": "mysql+taskdb://user_name:user_pass@10.211.55.22:3306/taskdb",
  "projectdb": "mysql+projectdb://user_name:user_pass@10.211.55.22:3306/projectdb",
  "resultdb": "mysql+resultdb://user_name:user_pass@10.211.55.22:3306/resultdb",
  "message_queue": "redis://10.211.55.22:6379/db",
  "logging-config": "/pyspider/logging.conf",
  "phantomjs-proxy":"10.211.55.23:25555",
  "webui": {
    "username": "",
    "password": "",
    "need-auth": false,
    "host":"10.211.55.24",
    "port":"5000",
    "scheduler-rpc":"http:// 10.211.55.22:5002",
    "fetcher-rpc":"http://10.211.55.23:5001"
  },
  "fetcher": {
    "xmlrpc":true,
    "xmlrpc-host": "0.0.0.0",
    "xmlrpc-port": "5001"
  },
  "scheduler": {
    "xmlrpc":true,
    "xmlrpc-host": "0.0.0.0",
    "xmlrpc-port": "5002"
  }
}

嘗試運(yùn)行下：

/root/anaconda3/envs/sbird/bin/python /root/git/pyspider/run.py -c /pyspider/config.json scheduler
# 報(bào)錯(cuò)
ImportError: No module named 'mysql'
# 下載 mysql-connector-python
cd ~/git/
git clone https://github.com/mysql/mysql-connector-python.git
# 安裝
source activate sbird
cd mysql-connector-python
python setup.py install
# 安裝redis
pip install redis
source deactivate
# 運(yùn)行
/root/anaconda3/envs/sbird/bin/python /root/git/pyspider/run.py -c /pyspider/config.json scheduler
# 輸出 ok
[I 161010 15:57:25 scheduler:644] scheduler starting...
[I 161010 15:57:25 scheduler:779] scheduler.xmlrpc listening on 0.0.0.0:5002
[I 161010 15:57:25 scheduler:583] in 5m: new:0,success:0,retry:0,failed:0

運(yùn)行成功后淮野，可直接更改/etc/supervisord.d/pyspider01.ini如下：

[program:pyspider01]

command      = /root/anaconda3/envs/sbird/bin/python /root/git/pyspider/run.py -c /pyspider/config.json scheduler
directory    = /root/git/pyspider
user         = root
process_name = %(program_name)s
autostart    = true
autorestart  = true
startsecs    = 3

redirect_stderr         = true
stdout_logfile_maxbytes = 500MB
stdout_logfile_backups  = 10
stdout_logfile          = /pyspider/supervisor/pyspider01.log

# 重載
supervisorctl reload
# 查看狀態(tài)
supervisorctl status

centos01部署完畢。

2.2.centos02

在centos02中，需要運(yùn)行result_worker、processor潭袱、phantomjs慎式、fetcher

分別建立文件：

/etc/supervisord.d/result_worker.ini

[program:result_worker]

command      = /root/anaconda3/envs/sbird/bin/python /root/git/pyspider/run.py -c /pyspider/config.json result_worker
directory    = /root/git/pyspider
user         = root
process_name = %(program_name)s
autostart    = true
autorestart  = true
startsecs    = 3

redirect_stderr         = true
stdout_logfile_maxbytes = 500MB
stdout_logfile_backups  = 10
stdout_logfile          = /pyspider/supervisor/result_worker.log

/etc/supervisord.d/processor.ini

[program:processor]

command      = /root/anaconda3/envs/sbird/bin/python /root/git/pyspider/run.py -c /pyspider/config.json processor
directory    = /root/git/pyspider
user         = root
process_name = %(program_name)s
autostart    = true
autorestart  = true
startsecs    = 3

redirect_stderr         = true
stdout_logfile_maxbytes = 500MB
stdout_logfile_backups  = 10
stdout_logfile          = /pyspider/supervisor/processor.log

/etc/supervisord.d/phantomjs.ini

[program:phantomjs]

command      = /pyspider/phantomjs --config=/pyspider/pjsconfig.json /pyspider/phantomjs_fetcher.js 25555
directory    = /root/git/pyspider
user         = root
process_name = %(program_name)s
autostart    = true
autorestart  = true
startsecs    = 3

redirect_stderr         = true
stdout_logfile_maxbytes = 500MB
stdout_logfile_backups  = 10
stdout_logfile          = /pyspider/supervisor/phantomjs.log

/etc/supervisord.d/fetcher.ini

[program:fetcher]

command      = /root/anaconda3/envs/sbird/bin/python /root/git/pyspider/run.py -c /pyspider/config.json fetcher
directory    = /root/git/pyspider
user         = root
process_name = %(program_name)s
autostart    = true
autorestart  = true
startsecs    = 3

redirect_stderr         = true
stdout_logfile_maxbytes = 500MB
stdout_logfile_backups  = 10
stdout_logfile          = /pyspider/supervisor/fetcher.log

在pyspider目錄中建立pjsconfig.json

{
  /*--ignore-ssl-errors=true */
  "ignoreSslErrors": true,
  
  /*--ssl-protocol=true */
  "sslprotocol": "any",
  
  /* Same as: --output-encoding=utf8 */
  "outputEncoding": "utf8",
  
  /* persistent Cookies. */
  /*cookiesfile="e:/phontjscookies.txt",*/
  cookiesfile="pyspider/phontjscookies.txt",
  
  /* load image */
  autoLoadImages = false
}

下載phantomjs至/pyspider/文件夾幕屹，將git/pyspider/pyspider/fetcher/phantomjs_fetcher.js復(fù)制到phantomjs_fetcher.js

# 重載
supervisorctl reload
# 查看狀態(tài)
supervisorctl status
# output
fetcher                          RUNNING   pid 3446, uptime 0:00:07
phantomjs                        RUNNING   pid 3448, uptime 0:00:07
processor                        RUNNING   pid 3447, uptime 0:00:07
result_worker                    RUNNING   pid 3445, uptime 0:00:07

centos02部署完畢。

2.3.centos03

部署這三個(gè)進(jìn)程fetcher, processor, result_worker和centos02 一樣，本服務(wù)器主要是在前面的基礎(chǔ)上加上webui

建立文件：

/etc/supervisord.d/webui.ini

[program:webui]

command      = /root/anaconda3/envs/sbird/bin/python /root/git/pyspider/run.py -c /pyspider/config.json webui
directory    = /root/git/pyspider
user         = root
process_name = %(program_name)s
autostart    = true
autorestart  = true
startsecs    = 3

redirect_stderr         = true
stdout_logfile_maxbytes = 500MB
stdout_logfile_backups  = 10
stdout_logfile          = /pyspider/supervisor/webui.log

# 重載
supervisorctl reload
# 查看狀態(tài)
supervisorctl status
# output
fetcher                          RUNNING   pid 2724, uptime 0:00:07
processor                        RUNNING   pid 2725, uptime 0:00:07
result_worker                    RUNNING   pid 2723, uptime 0:00:07
webui                            RUNNING   pid 2726, uptime 0:00:07

3.總結(jié)

訪問 http://10.211.55.24:5000 即可拟淮，盡情爬取吧很泊。

pyspider

最后編輯于：2017.12.04 05:16:54

?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請(qǐng)聯(lián)系作者

人面猴
序言：七十年代末争涌，一起剝皮案震驚了整個(gè)濱河市辣恋，隨后出現(xiàn)的幾起案子伟骨，更是在濱河造成了極大的恐慌，老刑警劉巖稀并，帶你破解...
沈念sama閱讀 222,729評(píng)論 6贊 517
死咒
序言：濱河連續(xù)發(fā)生了三起死亡事件，死亡現(xiàn)場(chǎng)離奇詭異，居然都是意外死亡境蜕，警方通過查閱死者的電腦和手機(jī)售滤，發(fā)現(xiàn)死者居然都...
沈念sama閱讀 95,226評(píng)論 3贊 399
救了他兩次的神仙讓他今天三更去死
文/潘曉璐我一進(jìn)店門，熙熙樓的掌柜王于貴愁眉苦臉地迎上來(lái)，“玉大人吉捶，你說(shuō)我怎么就攤上這事∈成耄” “怎么了仅胞？”我有些...
開封第一講書人閱讀 169,461評(píng)論 0贊 362
道士緝兇錄：失蹤的賣姜人
文/不壞的土叔我叫張陵剑辫，是天一觀的道長(zhǎng)妹蔽。經(jīng)常有香客問我，道長(zhǎng)胳岂，這世上最難降的妖魔是什么旦万？我笑而不...
開封第一講書人閱讀 60,135評(píng)論 1贊 300
?港島之戀（遺憾婚禮）
正文為了忘掉前任，我火速辦了婚禮，結(jié)果婚禮上淆两，老公的妹妹穿的比我還像新娘拂酣。我一直安慰自己，他們只是感情好剑勾，可當(dāng)我...
茶點(diǎn)故事閱讀 69,130評(píng)論 6贊 398
惡毒庶女頂嫁案：這布局不是一般人想出來(lái)的
文/花漫我一把揭開白布虽另。她就那樣靜靜地躺著族展，像睡著了一般。火紅的嫁衣襯著肌膚如雪。梳的紋絲不亂的頭發(fā)上刻炒，一...
開封第一講書人閱讀 52,736評(píng)論 1贊 312
城市分裂傳說(shuō)
那天访敌，我揣著相機(jī)與錄音，去河邊找鬼。笑死，一個(gè)胖子當(dāng)著我的面吹牛独柑，可吹牛的內(nèi)容都是我干的狂秘。我是一名探鬼主播，決...
沈念sama閱讀 41,179評(píng)論 3贊 422
雙鴛鴦連環(huán)套：你想象不到人心有多黑
文/蒼蘭香墨我猛地睜開眼，長(zhǎng)吁一口氣：“原來(lái)是場(chǎng)噩夢(mèng)啊……” “哼拥刻！你這毒婦竟也來(lái)了？” 一聲冷哼從身側(cè)響起，我...
開封第一講書人閱讀 40,124評(píng)論 0贊 277
萬(wàn)榮殺人案實(shí)錄
序言：老撾萬(wàn)榮一對(duì)情侶失蹤介劫，失蹤者是張志新（化名）和其女友劉穎誉碴，沒想到半個(gè)月后成黄，有當(dāng)?shù)厝嗽跇淞掷锇l(fā)現(xiàn)了一具尸體思瘟，經(jīng)...
沈念sama閱讀 46,657評(píng)論 1贊 320
?護(hù)林員之死
正文獨(dú)居荒郊野嶺守林人離奇死亡畜份，尸身上長(zhǎng)有42處帶血的膿包…… 初始之章·張勛以下內(nèi)容為張勛視角年9月15日...
茶點(diǎn)故事閱讀 38,723評(píng)論 3贊 342
?白月光啟示錄
正文我和宋清朗相戀三年，在試婚紗的時(shí)候發(fā)現(xiàn)自己被綠了。大學(xué)時(shí)的朋友給我發(fā)了我未婚夫和他白月光在一起吃飯的照片。...
茶點(diǎn)故事閱讀 40,872評(píng)論 1贊 353
活死人
序言：一個(gè)原本活蹦亂跳的男人離奇死亡鞭执，死狀恐怖大溜，靈堂內(nèi)的尸體忽然破棺而出，到底是詐尸還是另有隱情，我是刑警寧澤葵诈，帶...
沈念sama閱讀 36,533評(píng)論 5贊 351
?日本核電站爆炸內(nèi)幕
正文年R本政府宣布砖顷，位于F島的核電站，受9級(jí)特大地震影響，放射性物質(zhì)發(fā)生泄漏。R本人自食惡果不足惜崖咨，卻給世界環(huán)境...
茶點(diǎn)故事閱讀 42,213評(píng)論 3贊 336
男人毒藥：我在死后第九天來(lái)索命
文/蒙蒙一世曾、第九天我趴在偏房一處隱蔽的房頂上張望岭佳。院中可真熱鬧叶洞，春花似錦艺晴、人聲如沸碗淌。這莊子的主人今日做“春日...
開封第一講書人閱讀 32,700評(píng)論 0贊 25
一樁弒父案爹耗，背后竟有這般陰謀
文/蒼蘭香墨我抬頭看了看天上的太陽(yáng)山卦。三九已至枚碗，卻和暖如春怨规，著一層夾襖步出監(jiān)牢的瞬間波丰，已是汗流浹背媚赖。一陣腳步聲響...
開封第一講書人閱讀 33,819評(píng)論 1贊 274
情欲美人皮
我被黑心中介騙來(lái)泰國(guó)打工，沒想到剛下飛機(jī)就差點(diǎn)兒被人妖公主榨干…… 1. 我叫王不留犁河，地道東北人酿秸。一個(gè)月前我還...
沈念sama閱讀 49,304評(píng)論 3贊 379
代替公主和親
正文我出身青樓，卻偏偏與公主長(zhǎng)得像唱矛，于是被迫代替她去往敵國(guó)和親包个。傳聞我的和親對(duì)象是個(gè)殘疾皇子，可洞房花燭夜當(dāng)晚...
茶點(diǎn)故事閱讀 45,876評(píng)論 2贊 361