本文描述的是haproxy做代理，負載均衡气破，haproxy狀態(tài)查看聊浅，后端主機的上下線，keepalived高可用
一、前提條件
1.1?至少兩臺服務(wù)器 操作系統(tǒng)為centos7
IP：192.168.238.138/24機器名：ceph4
IP：192.168.238.139/24機器名：ceph5
1.2?部署軟件（兩臺都有）：
Keepalived高可用
Haproxy1.7.9反向代理
Apache HTTP后端主機
1.3?在兩臺主機設(shè)置分別機器名
#hostnamectl ceph4
#hostnamectl ceph4
1.4編輯vi /etc/hosts中加入(兩臺都有)
#加入本機的域名解析
192.168.238.138 ceph4
192.168.238.139 ceph5
1.5?關(guān)閉防火墻低匙，selinux(兩臺都有)
#systemctl stop firewalld#停止正在運行的防火墻
#systemctl disabled firewalld#禁止開機啟動
#setenforce 0#臨時禁止selinux
#sed -i "s/^SELINUX\=enforcing/SELINUX\=disabled/g"/etc/selinux/config#永久禁止
二旷痕、安裝配置后端軟件apache http
2.1?安裝http軟件（兩臺都有）
#yum install httpd -y
2.2?編輯配置文件改監(jiān)聽的端口
安裝完編輯vi /etc/httpd/conf/httpd.conf
Listen 8080 #監(jiān)聽端口改為8080，可以不修改顽冶，我的是主機的80端口已經(jīng)被占用
2.3?配置訪問的頁面
Ceph4：
#echo 'ceph4' >/var/www/html/index.html
Ceph5：
#echo 'ceph5' >/var/www/html/index.html
2.4?啟動http并測試
#systemctl start httpd
#curl ceph4:8080
ceph4#結(jié)果不同欺抗，以方便之后的測試
#curl ceph5:8080
ceph5
三、Haproxy安裝部署
下邊的操作兩個節(jié)點都需要安裝
3.1?相關(guān)系統(tǒng)包的安裝
#yum install -y gcc glibc gcc-c++ make screen tree lrzsz
3.2 Haproxy安裝
#mkdir /soft#創(chuàng)建目錄
#cd /soft/
#wgethttp://www.haproxy.org/download/1.7/src/haproxy-1.7.9.tar.gz#下載安裝包
#tar xf haproxy-1.7.9.tar.gz#解壓
#cd haproxy-1.7.9
#make TARGET=linux2628 PREFIX=/usr/local/haproxy1.7.9#編譯
#make install#安裝
install -d "/usr/local/sbin"
install haproxy"/usr/local/sbin"
install -d "/usr/local/share/man"/man1
install -m 644 doc/haproxy.1 "/usr/local/share/man"/man1
install -d "/usr/local/doc/haproxy"
for x in configuration management architecture cookie-options luaWURFL-device-detection proxy-protocol linux-syn-cookies network-namespacesDeviceAtlas-device-detection 51Degrees-device-detectionnetscaler-client-ip-insertion-protocol close-options SPOE intro; do \
install -m 644doc/$x.txt "/usr/local/doc/haproxy" ; \
#cp /usr/local/sbin/haproxy /usr/sbin/#啟動文件
#haproxy-v#查看安裝結(jié)果
HA-Proxy version 1.7.9 2017/08/18
Copyright 2000-2017 Willy Tarreauwilly@haproxy.org
創(chuàng)建haproxy啟動腳本
#cp examples/haproxy.init /etc/init.d/haproxy
#/etc/init.d/haproxy start#啟動
創(chuàng)建需要的相關(guān)的目錄
#useradd -r haproxy
#mkdir /etc/haproxy
#mkdir /var/lib/haproxy
#mkdir /var/run/haproxy
編輯haproxy配置文件
#vi /etc/haproxy/haproxy.cfg
global
log 127.0.0.1 local3 info
chroot /var/lib/haproxy
maxconn10000#設(shè)置允許的最大連接數(shù)强重，需要考慮ulimit -n的限制
user haproxy
group haproxy
daemon
defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
frontend front#前端
mode http
bind *:8088#這里的端口為8088绞呈，也可以是其他為占用的端口
stats uri /haproxy?stats
default_backend default_backend
backend default_backend#后端
#source cookie SERVERID
option forwardfor header X-REAL-IP
option httpchk GET /index.html#檢查的url
balance roundrobin
server ceph5 192.168.238.139:8080 check inter 2000 rise 3 fall 3 weight1
server ceph4 192.168.238.138:8080 check inter 2000 rise 3 fall 3 weight1
日志設(shè)置
#sed -i 's@\#\$ModLoad imudp@\$ModLoad imudp@g' /etc/rsyslog.conf
#sed-i 's@\#\$UDPServerRun514@\$UDPServerRun 514@g' /etc/rsyslog.conf
#echo "local3.*/var/log/haproxy.log" >> /etc/rsyslog.conf
啟動：
#/etc/init.d/haproxy start
Startinghaproxy (via systemctl):[OK]
測試：
# ceph5的haproxy配置正常
[root@ceph4 ~]# curlceph5:8088
ceph5
[root@ceph4 ~]# curlceph5:8088
ceph4
# ceph4的haproxy配置正常
[root@ceph4 ~]# curlceph4:8088
ceph5
[root@ceph4 ~]# curlceph4:8088
ceph4
看到訪問url的結(jié)果是兩臺服務(wù)器輪換相應(yīng)。
狀態(tài)管理頁面
在瀏覽器訪問http://192.168.238.138:8088/haproxy?stats间景，查看狀態(tài)
3.3 Haproxy動態(tài)維護（兩點都需要）
在配置文件的global下添加socket文件
stats socket /var/lib/haproxy/haproxy.sockmode 600 level admin
stats timeout 2m
安裝socat
#yum install -y socat
查看haproxy的幫助
#echo "help" |socat stdio /var/lib/haproxy/haproxy.sock
查看info狀態(tài)信息佃声，可以通過zabbix來監(jiān)控相關(guān)狀態(tài)值
#echo "show info" |socat stdio/var/lib/haproxy/haproxy.sock
Name: HAProxy
Version: 1.7.9
Release_date: 2017/08/18
Nbproc: 1
Process_num: 1
Pid: 5145
Uptime: 0d 0h03m34s
Uptime_sec: 214
Memmax_MB: 0
PoolAlloc_MB: 0
PoolUsed_MB: 0
PoolFailed: 0
Ulimit-n: 20033
Maxsock: 20033
Maxconn: 10000
Hard_maxconn: 10000
CurrConns: 0
CumConns: 4
CumReq: 4
Maxpipes: 0
PipesUsed: 0
PipesFree: 0
ConnRate: 0
ConnRateLimit: 0
MaxConnRate: 0
SessRate: 0
SessRateLimit: 0
MaxSessRate: 0
CompressBpsIn: 0
CompressBpsOut: 0
CompressBpsRateLim: 0
Tasks: 7
Run_queue: 1
Idle_pct: 100
node: ceph4
haproxy維護模式（主機上下線）
在ceph4上做測試，下線default_backend下的ceph4主機
#echo "disable server default_backend/ceph4 " |socat stdio/var/lib/haproxy/haproxy.sock
注：ceph4已經(jīng)不在線
上線default_backend下的ceph4
#echo "enable server default_backend/ceph4 " |socat stdio/var/lib/haproxy/haproxy.sock
注：ceph4恢復(fù)
3.4 Haproxy生產(chǎn)使用建議
haproxy的本地端口會出現(xiàn)用盡情況拱燃，解決方案如下4條
1.更改local的端口范圍,調(diào)整內(nèi)核參數(shù)
#cat /proc/sys/net/ipv4/ip_local_port_range
3276861000
2.調(diào)整timewait的端口復(fù)用秉溉，設(shè)置為1
#cat /proc/sys/net/ipv4/tcp_tw_reuse
1
3.調(diào)整tcp_wait的時間，不建議修改
#cat /proc/sys/net/ipv4/tcp_fin_timeout
60
4.最佳方案：增加多個ip碗誉，端口數(shù)量就足夠
四、Keepalived
Mail配置使用
4.1安裝mailx郵件服務(wù)
yum install mailx -y
#配置文件追加信息（/etc/mail.rc）
vim /etc/mail.rc
#發(fā)件人信息
set from=zhouguanjie2005@163.com#發(fā)件人郵箱地址(163設(shè)置得開起允許代理)
set smtp=smtp.163.com#smtp地址
setsmtp-auth-user=zhouguanjie2005@163.com#郵箱用戶名父晶，不用加域名
set smtp-auth-password=******#郵箱密碼（郵件密碼是smtp代理授權(quán)碼）
set smtp-auth=login#郵箱驗證方式
#測試發(fā)送
echo "hello world" | mail -s"hello"18706768942@163.comzhuguanjie@qq.com#會看到測試郵件信息哮缺，可以發(fā)送多個郵件
#echo "郵件內(nèi)容"
| mail -s "標題"郵箱地址
#最好把你的發(fā)送郵件地址加入你接收郵箱的白名單，不然發(fā)多了可能被認為發(fā)送垃圾郵件而被163拒絕甲喝，這是真的
4.2.安裝配置keepalived（兩節(jié)點都需要）
# yum install -y keepalived
# keepalived -v#查看版本
Keepalived v1.3.5 (03/19,2017), git commitv1.3.5-6-g6fa32f2
在/etc/keepalived下建立文件如下（兩節(jié)點）：
# ls
check_haproxy.shcheck_haproxy_url.shdown.shkeepalived.confvrrp.sh
#主要是一些腳本和keepalived配置文件
# vi check_haproxy.sh#檢測haproxy進程是否村子尝苇，不存在的話重啟
#!/bin/bash
counter=$(ps -C haproxy --no-heading|wc -l)
if [ "${counter}" = "0"]; then
/etc/init.d/haproxy start
fi
exit 0
# vi check_haproxy_url.sh#通過url檢測如果不成功返回非0，待達到次數(shù)后埠胖，keepalived會降權(quán)值變?yōu)閎ackup節(jié)點
#!/bin/bash
# curl -ILhttp://localhost/member/login.htm
# curl --data"memberName=fengkan&password=22" http://localhost/member/login.htm
count=0
for (( k=0; k<2; k++ ))
do
check_code=$( curl --connect-timeout 3 -sL -w"%{http_code}\\n" http://localhost:8088/index.html -o /dev/null )
if [ "$check_code" != "200" ]; then
# count = count +1
let "count += 1"
continue
else
count=0
break
fi
done
if [ "$count" != "0" ];then
#/etc/init.d/keepalived stop
exit 1
else
exit 0
fi
# vi down.sh#維護用的腳本糠溜，不需要手動關(guān)閉keepalived
#!/bin/bash
#判斷down文件是否存在，在需要維護的時候直撤，建立一個down文件非竿，虛擬地址會自動轉(zhuǎn)移走
if [-f /etc/keepalived/down ]; then
exit 1
else
exit 0
fi
# vi vrrp.sh (ceph4)#狀態(tài)發(fā)生變換，郵件提醒
#!/bin/bash
#當(dāng)狀態(tài)發(fā)生變換的時候谋竖，發(fā)送郵件提醒
echo "192.168.238.138 ceph4$1狀態(tài)被激活红柱，請確認HAProxy服務(wù)運行狀態(tài)"|mail -s "HAProxy狀態(tài)切換警告"15063176713@139.com
# vi?vrrp.sh (ceph5)#狀態(tài)發(fā)生變換，郵件提醒
#!/bin/bash
echo "192.168.238.139 ceph5$1狀態(tài)被激活蓖乘，請確認HAProxy服務(wù)運行狀態(tài)"|mail -s "HAProxy狀態(tài)切換警告"15063176713@139.com
建立完腳本后不要忘記賦予可執(zhí)行的權(quán)限
#chmod +x check_haproxy.sh check_haproxy_url.sh vrrp.sh down.sh
Keepalived主配置文件
這里ceph4為master節(jié)點锤悄，ceph5為backup節(jié)點
ceph4：
vi /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
acassen
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 192.168.200.1
smtp_connect_timeout 30
router_id LVS_DEVEL
}
vrrp_script chk_haproxy_url {
script "/etc/keepalived/check_haproxy_url.sh"#查看鏈接是否能正常訪問，不正常兩次后降級嘉抒，看下邊的配置
interval 2# check every 2 seconds
weight -5
fall 2#失敗兩次后零聚，觸發(fā)weight減5操作,想有降級操作必須有
rise 2#成功兩次后，恢復(fù)
}
vrrp_script chk_haproxy {
script "/etc/keepalived/check_haproxy.sh"#查看haproxy進程是否存在，不存在的話啟動隶症，無降權(quán)
interval 2#check every 2 seconds政模，執(zhí)行的時間間隔
}
vrrp_script chk_mantaince_down {
script "/etc/keepalived/down.sh"
interval 2# check every 2 seconds
weight -5
fall 2#維護操作命令，在/etc/keepalived建立down文件開始維護
rise 2
}
vrrp_instance VI_1 {
state MASTER#這里主備不一樣沿腰，注意
interface ens33#根據(jù)自己的網(wǎng)卡修改
virtual_router_id 50
#nopreempt
priority 101#設(shè)置優(yōu)先級
advert_int 1
virtual_ipaddress {
192.168.238.200#虛擬IP地址
}
track_script {
chk_haproxy_url#與上邊的執(zhí)行vrrp_script腳本對應(yīng)
chk_haproxy
chk_mantaince_down
}
#狀態(tài)轉(zhuǎn)換的時候览徒，郵件告警
notify_backup "/etc/keepalived/vrrp.shBACKUP"
notify_master "/etc/keepalived/vrrp.shMASTER"
notify_fault"/etc/keepalived/vrrp.shFAULT"
}
Ceph5
backup節(jié)點
#這里只標出與master不一樣的地方，其他同上
vi /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
acassen
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 192.168.200.1
smtp_connect_timeout 30
router_id LVS_DEVEL
}
vrrp_script chk_haproxy_url {
script "/etc/keepalived/check_haproxy_url.sh"# cheaper than pidof
interval 2#check every 2 seconds
weight -5
fall 2
rise 2
}
vrrp_script chk_haproxy {
script "/etc/keepalived/check_haproxy.sh"# cheaper than pidof
interval 2#check every 2 seconds
}
vrrp_script chk_mantaince_down {
script "/etc/keepalived/down.sh"
interval 2#check every 2 seconds
weight -5
fall 2
rise 2
}
vrrp_instance VI_1 {
state BACKUP#這里為BACKUP
interface ens33
virtual_router_id 50
#nopreempt
priority 100#設(shè)置級別
advert_int 1
virtual_ipaddress {
192.168.238.200
}
track_script {
chk_haproxy_url
chk_haproxy
chk_mantaince_down
}
notify_backup "/etc/keepalived/vrrp.shBACKUP"
notify_master "/etc/keepalived/vrrp.shMASTER"
notify_fault"/etc/keepalived/vrrp.shFAULT"
}
測試：
分別啟動keepalived
會看到
Ceph4日志：
# tailf /var/log/messages
Sep 21 15:09:55 ceph4 Keepalived[50677]:Starting Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2
Sep 21 15:09:55 ceph4 Keepalived[50677]:Unable to resolve default script username 'keepalived_script' - ignoring
Sep 21 15:09:55 ceph4 Keepalived[50677]:Opening file '/etc/keepalived/keepalived.conf'.
Sep 21 15:09:55 ceph4 systemd: PID file/var/run/keepalived.pid not readable (yet?) after start.
Sep 21 15:09:55 ceph4 Keepalived[50678]:Starting Healthcheck child process, pid=50679
Sep 21 15:09:55 ceph4 Keepalived[50678]:Starting VRRP child process, pid=50680
Sep 21 15:09:55 ceph4 systemd: Started LVSand VRRP High Availability Monitor.
Sep 21 15:09:55 ceph4Keepalived_healthcheckers[50679]: Opening file '/etc/keepalived/keepalived.conf'.
Sep 21 15:09:55 ceph4Keepalived_vrrp[50680]: Registering Kernel netlink reflector
Sep 21 15:09:55 ceph4Keepalived_vrrp[50680]: Registering Kernel netlink command channel
Sep 21 15:09:55 ceph4Keepalived_vrrp[50680]: Registering gratuitous ARP shared channel
Sep 21 15:09:55 ceph4Keepalived_vrrp[50680]: Opening file '/etc/keepalived/keepalived.conf'.
Sep 21 15:09:55 ceph4Keepalived_vrrp[50680]: VRRP_Instance(VI_1) removing protocol VIPs.
Sep 21 15:09:55 ceph4 Keepalived_vrrp[50680]:Unsafe permissions found for script '/etc/keepalived/check_haproxy_url.sh'.
Sep 21 15:09:55 ceph4Keepalived_vrrp[50680]: SECURITY VIOLATION - scripts are being executed butscript_security not enabled. There are insecure scripts.
Sep 21 15:09:55 ceph4Keepalived_vrrp[50680]: Using LinkWatch kernel netlink reflector...
Sep 21 15:09:55 ceph4Keepalived_vrrp[50680]: VRRP sockpool: [ifindex(2), proto(112), unicast(0),fd(10,11)]
Sep 21 15:09:55 ceph4Keepalived_vrrp[50680]: VRRP_Script(chk_mantaince_down) succeeded
Sep 21 15:09:55 ceph4Keepalived_vrrp[50680]: VRRP_Script(chk_haproxy) succeeded
Sep 21 15:09:55 ceph4Keepalived_vrrp[50680]: VRRP_Script(chk_haproxy_url) succeeded
Sep 21 15:09:56 ceph4Keepalived_vrrp[50680]: VRRP_Instance(VI_1) Transition to MASTER STATE
Sep21 15:09:57 ceph4 Keepalived_vrrp[50680]: VRRP_Instance(VI_1) Entering MASTERSTATE#現(xiàn)在虛擬地址在主節(jié)點上
Sep 21 15:09:57 ceph4Keepalived_vrrp[50680]: VRRP_Instance(VI_1) setting protocol VIPs.
Sep 21 15:09:57 ceph4Keepalived_vrrp[50680]: Sending gratuitous ARP on ens33 for 192.168.238.200
測試一颂龙、現(xiàn)在我在ceph4（模擬不關(guān)機維護）
# touch /etc/keepalived/down#創(chuàng)建down文件
Sep 21 15:12:49 ceph4Keepalived_vrrp[50680]: /etc/keepalived/down.sh exited with status 1
Sep 21 15:12:51 ceph4Keepalived_vrrp[50680]: /etc/keepalived/down.sh exited with status 1
Sep 21 15:12:51 ceph4Keepalived_vrrp[50680]: VRRP_Script(chk_mantaince_down) failed
Sep 21 15:12:51 ceph4Keepalived_vrrp[50680]: VRRP_Instance(VI_1) Changing effective priority from101 to 96
Sep 21 15:12:52 ceph4Keepalived_vrrp[50680]: VRRP_Instance(VI_1) Received advert with higherpriority 100, ours 96
Sep 21 15:12:52 ceph4Keepalived_vrrp[50680]: VRRP_Instance(VI_1)Entering BACKUP STATE#在創(chuàng)建down文件后习蓬，weight降級了，變?yōu)榱薆ACKUP節(jié)點
Sep 21 15:12:52 ceph4 Keepalived_vrrp[50680]:VRRP_Instance(VI_1) removing protocol VIPs.
郵箱收到信息如下：
Ceph4變?yōu)閎ackup
Ceph5變?yōu)榧せ頼aster
說明測試成功措嵌，地址已經(jīng)漂移到ceph5躲叼，ceph4可以維護了
將down文件刪掉后
# rmdown
rm:remove regular empty file ‘down’? y
ceph4
Sep 21 15:17:18 ceph4Keepalived_vrrp[50680]: VRRP_Script(chk_mantaince_down) succeeded
Sep 21 15:17:18 ceph4Keepalived_vrrp[50680]: VRRP_Instance(VI_1) Changing effective priority from 96to 101
Sep 21 15:17:18 ceph4Keepalived_vrrp[50680]: VRRP_Instance(VI_1) forcing a new MASTER election
Sep 21 15:17:19 ceph4Keepalived_vrrp[50680]: VRRP_Instance(VI_1) Transition to MASTER STATE
Sep21 15:17:20 ceph4 Keepalived_vrrp[50680]: VRRP_Instance(VI_1) Entering MASTERSTATE#變?yōu)橹鞯臓顟B(tài)
Sep 21 15:17:20 ceph4Keepalived_vrrp[50680]: VRRP_Instance(VI_1) setting protocol VIPs.
Sep 21 15:17:20 ceph4Keepalived_vrrp[50680]: Sending gratuitous ARP on ens33 for 192.168.238.200
Ceph5：
Sep 21 15:17:18 ceph5Keepalived_vrrp[11531]: VRRP_Instance(VI_1) Received advert with higherpriority 101, ours 100
Sep 21 15:17:18 ceph5Keepalived_vrrp[11531]: VRRP_Instance(VI_1) Entering BACKUP STATE
Sep21 15:17:18 ceph5 Keepalived_vrrp[11531]: VRRP_Instance(VI_1) removing protocolVIPs.#地址已經(jīng)漂移走了,回到了ceph4
