Kubernetes 基于 haproxy 實(shí)現(xiàn) ingress 服務(wù)暴露

一逆皮、HAproxy Ingress 控制器

1.1 HAproxy Ingress 簡介

HAProxy Ingress watches in the k8s cluster and how it builds HAProxy configuration

和 Nginx 相類似，HAproxy 通過監(jiān)視 kubernetes api 獲取到 service 后端 pod 的狀態(tài)秽梅，動(dòng)態(tài)更新 haproxy 配置文件抹蚀，以實(shí)現(xiàn)七層的負(fù)載均衡环壤。

HAproxy Ingress 控制器具備的特性如下：
  • Fast钞诡，Carefully built on top of the battle-tested HAProxy load balancer. 基于 haproxy 性能有保障
  • Reliable，Trusted by sysadmins on clusters as big as 1,000 namespaces, 2,000 domains and 3,000 ingress objects. 可靠接箫，支持1000最多1000個(gè)命名空間和 2000多個(gè)域名
  • Highly customizable朵诫，100+ configuration options and growing. 可定制化強(qiáng)，支持100多個(gè)配置選項(xiàng)
  • 通過使用一個(gè) IP 地址和端口路由入口流量來簡化基礎(chǔ)架構(gòu)废累。根據(jù)主機(jī)請求頭和請求路徑將請求路由到正確的 Pod脱盲。
  • 利用世界上最快钱反，使用最廣泛的軟件負(fù)載平衡器 HAProxy。在性能诈铛，可靠性和安全性方面，HAProxy 設(shè)定了新標(biāo)準(zhǔn)耳峦。
  • 使用內(nèi)置的 SSL 終止焕毫，速率限制和 IP 白名單來保護(hù)您的集群。
  • 使用 HAProxy 的任何負(fù)載平衡算法（包括循環(huán)循签，最少連接疙咸，URL 哈希和隨機(jī)）來平衡 Pod 之間的流量。
  • 開箱即用的第7層卓越的可觀察性可盡早避免出現(xiàn)問題贼穆。HAProxy 附帶有一個(gè)儀表板兰粉，該儀表板可顯示吊艙的運(yùn)行狀況，當(dāng)前請求率愕秫，響應(yīng)時(shí)間等。
    HAProxy 的流量過載保護(hù)可帶來更高的吞吐量戴甩。服務(wù)器不會(huì)收到超出其處理能力的更多請求等恐。
HAproxy ingress 控制器版本
  • 社區(qū)版备蚓，基于 haproxy 社區(qū)高度定制符合 ingress 的控制器郊尝，功能相對有限
  • 企業(yè)版战惊，haproxy 企業(yè)版本，支持很多高級特性和功能吞获，大部分高級功能在企業(yè)版本中實(shí)現(xiàn)

1.2 HAproxy 控制器安裝

haproxy ingress 安裝相對簡單，官方提供了安裝的 yaml 文件刁绒，先將文件下載查看一下 kubernetes 資源配置烤黍，包含的資源類型有：

  • ServiceAccount 和 RBAC 認(rèn)證授權(quán)關(guān)聯(lián)
  • RBAC 認(rèn)證 Role、ClusterRole嫂丙、 ClusterRoleBinding
  • Service 后端的一個(gè) service
  • DaemonSet HAproxy 最核心的一個(gè)控制器规哲，關(guān)聯(lián)認(rèn)證 ServiceAccount 和配置 ConfigMap，定義了一個(gè) nodeSelector隅肥，label 為 role: ingress-controller，將運(yùn)行在特定的節(jié)點(diǎn)上
  • ConfigMap 實(shí)現(xiàn) haproxy ingress 自定義配置

安裝文件路徑https://haproxy-ingress.github.io/resources/haproxy-ingress.yaml
1议双、創(chuàng)建命名空間，haproxy ingress 部署在 ingress-controller 這個(gè)命名空間藕坯，先創(chuàng)建 ns
2宗雇、安裝 haproxy ingress 控制器
3赔蒲、檢查 haproxy ingress 安裝情況良漱，檢查 haproxy ingress 核心的DaemonSets，發(fā)現(xiàn) DS 并未部署 Pod矾兜，原因是配置文件中定義了nodeSelector 節(jié)點(diǎn)標(biāo)簽選擇器患久，因此需要給 node 設(shè)置合理的標(biāo)簽
4、 給 node 設(shè)置標(biāo)簽返帕，讓 DaemonSets 管理的 Pod 能調(diào)度到 node 節(jié)點(diǎn)上篙挽，生產(chǎn)環(huán)境中根據(jù)情況定義，將實(shí)現(xiàn) haproxy ingress 功能的節(jié)點(diǎn)定義到特定的節(jié)點(diǎn)观腊，對個(gè) node 節(jié)點(diǎn)的訪問算行，需要借助于負(fù)載均衡實(shí)現(xiàn)統(tǒng)一接入，本文主要以探究 haproxy ingress 功能儡陨，因此未部署負(fù)載均衡調(diào)度器，可根據(jù)實(shí)際的情況部署骗村。以 node-1 和 node-2 為例：
5、再次查看 ingress 部署情況笼痛，已完成部署琅拌，并調(diào)度至 node-1 和 node-2 節(jié)點(diǎn)上
6、 查看 haproxy ingress 的日志刻坊，通過查詢?nèi)罩究芍辰鄠€(gè) haproxy ingress 是通過選舉實(shí)現(xiàn)高可用 HA 機(jī)制。

其他資源包括 ServiceAccount灾而，ClusterRole扳剿，ConfigMaps 請單獨(dú)確認(rèn)，至此 HAproxy ingress controller 部署完畢。另外兩種部署方式：

Deployment 部署方式（http://dwz.date/cmP9）
Helm 部署方式（http://dwz.date/cmPB）

二癣猾、haproxy ingress 使用

2.1 haproxy ingress 基礎(chǔ)

Ingress 控制器部署完畢后需要定義 Ingress 規(guī)則，以方便 Ingress 控制器能夠識(shí)別到 service 后端 Pod 的資源夸盟，這個(gè)章節(jié)我們將來介紹在 HAproxy Ingress Controller 環(huán)境下 Ingress 的使用像捶。

1、環(huán)境準(zhǔn)備释簿，創(chuàng)建一個(gè) deployments 并暴露其端口

#創(chuàng)建應(yīng)用并暴露端口
[root@node-1 haproxy-ingress]# cat haproxy-ingress-nginx-test.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: haproxy-ingress-demo
  name: haproxy-ingress-demo
spec:
  replicas: 3
  selector:
    matchLabels:
      app: haproxy-ingress-demo
  template:
    metadata:
      labels:
        app: haproxy-ingress-demo
    spec:
      containers:
      - image: nginx:1.17.0
        name: nginx

---
apiVersion: v1
kind: Service
metadata:
  name: haproxy-ingress-demo
  namespace: default
spec:
  clusterIP: 10.109.197.67
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: haproxy-ingress-demo
  type: ClusterIP


#查看應(yīng)用
[root@node-1 haproxy-ingress]# kubectl get deployments haproxy-ingress-demo
NAME READY UP-TO-DATE AVAILABLE AGE
haproxy-ingress-demo 1/1     1            1           10s

#查看service情況
[root@node-1 haproxy-ingress]# kubectl get services haproxy-ingress-demo
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
haproxy-ingress-demo ClusterIP 10.109.197.67   <none>        80/TCP 17s

2庶溶、創(chuàng)建 ingress 規(guī)則,如果有多個(gè) ingress 控制器，可以通過 ingress.class 指定類型為 haproxy

[root@node-1 haproxy-ingress]# cat ingress-demo.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: haproxy-ingress-demo
  labels:
    ingresscontroller: haproxy
  annotations:
    kubernetes.io/ingress.class: haproxy
spec:
  rules:
  - host: www.happylau.cn 
    http:
      paths:
      - path: /
        backend:
          serviceName: haproxy-ingress-demo
          servicePort: 80

3行疏、應(yīng)用 ingress 規(guī)則套像，并查看 ingress 詳情，查看 Events 日志發(fā)現(xiàn)控制器已正常更新

[root@node-1 haproxy-ingress]# kubectl apply -f ingress-demo.yaml
ingress.extensions/haproxy-ingress-demo created

#查看詳情
[root@node-1 haproxy-ingress]# kubectl describe ingresses haproxy-ingress-demo
Name: haproxy-ingress-demo
Namespace: default
Address:
Default backend: default-http-backend:80 (<none>)
Rules:
  Host Path Backends
  ---- ---- --------
  www.happylau.cn
                   / haproxy-ingress-demo:80 (10.244.2.166:80)
Annotations:
  kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"haproxy"},"labels":{"ingresscontroller":"haproxy"},"name":"haproxy-ingress-demo","namespace":"default"},"spec":{"rules":[{"host":"www.happylau.cn","http":{"paths":[{"backend":{"serviceName":"haproxy-ingress-demo","servicePort":80},"path":"/"}]}}]}}

  kubernetes.io/ingress.class: haproxy
Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Normal CREATE 27s ingress-controller Ingress default/haproxy-ingress-demo
  Normal CREATE 27s ingress-controller Ingress default/haproxy-ingress-demo
  Normal UPDATE 20s ingress-controller Ingress default/haproxy-ingress-demo
  Normal UPDATE 20s ingress-controller Ingress default/haproxy-ingress-demo

4贞让、測試驗(yàn)證 ingress 規(guī)則震桶，可以將域名寫入到 hosts 文件中征绎，我們直接使用 gcurl 測試，地址指向 node-1 或 node-2 均可

[root@node-1 haproxy-ingress]# curl http://www.happylau.cn --resolve www.happylau.cn:80:10.254.100.101
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a >nginx.org</a>.<br/>
Commercial support is available at
<a >nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

5柴墩、測試正常凫岖，接下來到 haproxy ingress controller 中剛查看對應(yīng)生成規(guī)則配置文件

[root@node-1 ~]# kubectl exec -it haproxy-ingress-bdns8 -n ingress-controller /bin/sh

#查看配置文件
/etc/haproxy # cat /etc/haproxy/haproxy.cfg
  # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# #
# # HAProxy Ingress Controller
# # --------------------------
# # This file is automatically updated, do not edit
# #
# 全局配置文件內(nèi)容
global
    daemon
    nbthread 2
    cpu-map auto:1/1-2 0-1
    stats socket /var/run/haproxy-stats.sock level admin expose-fd listeners
    maxconn 2000
    hard-stop-after 10m
    lua-load /usr/local/etc/haproxy/lua/send-response.lua
    lua-load /usr/local/etc/haproxy/lua/auth-request.lua
    tune.ssl.default-dh-param 2048
    ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK
    ssl-default-bind-options no-sslv3 no-tls-tickets

#默認(rèn)配置內(nèi)容
defaults
    log global
    maxconn 2000
    option redispatch
    option dontlognull
    option http-server-close
    option http-keep-alive
    timeout client 50s
    timeout client-fin 50s
    timeout connect 5s
    timeout http-keep-alive 1m
    timeout http-request 5s
    timeout queue 5s
    timeout server 50s
    timeout server-fin 50s
    timeout tunnel 1h

#后端服務(wù)器哥放，即通過service服務(wù)發(fā)現(xiàn)機(jī)制，和后端的Pod關(guān)聯(lián)
backend default_haproxy-ingress-demo_80
    mode http
    balance roundrobin
    acl https-request ssl_fc
    http-request set-header X-Original-Forwarded-For %[hdr(x-forwarded-for)] if { hdr(x-forwarded-for) -m found }
    http-request del-header x-forwarded-for
    option forwardfor
    http-response set-header Strict-Transport-Security "max-age=15768000"
    server srv001 10.244.2.166:80 weight 1 check inter 2s #后端Pod的地址
    server srv002 127.0.0.1:1023 disabled weight 1 check inter 2s
    server srv003 127.0.0.1:1023 disabled weight 1 check inter 2s
    server srv004 127.0.0.1:1023 disabled weight 1 check inter 2s
    server srv005 127.0.0.1:1023 disabled weight 1 check inter 2s
    server srv006 127.0.0.1:1023 disabled weight 1 check inter 2s
    server srv007 127.0.0.1:1023 disabled weight 1 check inter 2s

backend _error404
    mode http
    http-request use-service lua.send-404


#前端監(jiān)聽的80端口轉(zhuǎn)發(fā)規(guī)則踩身，并配置有https跳轉(zhuǎn)挟阻，對應(yīng)的主機(jī)配置在/etc/haproxy/maps/_global_http_front.map文件中定義
frontend _front_http
    mode http
    bind *:80
    http-request set-var(req.base) base,lower,regsub(:[0-9]+/,/)
    http-request redirect scheme https if { var(req.base),map_beg(/etc/haproxy/maps/_global_https_redir.map,_nomatch) yes }
    http-request set-header X-Forwarded-Proto http
    http-request del-header X-SSL-Client-CN
    http-request del-header X-SSL-Client-DN
    http-request del-header X-SSL-Client-SHA1
    http-request del-header X-SSL-Client-Cert
    http-request set-var(req.backend) var(req.base),map_beg(/etc/haproxy/maps/_global_http_front.map,_nomatch)
    use_backend %[var(req.backend)] unless { var(req.backend) _nomatch }
    default_backend _default_backend

#前端監(jiān)聽的443轉(zhuǎn)發(fā)規(guī)則峭弟，對應(yīng)域名在/etc/haproxy/maps/ _front001_host.map文件中
frontend _front001
    mode http
    bind *:443 ssl alpn h2,http/1.1 crt /ingress-controller/ssl/default-fake-certificate.pem
    http-request set-var(req.hostbackend) base,lower,regsub(:[0-9]+/,/),map_beg(/etc/haproxy/maps/_front001_host.map,_nomatch)
    http-request set-header X-Forwarded-Proto https
    http-request del-header X-SSL-Client-CN
    http-request del-header X-SSL-Client-DN
    http-request del-header X-SSL-Client-SHA1
    http-request del-header X-SSL-Client-Cert
    use_backend %[var(req.hostbackend)] unless { var(req.hostbackend) _nomatch }
    default_backend _default_backend

#狀態(tài)監(jiān)聽器
listen stats
    mode http
    bind *:1936
    stats enable
    stats uri /
    no log
    option forceclose
    stats show-legends

#監(jiān)控健康檢查
frontend healthz
    mode http
    bind *:10253
    monitor-uri /healthz

查看主機(jī)名隱射文件瞒瘸，包含有前端主機(jī)名和轉(zhuǎn)發(fā)到后端 backend 的名稱

/etc/haproxy/maps # cat /etc/haproxy/maps/_global_http_front.map
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# #
# # HAProxy Ingress Controller
# # --------------------------
# # This file is automatically updated, do not edit
# #
#
www.happylau.cn/ default_haproxy-ingress-demo_80

通過上面的基礎(chǔ)配置可以實(shí)現(xiàn)基于 haproxy 的七層負(fù)載均衡實(shí)現(xiàn)，haproxy ingress controller 通過 kubernetes api 動(dòng)態(tài)識(shí)別到 service 后端規(guī)則配置并更新至haproxy.cfg 配置文件中情臭，從而實(shí)現(xiàn)負(fù)載均衡功能實(shí)現(xiàn)玉组。

2.2 動(dòng)態(tài)更新和負(fù)載均衡

后端 Pod 是實(shí)時(shí)動(dòng)態(tài)變化的惯雳，haproxy ingress 通過 service 的服務(wù)發(fā)現(xiàn)機(jī)制鸿摇，動(dòng)態(tài)識(shí)別到后端 Pod 的變化情況，并動(dòng)態(tài)更新 haproxy.cfg 配置文件拙吉，并重載配置（實(shí)際不需要重啟 haproxy 服務(wù)），本章節(jié)將演示 haproxy ingress 動(dòng)態(tài)更新和負(fù)載均衡功能往史。
1佛舱、動(dòng)態(tài)更新，我們以擴(kuò)容 pod 的副本為例订歪，將副本數(shù)從 replicas=1 擴(kuò)容至 3 個(gè)

[root@node-1 ~]# kubectl scale --replicas=3 deployment haproxy-ingress-demo
deployment.extensions/haproxy-ingress-demo scaled
[root@node-1 ~]# kubectl get deployments haproxy-ingress-demo
NAME READY UP-TO-DATE AVAILABLE AGE
haproxy-ingress-demo 3/3     3            3           43m

#查看擴(kuò)容后Pod的IP地址
[root@node-1 ~]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
haproxy-ingress-demo-5d487d4fc-5pgjt 1/1     Running 0          43m     10.244.2.166   node-3   <none>           <none>
haproxy-ingress-demo-5d487d4fc-pst2q 1/1     Running 0          18s 10.244.0.52    node-1   <none>           <none>
haproxy-ingress-demo-5d487d4fc-sr8tm 1/1     Running 0          18s 10.244.1.149   node-2   <none>           <none>

2肆捕、查看 haproxy 配置文件內(nèi)容,可以看到 backend 后端主機(jī)列表已動(dòng)態(tài)發(fā)現(xiàn)新增的pod 地址

backend default_haproxy-ingress-demo_80
    mode http
    balance roundrobin
    acl https-request ssl_fc
    http-request set-header X-Original-Forwarded-For %[hdr(x-forwarded-for)] if { hdr(x-forwarded-for) -m found }
    http-request del-header x-forwarded-for
    option forwardfor
    http-response set-header Strict-Transport-Security "max-age=15768000"
    server srv001 10.244.2.166:80 weight 1 check inter 2s #新增的pod地址
    server srv002 10.244.0.52:80 weight 1 check inter 2s
    server srv003 10.244.1.149:80 weight 1 check inter 2s
    server srv004 127.0.0.1:1023 disabled weight 1 check inter 2s
    server srv005 127.0.0.1:1023 disabled weight 1 check inter 2s
    server srv006 127.0.0.1:1023 disabled weight 1 check inter 2s
    server srv007 127.0.0.1:1023 disabled weight 1 check inter 2s

3慎陵、查看 haproxy ingress 日志，日志中提示 HAProxy updated without needing to reload席纽，即配置動(dòng)態(tài)識(shí)別，不需要重啟 haproxy 服務(wù)就能夠識(shí)別，自從1.8后haproxy 能支持動(dòng)態(tài)配置更新的能力诈豌，以適應(yīng)微服務(wù)的場景，詳情查看文章說明（http://dwz.date/cmPV）

[root@node-1 ~]# kubectl logs haproxy-ingress-bdns8 -n ingress-controller -f
I1227 12:21:11.523066       6 controller.go:274] Starting HAProxy update id=20
I1227 12:21:11.561001       6 instance.go:162] HAProxy updated without needing to reload. Commands sent: 3
I1227 12:21:11.561057       6 controller.go:325] Finish HAProxy update id=20: ingress=0.149764ms writeTmpl=37.738947ms total=37.888711ms

4彤蔽、接下來測試負(fù)載均衡的功能庙洼，為了驗(yàn)證測試效果镊辕，往 pod 中寫入不同的內(nèi)容征懈，以測試負(fù)載均衡的效果

[root@node-1 ~]# kubectl exec -it haproxy-ingress-demo-5d487d4fc-5pgjt /bin/bash
root@haproxy-ingress-demo-5d487d4fc-5pgjt:/# echo "web-1" > /usr/share/nginx/html/index.html

[root@node-1 ~]# kubectl exec -it haproxy-ingress-demo-5d487d4fc-pst2q /bin/bash
root@haproxy-ingress-demo-5d487d4fc-pst2q:/# echo "web-2" > /usr/share/nginx/html/index.html

[root@node-1 ~]# kubectl exec -it haproxy-ingress-demo-5d487d4fc-sr8tm /bin/bash
root@haproxy-ingress-demo-5d487d4fc-sr8tm:/# echo "web-3" > /usr/share/nginx/html/index.html

5揩悄、測試驗(yàn)證負(fù)載均衡效果,haproxy 采用輪詢的調(diào)度算法，因此可以明顯看到輪詢效果

[root@node-1 ~]# curl http://www.happylau.cn --resolve www.happylau.cn:80:10.254.100.102
web-1
[root@node-1 ~]# curl http://www.happylau.cn --resolve www.happylau.cn:80:10.254.100.102
web-2
[root@node-1 ~]# curl http://www.happylau.cn --resolve www.happylau.cn:80:10.254.100.102
web-3

這個(gè)章節(jié)驗(yàn)證了 haproxy ingress 控制器動(dòng)態(tài)配置更新的能力亏娜，相比于 nginx ingress 控制器而言蹬挺，haproxy ingress 控制器不需要重載服務(wù)進(jìn)程就能夠動(dòng)態(tài)識(shí)別到配置，在微服務(wù)場景下將具有非常大的優(yōu)勢溯泣；并通過一個(gè)實(shí)例驗(yàn)證了 ingress 負(fù)載均衡調(diào)度能力。

2.3 基于名稱虛擬主機(jī)

這個(gè)小節(jié)將演示 haproxy ingress 基于虛擬云主機(jī)功能的實(shí)現(xiàn)发乔，定義兩個(gè)虛擬主機(jī) news.happylau.cn 和 sports.happylau.cn雪猪，將請求各自轉(zhuǎn)發(fā)至 haproxy-1 和haproxy-2
1、 準(zhǔn)備環(huán)境測試環(huán)境译仗，創(chuàng)建兩個(gè)應(yīng)用 haproxy-1 和 haproxy 并暴露服務(wù)端口

[root@node-1 ~]# cat haproxy-ingress-nginx-test-1.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: haproxy-1
  name: haproxy-1
spec:
  replicas: 3
  selector:
    matchLabels:
      app: haproxy-1
  template:
    metadata:
      labels:
        app: haproxy-1
    spec:
      containers:
      - image: nginx:1.7.9
        name: nginx

---
apiVersion: v1
kind: Service
metadata:
  name: haproxy-1
  namespace: default
spec:
  clusterIP: 10.109.197.67
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: haproxy-1
  type: ClusterIP

[root@node-1 ~]# cat haproxy-ingress-nginx-test-2.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: haproxy-2
  name: haproxy-2
spec:
  replicas: 3
  selector:
    matchLabels:
      app: haproxy-2
  template:
    metadata:
      labels:
        app: haproxy-2
    spec:
      containers:
      - image: nginx:1.7.9
        name: nginx

---
apiVersion: v1
kind: Service
metadata:
  name: haproxy-2
  namespace: default
spec:
  clusterIP: 10.109.197.68
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: haproxy-2
  type: ClusterIP


查看應(yīng)用
[root@node-1 ~]# kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
haproxy-1              1/1     1            1           39s
haproxy-2              1/1     1            1           36s

查看service
[root@node-1 ~]# kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
haproxy-1              ClusterIP 10.109.197.67   <none>        80/TCP 55s
haproxy-2              ClusterIP 10.109.197.68    <none>        80/TCP 52s

2纵菌、定義 ingress 規(guī)則休涤，定義不同的主機(jī)并將請求轉(zhuǎn)發(fā)至不同的 service 中

[root@node-1 ~]# cat ingress-virtualhost.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: haproxy-ingress-virtualhost
  annotations:
    kubernetes.io/ingress.class: haproxy
spec:
  rules:
  - host: news.happylau.cn    
    http:
      paths:
      - path: /
        backend:
          serviceName: haproxy-1
          servicePort: 80
  - host: sports.happylau.cn 
    http:
      paths:
      - path: /
        backend:
          serviceName: haproxy-2
          servicePort: 80

#應(yīng)用ingress規(guī)則并查看列表
[root@node-1 haproxy-ingress]# kubectl apply -f ingress-virtualhost.yaml
ingress.extensions/haproxy-ingress-virtualhost created
[root@node-1 haproxy-ingress]# kubectl get ingresses haproxy-ingress-virtualhost
NAME HOSTS ADDRESS PORTS AGE
haproxy-ingress-virtualhost news.happylau.cn,sports.happylau.cn             80      8s

查看ingress規(guī)則詳情
[root@node-1 haproxy-ingress]# kubectl describe ingresses haproxy-ingress-virtualhost
Name: haproxy-ingress-virtualhost
Namespace: default
Address:          
Default backend: default-http-backend:80 (<none>)
Rules:
  Host Path Backends
  ---- ---- --------
  news.happylau.cn    
                      / haproxy-1:80 (10.244.2.168:80)
  sports.happylau.cn  
                      / haproxy-2:80 (10.244.2.169:80)
Annotations:
  kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"haproxy"},"name":"haproxy-ingress-virtualhost","namespace":"default"},"spec":{"rules":[{"host":"news.happylau.cn","http":{"paths":[{"backend":{"serviceName":"haproxy-1","servicePort":80},"path":"/"}]}},{"host":"sports.happylau.cn","http":{"paths":[{"backend":{"serviceName":"haproxy-2","servicePort":80},"path":"/"}]}}]}}

  kubernetes.io/ingress.class:  haproxy
Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Normal CREATE 37s ingress-controller Ingress default/haproxy-ingress-virtualhost
  Normal CREATE 37s ingress-controller Ingress default/haproxy-ingress-virtualhost
  Normal UPDATE 20s ingress-controller Ingress default/haproxy-ingress-virtualhost
  Normal UPDATE 20s ingress-controller Ingress default/haproxy-ingress-virtualhost

3功氨、測試驗(yàn)證虛擬機(jī)主機(jī)配置，通過 curl 直接解析的方式忱详，或者通過寫 hosts 文件
4跺涤、查看配置配置文件內(nèi)容监透，配置中更新了 haproxy.cfg 的 front 段和 backend 段的內(nèi)容

/etc/haproxy/haproxy.cfg 配置文件內(nèi)容
backend default_haproxy-1_80 #haproxy-1后端
    mode http
    balance roundrobin
    acl https-request ssl_fc
    http-request set-header X-Original-Forwarded-For %[hdr(x-forwarded-for)] if { hdr(x-forwarded-for) -m found }
    http-request del-header x-forwarded-for
    option forwardfor
    http-response set-header Strict-Transport-Security "max-age=15768000"
    server srv001 10.244.2.168:80 weight 1 check inter 2s
    server srv002 127.0.0.1:1023 disabled weight 1 check inter 2s
    server srv003 127.0.0.1:1023 disabled weight 1 check inter 2s
    server srv004 127.0.0.1:1023 disabled weight 1 check inter 2s
    server srv005 127.0.0.1:1023 disabled weight 1 check inter 2s
    server srv006 127.0.0.1:1023 disabled weight 1 check inter 2s
    server srv007 127.0.0.1:1023 disabled weight 1 check inter 2s

#haproxy-2后端
backend default_haproxy-2_80
    mode http
    balance roundrobin
    acl https-request ssl_fc
    http-request set-header X-Original-Forwarded-For %[hdr(x-forwarded-for)] if { hdr(x-forwarded-for) -m found }
    http-request del-header x-forwarded-for
    option forwardfor
    http-response set-header Strict-Transport-Security "max-age=15768000"
    server srv001 10.244.2.169:80 weight 1 check inter 2s
    server srv002 127.0.0.1:1023 disabled weight 1 check inter 2s
    server srv003 127.0.0.1:1023 disabled weight 1 check inter 2s
    server srv004 127.0.0.1:1023 disabled weight 1 check inter 2s
    server srv005 127.0.0.1:1023 disabled weight 1 check inter 2s
    server srv006 127.0.0.1:1023 disabled weight 1 check inter 2s
    server srv007 127.0.0.1:1023 disabled weight 1 check inter 2s

配置關(guān)聯(lián)內(nèi)容
/ # cat /etc/haproxy/maps/_global_http_front.map 
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# #
# # HAProxy Ingress Controller
# # --------------------------
# # This file is automatically updated, do not edit
# #
#
news.happylau.cn/ default_haproxy-1_80
sports.happylau.cn/ default_haproxy-2_80

2.4 URL 自動(dòng)跳轉(zhuǎn)

haproxy ingress 支持自動(dòng)跳轉(zhuǎn)的能力胀蛮，需要通過 annotations 定義佛点，通過ingress.kubernetes.io/ssl-redirect 設(shè)置即可，默認(rèn)為 false鸳玩，設(shè)置為 true 即可實(shí)現(xiàn) http 往 https 跳轉(zhuǎn)的能力演闭，當(dāng)然可以將配置寫入到 ConfigMap 中實(shí)現(xiàn)默認(rèn)跳轉(zhuǎn)的能力，本文以編寫 annotations 為例米碰，實(shí)現(xiàn)訪問 http 跳轉(zhuǎn) https 的能力。
1虐译、定義 ingress 規(guī)則吴趴，設(shè)置 ingress.kubernetes.io/ssl-redirect 實(shí)現(xiàn)跳轉(zhuǎn)功能

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: haproxy-ingress-virtualhost
  annotations:
    kubernetes.io/ingress.class: haproxy
    ingress.kubernetes.io/ssl-redirect: true #實(shí)現(xiàn)跳轉(zhuǎn)功能
spec:
  rules:
  - host: news.happylau.cn
    http:
      paths:
      - path: /
        backend:
          serviceName: haproxy-1
          servicePort: 80
  - host: sports.happylau.cn
    http:
      paths:
      - path: /
        backend:
          serviceName: haproxy-2
          servicePort: 80

按照上圖測試了一下功能，未能實(shí)現(xiàn)跳轉(zhuǎn)實(shí)現(xiàn)跳轉(zhuǎn)的功能厢拭，開源版本中未能找到更多文檔說明，企業(yè)版由于鏡像需要認(rèn)證授權(quán)下載供鸠，未能進(jìn)一步做測試驗(yàn)證陨闹。

2.5 基于 TLS 加密

haproxy ingress 默認(rèn)集成了一個(gè)
1、生成自簽名證書和私鑰

[root@node-1 haproxy-ingress]# openssl req -x509 -newkey rsa:2048 -nodes -days 365 -keyout tls.key -out tls.crt
Generating a 2048 bit RSA private key
...........+++
.......+++
writing new private key to 'tls.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:CN
State or Province Name (full name) []:GD
Locality Name (eg, city) [Default City]:ShenZhen
Organization Name (eg, company) [Default Company Ltd]:Tencent
Organizational Unit Name (eg, section) []:HappyLau
Common Name (eg, your name or your server's hostname) []:www.happylau.cn
Email Address []:573302346@qq.com

2寨闹、創(chuàng)建 Secrets觅廓，關(guān)聯(lián)證書和私鑰

[root@node-1 haproxy-ingress]# kubectl create secret tls haproxy-tls --cert=tls.crt --key=tls.key
secret/haproxy-tls created

[root@node-1 haproxy-ingress]# kubectl describe secrets haproxy-tls
Name: haproxy-tls
Namespace: default
Labels:       <none>
Annotations:  <none>

Type: kubernetes.io/tls

Data
====
tls.crt:  1424 bytes
tls.key: 1704 bytes

3杈绸、編寫 ingress 規(guī)則帖蔓，通過 tls 關(guān)聯(lián) Secrets

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: haproxy-ingress-virtualhost
  annotations:
    kubernetes.io/ingress.class: haproxy
spec:
  tls:
  - hosts:
    - news.happylau.cn
    - sports.happylau.cn
    secretName: haproxy-tls
  rules:
  - host: news.happylau.cn
    http:
      paths:
      - path: /
        backend:
          serviceName: haproxy-1
          servicePort: 80
  - host: sports.happylau.cn
    http:
      paths:
      - path: /
        backend:
          serviceName: haproxy-2
          servicePort: 80

4塑娇、應(yīng)用配置并查看詳情,在 TLS 中可以看到 TLS 關(guān)聯(lián)的證書

[root@node-1 haproxy-ingress]# kubectl apply -f ingress-virtualhost.yaml
ingress.extensions/haproxy-ingress-virtualhost configured

[root@node-1 haproxy-ingress]# kubectl describe ingresses haproxy-ingress-virtualhost
Name: haproxy-ingress-virtualhost
Namespace: default
Address:          
Default backend: default-http-backend:80 (<none>)
TLS:
  haproxy-tls terminates news.happylau.cn,sports.happylau.cn
Rules:
  Host Path Backends
  ---- ---- --------
  news.happylau.cn    
                      / haproxy-1:80 (10.244.2.168:80)
  sports.happylau.cn  
                      / haproxy-2:80 (10.244.2.169:80)
Annotations:
  kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"haproxy"},"name":"haproxy-ingress-virtualhost","namespace":"default"},"spec":{"rules":[{"host":"news.happylau.cn","http":{"paths":[{"backend":{"serviceName":"haproxy-1","servicePort":80},"path":"/"}]}},{"host":"sports.happylau.cn","http":{"paths":[{"backend":{"serviceName":"haproxy-2","servicePort":80},"path":"/"}]}}],"tls":[{"hosts":["news.happylau.cn","sports.happylau.cn"],"secretName":"haproxy-tls"}]}}

  kubernetes.io/ingress.class:  haproxy
Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Normal CREATE 37m               ingress-controller Ingress default/haproxy-ingress-virtualhost
  Normal CREATE 37m               ingress-controller Ingress default/haproxy-ingress-virtualhost
  Normal UPDATE 7s (x2 over 37m) ingress-controller Ingress default/haproxy-ingress-virtualhost
  Normal UPDATE 7s (x2 over 37m) ingress-controller Ingress default/haproxy-ingress-virtualhost

5埋酬、測試 https 站點(diǎn)訪問烧栋，可以看到安全的 https 訪問

三、寫在最后

haproxy 實(shí)現(xiàn) ingress 實(shí)際是通過配置更新 haproxy.cfg 配置珍特，結(jié)合 service 的服務(wù)發(fā)現(xiàn)機(jī)制動(dòng)態(tài)完成 ingress 接入，相比于 nginx 來說扎筒，haproxy 不需要重載實(shí)現(xiàn)配置變更酬姆。在測試 haproxy ingress 過程中，有部分功能配置驗(yàn)證沒有達(dá)到預(yù)期辞色，更豐富的功能支持在 haproxy ingress 企業(yè)版中支持，社區(qū)版能支持藍(lán)綠發(fā)布和 WAF 安全掃描功能诱篷，詳情可以參考社區(qū)文檔 haproxy 藍(lán)綠發(fā)布和 WAF 安全支持雳灵。

haproxy ingress 控制器目前在社區(qū)活躍度一般，相比于 nginx悯辙，traefik，istio 還有一定的差距针贬，實(shí)際環(huán)境中不建議使用社區(qū)版的 haproxy ingress。

四桦他、參考文檔

官方安裝文檔：https://haproxy-ingress.github.io/docs/getting-started/
haproxy ingress 官方配置：https://www.haproxy.com/documentation/hapee/1-7r2/traffic-management/k8s-image-controller/
博客參考文檔：https://cloud.tencent.com/developer/article/1564819

五谆棱、附錄配置文件詳解

#RBAC認(rèn)證賬號(hào)圆仔，和角色關(guān)聯(lián)
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ingress-controller
  namespace: ingress-controller
---
# 集群角色坪郭，訪問資源對象和具體訪問權(quán)限
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: ingress-controller
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses/status
    verbs:
      - update

---
#角色定義
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: ingress-controller
  namespace: ingress-controller
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get
      - create
      - update

---
#集群角色綁定ServiceAccount和ClusterRole關(guān)聯(lián)
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: ingress-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ingress-controller
subjects:
  - kind: ServiceAccount
    name: ingress-controller
    namespace: ingress-controller
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: ingress-controller

---
#角色綁定
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: ingress-controller
  namespace: ingress-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: ingress-controller
subjects:
  - kind: ServiceAccount
    name: ingress-controller
    namespace: ingress-controller
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: ingress-controller

---
#后端應(yīng)用的service定義
apiVersion: v1
kind: Service
metadata:
  name: ingress-default-backend
  namespace: ingress-controller
spec:
  ports:
  - port: 8080
  selector:
    run: ingress-default-backend

---
#haproxy ingress配置歪沃，實(shí)現(xiàn)自定義配置功能
apiVersion: v1
kind: ConfigMap
metadata:
  name: haproxy-ingress
  namespace: ingress-controller

---
#haproxy ingress核心的DaemonSet
apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    run: haproxy-ingress
  name: haproxy-ingress
  namespace: ingress-controller
spec:
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      run: haproxy-ingress
  template:
    metadata:
      labels:
        run: haproxy-ingress
    spec:
      hostNetwork: true #網(wǎng)絡(luò)模式為hostNetwork,即使用宿主機(jī)的網(wǎng)絡(luò)
      nodeSelector: #節(jié)點(diǎn)選擇器嫌松，將調(diào)度至包含特定標(biāo)簽的節(jié)點(diǎn)
        role: ingress-controller
      serviceAccountName: ingress-controller #實(shí)現(xiàn)RBAC認(rèn)證授權(quán)
      containers:
      - name: haproxy-ingress
        image: quay.io/jcmoraisjr/haproxy-ingress
        args:
        - --default-backend-service=$(POD_NAMESPACE)/ingress-default-backend
        - --configmap=$(POD_NAMESPACE)/haproxy-ingress
        - --sort-backends
        ports:
        - name: http
          containerPort: 80
        - name: https
          containerPort: 443
        - name: stat
          containerPort: 1936
        livenessProbe:
          httpGet:
            path: /healthz
            port: 10253
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace

出處：http://dwz.date/cmQf
可聯(lián)系本人vx:17812796384

?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請聯(lián)系作者
  • 人面猴
    序言：七十年代末，一起剝皮案震驚了整個(gè)濱河市珊蟀，隨后出現(xiàn)的幾起案子外驱，更是在濱河造成了極大的恐慌，老刑警劉巖昵宇，帶你破解...
    沈念sama閱讀 206,126評論 6贊 481
  • 死咒
    序言：濱河連續(xù)發(fā)生了三起死亡事件，死亡現(xiàn)場離奇詭異砸喻，居然都是意外死亡，警方通過查閱死者的電腦和手機(jī)割岛，發(fā)現(xiàn)死者居然都...
    沈念sama閱讀 88,254評論 2贊 382
  • 救了他兩次的神仙讓他今天三更去死
    文/潘曉璐 我一進(jìn)店門犯助，熙熙樓的掌柜王于貴愁眉苦臉地迎上來，“玉大人剂买，你說我怎么就攤上這事』樗粒” “怎么了？”我有些...
    開封第一講書人閱讀 152,445評論 0贊 341
  • 道士緝兇錄：失蹤的賣姜人
    文/不壞的土叔 我叫張陵较性，是天一觀的道長。 經(jīng)常有香客問我永毅，道長人弓，這世上最難降的妖魔是什么着逐？ 我笑而不...
    開封第一講書人閱讀 55,185評論 1贊 278
  • ?港島之戀（遺憾婚禮）
    正文 為了忘掉前任，我火速辦了婚禮健芭，結(jié)果婚禮上，老公的妹妹穿的比我還像新娘慈迈。我一直安慰自己省有，他們只是感情好，可當(dāng)我...
    茶點(diǎn)故事閱讀 64,178評論 5贊 371
  • 惡毒庶女頂嫁案：這布局不是一般人想出來的
    文/花漫 我一把揭開白布伸头。 她就那樣靜靜地躺著，像睡著了一般恤磷。 火紅的嫁衣襯著肌膚如雪。 梳的紋絲不亂的頭發(fā)上扫步，一...
    開封第一講書人閱讀 48,970評論 1贊 284
  • 城市分裂傳說
    那天匈子，我揣著相機(jī)與錄音，去河邊找鬼旬牲。 笑死，一個(gè)胖子當(dāng)著我的面吹牛原茅，可吹牛的內(nèi)容都是我干的。 我是一名探鬼主播晌区，決...
    沈念sama閱讀 38,276評論 3贊 399
  • 雙鴛鴦連環(huán)套：你想象不到人心有多黑
    文/蒼蘭香墨 我猛地睜開眼，長吁一口氣：“原來是場噩夢啊……” “哼朗若！你這毒婦竟也來了？” 一聲冷哼從身側(cè)響起灾馒，我...
    開封第一講書人閱讀 36,927評論 0贊 259
  • 萬榮殺人案實(shí)錄
    序言：老撾萬榮一對情侶失蹤遣总，失蹤者是張志新（化名）和其女友劉穎，沒想到半個(gè)月后旭斥，有當(dāng)?shù)厝嗽跇淞掷锇l(fā)現(xiàn)了一具尸體，經(jīng)...
    沈念sama閱讀 43,400評論 1贊 300
  • ?護(hù)林員之死
    正文 獨(dú)居荒郊野嶺守林人離奇死亡花盐，尸身上長有42處帶血的膿包…… 初始之章·張勛 以下內(nèi)容為張勛視角 年9月15日...
    茶點(diǎn)故事閱讀 35,883評論 2贊 323
  • ?白月光啟示錄
    正文 我和宋清朗相戀三年菇爪，在試婚紗的時(shí)候發(fā)現(xiàn)自己被綠了。 大學(xué)時(shí)的朋友給我發(fā)了我未婚夫和他白月光在一起吃飯的照片凳宙。...
    茶點(diǎn)故事閱讀 37,997評論 1贊 333
  • 活死人
    序言：一個(gè)原本活蹦亂跳的男人離奇死亡，死狀恐怖诈嘿，靈堂內(nèi)的尸體忽然破棺而出，到底是詐尸還是另有隱情奖亚，我是刑警寧澤，帶...
    沈念sama閱讀 33,646評論 4贊 322
  • ?日本核電站爆炸內(nèi)幕
    正文 年R本政府宣布昔字，位于F島的核電站首繁，受9級特大地震影響，放射性物質(zhì)發(fā)生泄漏弦疮。R本人自食惡果不足惜，卻給世界環(huán)境...
    茶點(diǎn)故事閱讀 39,213評論 3贊 307
  • 男人毒藥：我在死后第九天來索命
    文/蒙蒙 一咏尝、第九天 我趴在偏房一處隱蔽的房頂上張望压语。 院中可真熱鬧胎食，春花似錦、人聲如沸厕怜。這莊子的主人今日做“春日...
    開封第一講書人閱讀 30,204評論 0贊 19
  • 一樁弒父案，背后竟有這般陰謀
    文/蒼蘭香墨 我抬頭看了看天上的太陽谤专。三九已至午绳，卻和暖如春，著一層夾襖步出監(jiān)牢的瞬間拦焚，已是汗流浹背。 一陣腳步聲響...
    開封第一講書人閱讀 31,423評論 1贊 260
  • 情欲美人皮
    我被黑心中介騙來泰國打工秕衙， 沒想到剛下飛機(jī)就差點(diǎn)兒被人妖公主榨干…… 1. 我叫王不留，地道東北人据忘。 一個(gè)月前我還...
    沈念sama閱讀 45,423評論 2贊 352
  • 代替公主和親
    正文 我出身青樓搞糕，卻偏偏與公主長得像，于是被迫代替她去往敵國和親窍仰。 傳聞我的和親對象是個(gè)殘疾皇子，可洞房花燭夜當(dāng)晚...
    茶點(diǎn)故事閱讀 42,722評論 2贊 345