關(guān)鍵字： k8s（kubernetes），docker鞠呈，kubelet,kubeadm,kubectl
環(huán)境描述
-
安裝方式概述
- k8s 組件為鏡像安裝
- yum 安裝kubelet,kubeadm,kubectl,利用kubeamd 進(jìn)行初始化集群，加入主節(jié)點(diǎn)和從節(jié)點(diǎn)
- 利用keepalived + HAproxy 實(shí)現(xiàn)負(fù)載
主機(jī)環(huán)境描述
| IP | 節(jié)點(diǎn) | 安裝服務(wù) |
|---|---|---|
| 172.16.0.105 | master-1 | docker-ce19.03.5徙邻，etcd3.3.12搁嗓，kubernetes1.15.1 |
| 172.16.0.106 | master-2 | docker-ce19.03.5，etcd3.3.12兜叨，kubernetes1.15.1 |
| 172.16.0.136 | master-3 | docker-ce19.03.5穿扳，etcd3.3.12衩侥，kubernetes1.15.1 |
| 172.16.0.109 | k8s-node-1 | kubernetes1.15.1 |
| 172.16.0.110 | k8s-node-2 | kubernetes1.15.1 |
| IP | 節(jié)點(diǎn) | 安裝服務(wù) |
|---|---|---|
| 172.16.0.129 | keepalived+haproxy | keepalived2.0.18,haproxy2.0.7 |
| 172.16.0.130 | keepalived+haproxy | keepalived2.0.18,haproxy2.0.7 |
| VIP | 作用 |
|---|---|
| 172.16.0.202 | k8s集群的apiserver代理 |
部署用戶：app
安裝步驟
keepalived+haproxy安裝 -keepalived + haproxy 節(jié)點(diǎn)操作
1国旷、安裝keepalived + haproxy (這里為yum 安裝-可選更新yum源再安裝高版本)
sudo yum install -y keepalived haproxy
2、配置HA
# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
vrrp_script check_haproxy {
script "/etc/keepalived/check_haproxy.sh"
interval 3
weight -20
}
vrrp_instance K8S {
state backup
interface ens192 #ip addr 查看網(wǎng)卡名稱
virtual_router_id 44
priority 200 #master1 200; master2 190; master3 180 其他配置一樣
advert_int 5
authentication {
auth_type PASS
auth_pass kubernetes
}
virtual_ipaddress {
172.16.0.202
}
track_script {
check_haproxy
}
}
檢測(cè)腳本： sudo vim /etc/keepalived/check_haproxy.sh
#!/bin/bash
active_status=`netstat -lntp|grep haproxy|wc -l`
if [ $active_status -gt 0 ]; then
exit 0
else
/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg
sleep 2
if [ `ps -C haproxy --no-header |wc -l` -eq 0 ];then
killall5 keepalived
fi
fi
賦予執(zhí)行權(quán)限 chmod u+x /etc/keepalived/check_haproxy.sh
3茫死、配置haproxy sudo vim /etc/haproxy/haproxy
global
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user app
group app
daemon
stats socket /var/lib/haproxy/stats
defaults
mode tcp
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
frontend k8s_apiserver
mode tcp
bind *:6443
default_backend k8s-master
backend k8s-master
mode tcp
balance roundrobin
server k8s-master-1 172.16.0.105:6443 check # 三個(gè)master主機(jī)
server k8s-master-2 172.16.0.106:6443 check
server k8s-master-3 172.16.0.136:6443 check
4跪但、重定向keepalived haproxy日志
- keepalived：
sudo vim /etc/sysconfig/keepalived
將KEEPALIVED_OPTIONS="-D" 修改
KEEPALIVED_OPTIONS="-D -d -S 4"
- sudo vim /etc/rsyslog.conf添加如下內(nèi)容
local2.* /data/logs/haproxy.log
local4.* /data/logs/keepalived.log
- 創(chuàng)建日志路徑：mkdir /data/logs
- 重啟rsyslog 服務(wù)： sudo systemctl restart rsyslog
5、啟動(dòng)keepalived haproxy
- keepalived: sudo systemctl start keepalived
- haproxy: sudo haproxy -f /etc/haproxy/haproxy.cfg
6峦萎、驗(yàn)證查看keepalived 日志屡久，關(guān)閉主上haproxy看是否能正常切換到從
k8s集群搭建
1、 環(huán)境準(zhǔn)備 -沒(méi)有特別備注則master,node節(jié)點(diǎn)操作
- 關(guān)閉firewlld: sudo systemctl stop firewall (centos7) sudo service firewalld stop(centos 6)
- 關(guān)閉swap: sudo swapoff -a
- 直接設(shè)置內(nèi)核參數(shù)：
vim /etc/sysctl.conf
net.ipv4.ip_nonlocal_bind = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_port_range = 10000 65000
fs.file-max = 2000000
net.ipv4.ip_forward = 1
vm.swappiness = 0
- 使其生效
sysctl -p
- 創(chuàng)建所需目錄
sudo mkdir -p /opt/kubernetes/{cfg,ssl,bin}
chown app:app -R /opt/kubernets ---根據(jù)實(shí)際環(huán)境環(huán)境用戶修改
- 添加/etc/hosts
172.16.0.105 master-1
172.16.0.106 master-2
172.16.0.136 master-3
172.16.0.109 k8s-node-1
172.16.0.110 k8s-node-2
- 安裝docker-ce
安裝依賴：sudo yum install -y yum-utils device-mapper-persisren-data lvm2
添加docker軟件包源：sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
更新yum包索引：sudo yum makecache fast
安裝docker CE : sudo yum install -y docker-ce
利用國(guó)內(nèi)的源提供下載速度：sudo curl -sSL https://get.daocloud.io/daotools/set_mirror.sh|sh -s http://04be47cf.m.daocloud.io
啟動(dòng)：sudo systemctl start docker
sudo systemctl enable docker
- 下載cfssl - master-1操作
cd /opt/kubernetes/bin
curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o /opt/kubernetes/bin/cfssl
curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o /opt/kubernetes/bin/cfssljson
curl -L https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -o /opt/kubernetes/bin/cfssl-certinfo
chmod u+x ./*
- 添加環(huán)境變量 - master-1操作
echo 'PATH=$PATH:/opt/kubernetes/bin'>>/etc/profile
echo 'PATH=$PATH:/opt/kubernetes/bin'>>~/.bash_profile
. /etc/profile
. ~/.bash_profile
2爱榔、 制作CA,ETCD證書
cd /opt/kubernetes/ssl
vim ca-csr.json
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
vim ca-config.json
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"kubernetes": {
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
],
"expiry": "87600h"
}
}
}
}
vim etcd-csr.json
{
"CN": "etcd",
"hosts": [
"127.0.0.1",
"172.16.0.105",
"172.16.0.106",
"172.16.0.136",
"172.16.0.202"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
- 生成key文件
cd /opt/kubernetes/ssl
cfssl gencert -initca ca-csr.json | cfssljson -bare ca # 生成ca key文件
cfssl gencert -ca=ca.pem \
-ca-key=ca-key.pem \
-config=ca-config.json \
-profile=kubernetes etcd-csr.json | cfssljson -bare etcd #生成etcd key文件
- 將生成的證書分發(fā)到其他主節(jié)點(diǎn)
scp -r /opt/kubernetes/ssl app@172.16.0.106:/opt/kubernetes/ssl
scp -r /opt/kubernetes/ssl app@172.16.0.136:/opt/kubernetes/ssl
3被环、etcd集群安裝
- 安裝etcd,master節(jié)點(diǎn)集群同時(shí)操作
解壓安裝包，放置/data/package
cd /data/package/etcd-v3.3.12-linux-amd64
cp -a etcd* /opt/kubernetes/bin
- 配置etcd配置文件--在3臺(tái)主節(jié)點(diǎn)同時(shí)操作
#[member]
ETCD_NAME="etcd-node-1"
ETCD_DATA_DIR="/var/lib/etcd/"
#ETCD_SNAPSHOT_COUNTER="10000"
#ETCD_HEARTBEAT_INTERVAL="100"
#ETCD_ELECTION_TIMEOUT="1000"
ETCD_LISTEN_PEER_URLS="https://172.16.0.105:2380"
ETCD_LISTEN_CLIENT_URLS="https://172.16.0.105:2379,https://127.0.0.1:2379"
#ETCD_MAX_SNAPSHOTS="5"
#ETCD_MAX_WALS="5"
#ETCD_CORS=""
#[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.16.0.105:2380"
# if you use different ETCD_NAME (e.g. test),
# set ETCD_INITIAL_CLUSTER value for this name, i.e. "test=http://..."
ETCD_INITIAL_CLUSTER="etcd-node-1=https://172.16.0.105:2380,etcd-node-2=https://172.16.0.106:2380,etcd-node-3=https://172.16.0.136:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="k8s-etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="https://172.16.0.105:2379"
#[security]
CLIENT_CERT_AUTH="true"
ETCD_CA_FILE="/opt/kubernetes/ssl/ca.pem"
ETCD_CERT_FILE="/opt/kubernetes/ssl/etcd.pem"
ETCD_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem"
PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_CA_FILE="/opt/kubernetes/ssl/ca.pem"
ETCD_PEER_CERT_FILE="/opt/kubernetes/ssl/etcd.pem"
ETCD_PEER_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem"
- 創(chuàng)建etcd的服務(wù)器啟動(dòng)文件 --3臺(tái)主節(jié)點(diǎn)同時(shí)操作
sudo vim /usr/lib/systemd/system/etcd.service
[Unit]
Description=etcd
Documentation=https://github.com/coreos/etcd
Conflicts=etcd.service
Conflicts=etcd2.service
[Service]
Type=notify
Restart=always
RestartSec=5s
LimitNOFILE=40000
TimeoutStartSec=0
WorkingDirectory=/var/lib/etcd
EnvironmentFile=-/opt/kubernetes/cfg/etcd.conf
# set GOMAXPROCS to number of processors
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /opt/kubernetes/bin/etcd"
[Install]
WantedBy=multi-user.target
- 啟動(dòng)etcd集群
[app@master01~]$ sudo mkdir -p /var/lib/etcd/default.etcd
[app@master01~]$ sudo systemctl daemon-reload
[app@master01~]$ sudo systemctl start etcd
[app@master01~]$ sudo systemctl enable etcd
- 驗(yàn)證集群
etcdctl --endpoints=https://172.16.0.105:2379 \
--ca-file=/opt/kubernetes/ssl/ca.pem \
--cert-file=/opt/kubernetes/ssl/etcd.pem \
--key-file=/opt/kubernetes/ssl/etcd-key.pem cluster-health
輸出如下為搭建正常：
member 62433e04590b6324 is healthy: got healthy result from https://172.16.0.106:2379
member 775c242b93af59c7 is healthy: got healthy result from https://172.16.0.105:2379
member 9359be5f6c73cdb8 is healthy: got healthy result from https://172.16.0.136:2379
cluster is healthy
4详幽、kubernetes master節(jié)點(diǎn)安裝
- 安裝kubelet,kubeadm,kubectl
配置阿里鏡像源
sudo vim /etc/yum.repos.d/kubernets.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
sudo yum install -y kubelet-1.15.1 kubeadm-1.15.1 kubectl-1.15.1
- 下載相關(guān)鏡像拉取與裝載 --教程可參考https://www.cnblogs.com/baylorqu/p/10754924.html master操作
- 我這邊直接使用別人下載好的鏡像上傳安裝：https://pan.baidu.com/s/12T4Su-KxAlq-qYXD7zOCeQ#list/path=%2F 提取碼： z6dg
- ll /data/k8s-images
total 617528
-rw-r--r-- 1 app app 40542720 Dec 23 15:01 coredns_1.3.1.tar
-rw-r--r-- 1 app app 55390720 Dec 23 15:09 flannel_v0.11.0-amd64.tar
-rw-r--r-- 1 app app 208394752 Dec 23 15:01 kube-apiserver_v1.15.1.tar
-rw-r--r-- 1 app app 160290304 Dec 23 15:01 kube-controller-manager_v1.15.1.tar
-rw-rw-r-- 1 app app 84282368 Dec 23 15:42 kube-proxy_v1.15.1.tar
-rw-r--r-- 1 app app 82675200 Dec 23 15:00 kube-scheduler_v1.15.1.tar
-rw-rw-r-- 1 app app 754176 Dec 23 15:42 pause_3.1.tar
裝載鏡像： for i in *;do sudo docker load < $i;done
- 下載修改kubeadm-conf.yaml 和 kube-flannel.yml 配置文件 --master-1 操作 可在上個(gè)網(wǎng)盤下載
cd /opt/kubernetes/cfg
vim kubeadm-conf.yaml
apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
token: abcdef.0123456789abcdef
ttl: 24h0m0s
usages:
- signing
- authentication
kind: InitConfiguration
---
apiServer:
timeoutForControlPlane: 4m0s
certSANs:
- 172.16.0.202
- 172.16.0.105
- 172.16.0.106
- 172.16.0.136
- "master-1"
- "master-2"
- "master-3"
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns:
type: CoreDNS
etcd:
external:
endpoints:
- https://172.16.0.105:2379
- https://172.16.0.106:2379
- https://172.16.0.136:2379
caFile: /opt/kubernetes/ssl/ca.pem
certFile: /opt/kubernetes/ssl/etcd.pem
keyFile: /opt/kubernetes/ssl/etcd-key.pem
imageRepository: k8s.gcr.io
kind: ClusterConfiguration
kubernetesVersion: v1.15.1
controlPlaneEndpoint: "172.16.0.202:6443"
networking:
dnsDomain: cluster.local
serviceSubnet: 10.92.0.0/16
podSubnet: 10.2.0.0/16
scheduler: {}
vim kube-flannel.yml
將此處Network 后的ip修改為上個(gè)配置文件定義的pod 的ip網(wǎng)段
net-conf.json: |
{
"Network": "10.2.0.0/16",
"Backend": {
"Type": "vxlan"
}
}
- 初始化主節(jié)點(diǎn)
kubeadm init --config kubeadm-conf.yaml --ignore-preflight-errors=swap
出現(xiàn) mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube
sudo chown $(id-u):$(id-g) $HOME/.kube
kubeadm join 172.16.0.202:6443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:f40bed66ee0ddf93157f1c3f98d7aa9378c6d93ec1c5b4177d9e616bfb43534b \
--control-plane #主節(jié)點(diǎn)加入
kubeadm join 172.16.0.202:6443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:f40bed66ee0ddf93157f1c3f98d7aa9378c6d93ec1c5b4177d9e616bfb43534b #從節(jié)點(diǎn)加入
- master-1 上執(zhí)行如下語(yǔ)句
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
- 啟動(dòng)flannel網(wǎng)絡(luò)
cd /opt/kubernetes/cfg
kubectl create -f kube-flannel.yml
- 添加master-2筛欢，master-3
- 將master-1節(jié)點(diǎn) /etc/kubernetes/pki 拷貝到master-2,master-3
sudo scp -r /etc/kubernetes/pki app@172.16.0.106:/etc/kubernetes/pki
- 主節(jié)點(diǎn)加入集群
sudo kubeadm join 172.16.0.202:6443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:f40bed66ee0ddf93157f1c3f98d7aa9378c6d93ec1c5b4177d9e616bfb43534b \
--control-plane
5浸锨、node節(jié)點(diǎn)加入集群
- 環(huán)境準(zhǔn)備 ：... ...
- 裝載flannel鏡像
- 在master節(jié)點(diǎn)查看加入node節(jié)點(diǎn)命令
sudo kubeadm token create --print-join-command
- 加入node節(jié)點(diǎn)--從節(jié)點(diǎn)操作
sudo kubeadm join 172.16.0.202:6443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:f40bed66ee0ddf93157f1c3f98d7aa9378c6d93ec1c5b4177d9e616bfb43534b
- 查看節(jié)點(diǎn)狀態(tài)-master節(jié)點(diǎn)操作
[app@master-1 cfg]$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-node-1 Ready <none> 153m v1.15.1
k8s-node-2 Ready <none> 151m v1.15.1
master-1 Ready master 167m v1.15.1
master-2 Ready master 159m v1.15.1
master-3 Ready master 157m v1.15.1
dashbord安裝——master-1上操作
- 下載kubernets-dashboard.ymal文件
/data/kubernets-dashboard
wget http://pencil-file.oss-cn-hangzhou.aliyuncs.com/blog/kubernetes-dashboard.yaml
- 修改kubernet-dashboard.ymal中的鏡像源地址和port類型
--修改鏡像源
vim kubernet-dashboard.ymal匹配image將其改為
registry.cn-hangzhou.aliyuncs.com/lynchj/kubernetes-dashboard-amd64:v1.10.1 //版本可根據(jù)實(shí)際去選擇
--匹配Service，將文件中
type: service 修改為
type: NodePort
若無(wú)此項(xiàng)直接添加
如下：
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kube-system
spec:
type: NodePort
ports:
- port: 443
targetPort: 8443
nodePort:30001 #指定訪問(wèn)端口
selector:
k8s-app: kubernetes-dashboard
- 創(chuàng)建kubernet-dashboard pod
kubectl apply -f kubernetes-dashboard.yaml //創(chuàng)建
kubectl get pods --namespace=kube-system //NAME下有kubernet-dashboard
- 創(chuàng)建kubenerts-dash admin 賬號(hào)
vim admin-token.yaml
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: admin
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
roleRef:
kind: ClusterRole
name: cluster-admin
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: admin
namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: admin
namespace: kube-system
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcil
創(chuàng)建用戶：
kubectl create -f admin-token.yaml
- 獲取對(duì)應(yīng)用戶的token
kubectl get secret -n kube-system |grep admin|awk '{print $1}'
輸出結(jié)果：admin-token-f5q68
kubectl describe secret admin-token-f5q68 -n kube-system|grep '^token'|awk '{print $2}'
輸出結(jié)果： admin 賬號(hào)登錄的token
- 登錄
https://172.16.0.105:30001
參考資料
https://blog.csdn.net/qq_31547771/article/details/100699573
https://blog.csdn.net/wangshuminjava/article/details/92783296
