環(huán)境
關閉防火墻吵血，selinux,基于主機名訪問（dns或者dns）,同步時間（時間不一致，集群運行異常偷溺，如不能啟動）,關閉swap分區(qū)
#hosts
10.1.101.202 inspur2.ops.bds.bj1 inspur2
10.1.101.203 inspur2.ops.bds.bj1 inspur3
系統(tǒng)初始化
# 關掉 selinux
setenforce 0
sed -i "s/^SELINUX=enforcing/SELINUX=disabled/g" /etc/sysconfig/selinux
# 關掉防火墻
systemctl stop firewalld
systemctl disable firewalld
#關閉swap
swapoff -a
#開啟轉發(fā)的參數(shù)蹋辅，根據(jù)實際報錯情況開啟，一般有如下三項
cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
#加載內核配置
sysctl --system /etc/sysctl.d/k8s.conf
# 加載ipvs相關內核模塊
# 如果重新開機挫掏，需要重新加載
modprobe ip_vs
modprobe ip_vs_rr
modprobe ip_vs_wrr
modprobe ip_vs_sh
modprobe nf_conntrack_ipv4
lsmod | grep ip_vs
master 配置
docker鏡像
cd /etc/yum.repos.d/
wget https://mirrors.tuna.tsinghua.edu.cn/docker-ce/linux/centos/docker-ce.repo
wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
配置docker鏡像加速
mkdir -p /etc/docker
vim /etc/docker/daemon.json
{
"registry-mirrors": ["http://08b61f14.m.daocloud.io"],
"exec-opts": ["native.cgroupdriver=systemd"]
}
阿里云加速
https://cr.console.aliyun.com/undefined/instances/mirrors
sudo mkdir -p /etc/docker
sudo tee /etc/docker/daemon.json <<-'EOF'
{
"registry-mirrors": ["https://27j02xby.mirror.aliyuncs.com"],
"exec-opts": ["native.cgroupdriver=systemd"]
}
EOF
sudo systemctl daemon-reload
sudo systemctl restart docker
配置拉取gcr.io鏡像
1 通過配置代理來下載k8s鏡像
vim /usr/lib/systemd/system/docker.service
Environment="HTTPS_PROXY=http://www.ik8s.io:10080"
Environment="HTTP_PROXY=http://www.ik8s.io:10080"
Environment="NO_PROXY=127.0.0.0/8,172.20.0.0/16"
#保存退出后侦另，執(zhí)行
systemctl daemon-reload
systemctl enable docker
systemctl start docker
2 技術大牛github拉取鏡像
vim pullimages.sh
#!/bin/bash
images=(kube-proxy-amd64:v1.11.2 kube-scheduler-amd64:v1.11.2 kube-controller-manager-amd64:v1.11.2 kube-apiserver-amd64:v1.11.2
etcd-amd64:3.2.18 coredns:1.1.3 pause:3.1 )
for imageName in ${images[@]} ; do
docker pull anjia0532/google-containers.$imageName
docker tag anjia0532/google-containers.$imageName k8s.gcr.io/$imageName
docker rmi anjia0532/google-containers.$imageName
done
k8s repo鏡像
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
install kubeadm
yum -y install kubeadm kubelet kubectl
kubeadm config
vim /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs"
systemctl enable kubelet.service
systemctl start kubelet
kubelet error log
注意，此時啟動會出現(xiàn)報錯尉共，查看/var/log/messages的日志
tail -f /var/log/messages
如果出現(xiàn)如下的報錯
failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file "/var/lib/kubelet/config.yaml", error: open /var/lib/kubelet/config.yaml: no such file or directory
忽略以上的報錯褒傅，設置為開機自啟動即可，因為此時的配置還沒初始化完成袄友，所以此時不能啟動kubelet,等后續(xù)kubeadm啟動成功后再查看
swap 分區(qū)注意
注意殿托，需要關閉swap分區(qū)，或者在如下的配置文件里修改杠河，表示添加而且的
啟動選項
vim /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS="--fail-swap-on=false"
建議執(zhí)行 swapoff -a 關閉swap分區(qū)碌尔，不用配置上述選項
k8s 初始化
master節(jié)點上用kubeadm命令來初始化集群 這個時候會拉取gcr.io鏡像時間會比較長
kubeadm init --kubernetes-version=v1.11.1 --pod-network-cidr=10.244.0.0/16 --service-cidr=10.96.0.0/12
因為我們這里選擇flannel作為Pod網(wǎng)絡插件，所以需要制定--pod-netwrok-cidr=10.244.0.0/16
初始化過程會生成相關的證書券敌，kubeconfig文件唾戚，bootstraptoken等等
后面是使用kubeadm join往集群中添加節(jié)點時用到的命令，這個命令后面的
--token和--discovery-token-ca-cert-hash一定要保留備份好待诅，后面添加節(jié)點的時候會使用到叹坦，如果丟失也可以通過命里找回，但是比較麻煩卑雁。
配置使用kubectl訪問集群
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
查看master k8s狀態(tài)
#查看組件的健康狀態(tài)
[root@inspur2 ~]# kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Healthy {"health": "true"}
#查看集群中node
[root@inspur2 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
inspur2.ops.haodf.bj1 Ready master 16h v1.11.3
#查看nodespace名稱空間
[root@inspur2 ~]# kubectl get ns
NAME STATUS AGE
default Active 16h
kube-public Active 16h
kube-system Active 16h
install Pod network
master節(jié)點安裝下載
[root@inspur2 ~]# wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
[root@inspur2 ~]# kubectl apply -f kube-flannel.yml
clusterrole.rbac.authorization.k8s.io "flannel" created
clusterrolebinding.rbac.authorization.k8s.io "flannel" created
serviceaccount "flannel" created
configmap "kube-flannel-cfg" created
daemonset.extensions "kube-flannel-ds" created
多網(wǎng)卡配置
如果你的節(jié)點有多個網(wǎng)卡的話,需要在kube-flannel.yml 中使用 --iface 參數(shù)指定集群主機內網(wǎng)網(wǎng)卡的名稱募书，否則可能會出現(xiàn)dns無法解析。 flanneld 啟動參數(shù)加上 --iface=
args:
- --ip-masq
- --kube-subnet-mgr
- --iface=eth0
查看gcr.io鏡像和fannel鏡像
[root@inspur2 ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
k8s.gcr.io/kube-proxy-amd64 v1.11.1 d5c25579d0ff 8 weeks ago 97.8MB
k8s.gcr.io/kube-scheduler-amd64 v1.11.1 272b3a60cd68 8 weeks ago 56.8MB
k8s.gcr.io/kube-controller-manager-amd64 v1.11.1 52096ee87d0e 8 weeks ago 155MB
k8s.gcr.io/kube-apiserver-amd64 v1.11.1 816332bd9d11 8 weeks ago 187MB
k8s.gcr.io/coredns 1.1.3 b3b94275d97c 3 months ago 45.6MB
k8s.gcr.io/etcd-amd64 3.2.18 b8df3b177be2 5 months ago 219MB
quay.io/coreos/flannel v0.10.0-amd64 f0fad859c909 7 months ago 44.6MB
k8s.gcr.io/pause 3.1 da86e6ba6ca1 8 months ago 742kB
[root@inspur2 ~]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-78fcdf6894-m9kwr 1/1 Running 1 16h
coredns-78fcdf6894-tgjn6 1/1 Running 1 16h
etcd-inspur2.ops.haodf.bj1 1/1 Running 1 16h
kube-apiserver-inspur2.ops.haodf.bj1 1/1 Running 1 16h
kube-controller-manager-inspur2.ops.haodf.bj1 1/1 Running 1 16h
kube-flannel-ds-amd64-d8vxk 1/1 Running 5 16h
kube-proxy-24gvj 1/1 Running 1 16h
kube-scheduler-inspur2.ops.haodf.bj1 1/1 Running 1 16h
如果集群遇到問題测蹲，使用下面命令來重置
kubeadm reset
ifconfig cni0 down && ip link delete cni0
ifconfig flannel.1 down && ip link delete flannel.1
rm -rf /var/lib/cni/
node節(jié)點操作
yum倉庫準備好后莹捡，在以下的兩個節(jié)點上執(zhí)行安裝如下包，
yum -y install kubeadm kubelet kubectl docker-ce
系統(tǒng)初始化
# 關掉 selinux
setenforce 0
sed -i "s/^SELINUX=enforcing/SELINUX=disabled/g" /etc/sysconfig/selinux
# 關掉防火墻
systemctl stop firewalld
systemctl disable firewalld
#關閉swap
swapoff -a
#開啟轉發(fā)的參數(shù)扣甲，根據(jù)實際報錯情況開啟篮赢，一般有如下三項
cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
#加載內核配置
sysctl --system /etc/sysctl.d/k8s.conf
start docker
# 設置開啟自啟動
systemctl enable docker.service
systemctl enable kubelet.service
#
systemctl start docker
把配置文件從master拷貝到node節(jié)點 保持配置一致
scp /usr/lib/systemd/system/docker.service 10.1.101.203:/usr/lib/systemd/system/docker.service
scp /etc/systemd/system/kubelet.service.d/10-kubeadm.conf 10.1.101.203:/etc/systemd/system/kubelet.service.d/10-kubeadm.conf
scp /etc/sysconfig/kubelet 10.1.101.203:/etc/sysconfig/
node節(jié)點加入master
kubeadm join 10.1.101.202:6443 --token gt8abw.jaaivoggg2imqnld --discovery-token-ca-cert-hash sha256:2220de69ce7084d672dd1f282246a76c4a5bb04661b2ada23a3014e203eb14b3
node執(zhí)行上面命令成功后，提示信息如下
......
This node has joined the cluster:
* Certificate signing request was sent to master and a response
was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the master to see this node join the cluster.
看到上面信息代表發(fā)布成功
node 節(jié)點容器信息
[root@inspur3 ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
k8s.gcr.io/kube-proxy-amd64 v1.11.1 d5c25579d0ff 8 weeks ago 97.8MB
quay.io/coreos/flannel v0.10.0-amd64 f0fad859c909 7 months ago 44.6MB
k8s.gcr.io/pause 3.1 da86e6ba6ca1 8 months ago 742kB
驗證集群
[root@inspur2 ~]# kubectl get pod -n kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE
coredns-78fcdf6894-m9kwr 1/1 Running 1 16h 10.244.0.5 inspur2.ops.haodf.bj1
coredns-78fcdf6894-tgjn6 1/1 Running 1 16h 10.244.0.4 inspur2.ops.haodf.bj1
etcd-inspur2.ops.haodf.bj1 1/1 Running 1 16h 10.1.101.202 inspur2.ops.haodf.bj1
kube-apiserver-inspur2.ops.haodf.bj1 1/1 Running 1 16h 10.1.101.202 inspur2.ops.haodf.bj1
kube-controller-manager-inspur2.ops.haodf.bj1 1/1 Running 1 16h 10.1.101.202 inspur2.ops.haodf.bj1
kube-flannel-ds-amd64-d8vxk 1/1 Running 7 16h 10.1.101.202 inspur2.ops.haodf.bj1
kube-flannel-ds-amd64-lh2jf 1/1 Running 0 5m 10.1.101.203 inspur3.ops.haodf.bj1
kube-proxy-24gvj 1/1 Running 1 16h 10.1.101.202 inspur2.ops.haodf.bj1
kube-proxy-n29ch 1/1 Running 0 5m 10.1.101.203 inspur3.ops.haodf.bj1
kube-scheduler-inspur2.ops.haodf.bj1 1/1 Running 1 16h 10.1.101.202 inspur2.ops.haodf.bj1
以上信息有這個 inspur3.ops.haodf.bj1從節(jié)點的信息琉挖，flannel和proxy都有兩個pod
[root@inspur2 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
inspur2.ops.bds.bj1 Ready master 16h v1.11.3
inspur3.ops.bds.bj1 Ready <none> 4m v1.11.3
忘記初始master節(jié)點時的node節(jié)點加入集群命令
kubeadm token create $token --print-join-command --ttl=0
k8s通過kubeadm搭建集群成功
