目錄
前言
節(jié)點(diǎn)親和 Node affinity
示例1： nodeAffinity 硬親和
示例2: nodeAffinity 硬親和
示例3: nodeAffinity 軟親和
前言
在k8s集群建設(shè)過(guò)程中另假，一般情況下我們部署的 Pod 是通過(guò)集群的自動(dòng)調(diào)度策略來(lái)選擇節(jié)點(diǎn)的壁顶，默認(rèn)情況下調(diào)度器考慮的是資源足夠匾南，并且負(fù)載盡量平均。但是有的時(shí)候我們需要能夠更加細(xì)粒度的去控制 Pod 的調(diào)度；有時(shí)我們希望對(duì)內(nèi)和對(duì)外的兩類業(yè)務(wù)分別跑在不同的節(jié)點(diǎn)上是掰，相互有依賴的兩個(gè)pod跑在同一節(jié)點(diǎn)上复凳，等情況；這就需要我們更好的控制pod的部署谋右；k8s給我們提供了親和性和反親和性硬猫，污點(diǎn)(taint)和Toleration（容忍）等概念。
- 節(jié)點(diǎn)node親和性
- 其中又分為硬親和與軟親和
- 硬親和表示條件必須滿足
- 軟親和表示盡量滿足
節(jié)點(diǎn)親和 Node affinity:
硬親和表示條件必須滿足條件
requiredDuringSchedulingIgnoredDuringExecution
表示pod必須部署到滿足條件的節(jié)點(diǎn)上改执，如果沒(méi)有滿足條件的節(jié)點(diǎn)啸蜜，就不停重試。其中IgnoreDuringExecution表示pod部署之后運(yùn)行的時(shí)候辈挂，如果節(jié)點(diǎn)標(biāo)簽發(fā)生了變化盔性，不再滿足pod指定的條件，pod也會(huì)繼續(xù)運(yùn)行呢岗。軟親和表示盡量滿足條件
preferredDuringSchedulingIgnoredDuringExecution
表示優(yōu)先部署到滿足條件的節(jié)點(diǎn)上冕香，如果沒(méi)有滿足條件的節(jié)點(diǎn)，就忽略這些條件后豫，按照正常邏輯部署悉尾。
查看nodeAffinity的詳細(xì)說(shuō)明
[root@k8s-master scheduler]# kubectl explain pods.spec.affinity.nodeAffinity
KIND: Pod
...
FIELDS:
preferredDuringSchedulingIgnoredDuringExecution <[]Object> #軟親和
The scheduler will prefer to schedule pods to nodes that satisfy the
affinity expressions specified by this field, but it may choose a node that
violates one or more of the expressions. The node that is most preferred is
the one with the greatest sum of weights, i.e. for each node that meets all
of the scheduling requirements (resource request, requiredDuringScheduling
affinity expressions, etc.), compute a sum by iterating through the
elements of this field and adding "weight" to the sum if the node matches
the corresponding matchExpressions; the node(s) with the highest sum are
the most preferred.
requiredDuringSchedulingIgnoredDuringExecution <Object> #硬親和
If the affinity requirements specified by this field are not met at
scheduling time, the pod will not be scheduled onto the node. If the
affinity requirements specified by this field cease to be met at some point
during pod execution (e.g. due to an update), the system may or may not try
to eventually evict the pod from its node.
[root@k8s-master ~]# kubectl explain pods.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution #硬親和詳情說(shuō)明
...
FIELDS:
nodeSelectorTerms <[]Object> -required- #節(jié)點(diǎn)選擇條件
Required. A list of node selector terms. The terms are ORed.
[root@k8s-master scheduler]# kubectl explain pods.spec.affinity.nodeAffinity.preferredDuringSchedulingIgnoredDuringExecution #軟親和詳情說(shuō)明
...
FIELDS:
preference <Object> -required- #親和偏向與權(quán)重一起使用
A node selector term, associated with the corresponding weight.
weight <integer> -required- #權(quán)重
Weight associated with matching the corresponding nodeSelectorTerm, in the
range 1-100.
[root@k8s-master scheduler]# kubectl explain pods.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution
KIND: Pod
VERSION: v1
RESOURCE: requiredDuringSchedulingIgnoredDuringExecution <Object>
DESCRIPTION:
If the affinity requirements specified by this field are not met at
scheduling time, the pod will not be scheduled onto the node. If the
affinity requirements specified by this field cease to be met at some point
during pod execution (e.g. due to an update), the system may or may not try
to eventually evict the pod from its node.
A node selector represents the union of the results of one or more label
queries over a set of nodes; that is, it represents the OR of the selectors
represented by the node selector terms.
FIELDS:
nodeSelectorTerms <[]Object> -required-
Required. A list of node selector terms. The terms are ORed.
示例1： 節(jié)點(diǎn)硬親和
[root@k8s-master Scheduler]# cat pod-with-nodeselector.yaml
apiVersion: v1
kind: Pod
metadata:
name: pod-with-nodeselector
spec:
containers:
- name: demoapp
image: ikubernetes/demoapp:v1.0
nodeSelector: #硬親和選項(xiàng)
gpu: '' #為空
[root@k8s-master Scheduler]# kubectl get pod
NAME READY STATUS RESTARTS AGE
deployment-demo-5fddfb8ffc-lssq8 1/1 Running 0 23m
deployment-demo-5fddfb8ffc-r277n 1/1 Running 0 23m
deployment-demo-5fddfb8ffc-wrpjx 1/1 Running 0 23m
deployment-demo-5fddfb8ffc-zzwck 1/1 Running 0 23m
pod-with-nodeselector 0/1 Pending 0 3m8s #掛起狀態(tài)
[root@k8s-master Scheduler]# kubectl describe pod pod-with-nodeselector
......
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 15s default-scheduler 0/4 nodes are available: 4 node(s) didn't match node selector. #提示所有節(jié)點(diǎn)沒(méi)有匹配的標(biāo)簽
Warning FailedScheduling 15s default-scheduler 0/4 nodes are available: 4 node(s) didn't match node selector.
[root@k8s-master Scheduler]# kubectl label node k8s-node3 gpu='' #給node3 打標(biāo)簽gpu為空
node/k8s-node3 labeled
[root@k8s-master Scheduler]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
deployment-demo-5fddfb8ffc-lssq8 1/1 Running 0 24m 192.168.113.14 k8s-node1 <none> <none>
deployment-demo-5fddfb8ffc-r277n 1/1 Running 0 24m 192.168.12.12 k8s-node2 <none> <none>
deployment-demo-5fddfb8ffc-wrpjx 1/1 Running 0 24m 192.168.12.11 k8s-node2 <none> <none>
deployment-demo-5fddfb8ffc-zzwck 1/1 Running 0 24m 192.168.51.19 k8s-node3 <none> <none>
pod-with-nodeselector 1/1 Running 0 3m59s 192.168.51.20 k8s-node3 <none> <none> #運(yùn)行在node3
[root@k8s-master Scheduler]# kubectl label node k8s-node3 gpu- #刪除標(biāo)簽不會(huì)對(duì)已創(chuàng)建的Pod產(chǎn)生影響，因?yàn)檎{(diào)度只發(fā)生在創(chuàng)建之前
node/k8s-node3 labeled
[root@k8s-master Scheduler]# kubectl get pod
NAME READY STATUS RESTARTS AGE
deployment-demo-5fddfb8ffc-lssq8 1/1 Running 0 28m
deployment-demo-5fddfb8ffc-r277n 1/1 Running 0 28m
deployment-demo-5fddfb8ffc-wrpjx 1/1 Running 0 28m
deployment-demo-5fddfb8ffc-zzwck 1/1 Running 0 28m
pod-with-nodeselector 1/1 Running 0 8m9s
[root@k8s-master Scheduler]# cat node-affinity-required-demo.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: node-affinity-required
namespace: default
spec:
replicas: 5
selector:
matchLabels :
app: demoapp
ctlr: node-affinity-required
template:
metadata:
labels :
app: demoapp
ctlr: node-affinity-required
spec:
containers :
- name: demoapp
image: ikubernetes/demoapp:v1.0
livenessProbe :
httpGet:
path: '/livez'
port: 80
initialDelaySeconds: 5
readinessProbe:
httpGet :
path: '/readyz'
port: 80
initialDelaySeconds: 15
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions: #匹配條件
- key: gpu #擁有這個(gè)標(biāo)簽
operator: Exists
- key: node-role.kubernetes.io/master #不能為主節(jié)點(diǎn) 兩個(gè)條件同時(shí)滿足
operator: DoesNotExist
[root@k8s-master Scheduler]# kubectl apply -f node-affinity-required-demo.yaml
[root@k8s-master Scheduler]# kubectl get pod
NAME READY STATUS RESTARTS AGE
node-affinity-required-5cb67df4b-d5nk6 0/1 Pending 0 3m51s
node-affinity-required-5cb67df4b-m6zxf 0/1 Pending 0 3m52s
node-affinity-required-5cb67df4b-sq5k9 0/1 Pending 0 3m51s
node-affinity-required-5cb67df4b-tvpwf 0/1 Pending 0 3m51s
node-affinity-required-5cb67df4b-vkx7j 0/1 Pending 0 3m52s
pod-with-nodeselector 0/1 Pending 0 31m #Pod掛起
[root@k8s-master Scheduler]# kubectl label node k8s-node2 gpu='true'
node/k8s-node2 labeled #為節(jié)點(diǎn)添加標(biāo)簽以符合條件
[root@k8s-master Scheduler]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
node-affinity-required-5cb67df4b-d5nk6 0/1 ContainerCreating 0 5m14s <none> k8s-node2 <none> <none>
node-affinity-required-5cb67df4b-m6zxf 0/1 ContainerCreating 0 5m15s <none> k8s-node2 <none> <none>
node-affinity-required-5cb67df4b-sq5k9 0/1 ContainerCreating 0 5m14s <none> k8s-node2 <none> <none>
node-affinity-required-5cb67df4b-tvpwf 0/1 ContainerCreating 0 5m14s <none> k8s-node2 <none> <none>
node-affinity-required-5cb67df4b-vkx7j 0/1 ContainerCreating 0 5m15s <none> k8s-node2 <none> <none>
示例2: nodeAffinity 硬親和
- requiredDuringSchedulingIgnoredDuringExecution
[root@k8s-master Scheduler]# cat node-affinity-and-resourcefits.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: node-affinity-and-resourcefits
namespace: default
spec:
replicas: 5
selector:
matchLabels:
app: demoapp
ctlr: node-affinity-and-resourcefits
template:
metadata:
labels:
app: demoapp
ctlr: node-affinity-and-resourcefits
spec:
containers:
- name: demoapp
image: ikubernetes/demoapp:v1.0
resources: #預(yù)選函數(shù) 只有滿足資源需求的node 才會(huì)進(jìn)行下面的權(quán)重打分 進(jìn)行優(yōu)選
requests:
cpu: 1000m
memory: 200Mi
livenessProbe:
httpGet:
path: '/livez'
port: 80
initialDelaySeconds: 5
readinessProbe:
httpGet:
path: '/readyz'
port: 80
initialDelaySeconds: 15
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution: #硬親和
nodeSelectorTerms:
- matchExpressions:
- key: gpu
operator: Exists
示例3: nodeAffinity 軟親和
- preferredDuringSchedulingIgnoredDuringExecution
[root@k8s-master Scheduler]# cat node-affinity-preferred-demo.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: node-affinity-preferred
spec:
replicas: 5
selector:
matchLabels:
app: demoapp
ctir: node-affinity-preferred
template:
metadata:
name: demoapp
labels:
app: demoapp
ctir: node-affinity-preferred
spec:
containers :
- name: demoapp
image: ikubernetes/demoapp:v1.0
resources: #預(yù)選函數(shù) 只有滿足資源需求的node 才會(huì)進(jìn)行下面的權(quán)重打分 進(jìn)行優(yōu)選
requests:
cpu: 100m
memory: 100Mi
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution: #軟親和
- weight: 60
preference:
matchExpressions: #帶gpu標(biāo)簽的加60權(quán)重
- key: gpu
operator: Exists
- weight: 30
preference:
matchExpressions: #包含foo挫酿、bar標(biāo)簽的加30權(quán)重
- key: region
operator: In
values: ["foo","bar"]
[root@k8s-master Scheduler]# kubectl apply -f node-affinity-preferred-demo.yaml
deployment.apps/node-affinity-preferred created
#為節(jié)點(diǎn)添加標(biāo)簽
[root@k8s-master ~]# kubectl label node k8s-node1.org gpu=2
node/k8s-node1.org labeled
[root@k8s-master ~]# kubectl label node k8s-node3.org region=foo
node/k8s-node3.org labeled
[root@k8s-master ~]# kubectl label node k8s-node2.org region=bar
node/k8s-node2.org labeled
[root@k8s-master ~]# kubectl get node -l gpu # node1為gpu
NAME STATUS ROLES AGE VERSION
k8s-node1.org Ready <none> 47d v1.22.2
[root@k8s-master ~]# kubectl get node -l region #node2构眯、node3為
NAME STATUS ROLES AGE VERSION
k8s-node2.org Ready <none> 47d v1.22.2
k8s-node3.org Ready <none> 47d v1.22.2
[root@k8s-master Scheduler]# kubectl apply -f node-affinity-preferred-demo.yaml
deployment.apps/node-affinity-preferred created
- 理論上 node1權(quán)重更高 pod會(huì)都運(yùn)行在node1上
[root@k8s-master ~]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
details-v1-79f774bdb9-vjmll 2/2 Running 0 27d 10.244.42.13 k8s-node3.org <none> <none>
node-affinity-preferred-5579fd76bc-5hfd2 0/2 Init:0/1 0 15s <none> k8s-node1.org <none> <none>
node-affinity-preferred-5579fd76bc-gzhd6 0/2 Init:0/1 0 15s <none> k8s-node1.org <none> <none>
node-affinity-preferred-5579fd76bc-q8wrc 0/2 Init:0/1 0 15s <none> k8s-node1.org <none> <none>
node-affinity-preferred-5579fd76bc-v42sn 0/2 Init:0/1 0 15s <none> k8s-node1.org <none> <none>
node-affinity-preferred-5579fd76bc-vvc42 0/2 Init:0/1 0 15s <none> k8s-node1.org <none> <none>
productpage-v1-6b746f74dc-q564k 2/2 Running 0 27d 10.244.42.21 k8s-node3.org <none> <none>
ratings-v1-b6994bb9-vh57t 2/2 Running 0 27d 10.244.42.19 k8s-node3.org <none> <none>
reviews-v1-545db77b95-clh87 2/2 Running 0 27d 10.244.42.12 k8s-node3.org <none> <none>
reviews-v2-7bf8c9648f-hdbdl 2/2 Running 0 27d 10.244.42.9 k8s-node3.org <none> <none>
reviews-v3-84779c7bbc-4vzcz 2/2 Running 0 27d 10.244.42.17 k8s-node3.org <none> <none>
