詳解 Kubernetes Pod imagePullPolicy
問題背景
Kubernetes 管理下的容器會(huì)在什么情況下對(duì)容器鏡像重新拉取？
概念理解
官方文檔：https://v1-12.docs.kubernetes.io/docs/concepts/containers/images/#updating-images
對(duì)一個(gè) Pod 來說振惰，spec.containers.imagePullPolicy字段用于管理容器鏡像的拉取策略把将，可選項(xiàng)為IfNotPresent和Always拱雏。
默認(rèn)的 imagePullPolicy 為IfNotPresent贪惹，image配置如image: nginx:1.12.5滴劲，當(dāng)宿主存在該鏡像時(shí)攻晒，kubelet 會(huì)自動(dòng)跳過鏡像拉取的步驟；
如果希望每次容器啟動(dòng)時(shí)都從鏡像庫拉取鏡像班挖，可以通過以下方式中的任一一種來配置:
- 將 imagePullPolicy 設(shè)為
Always; - 不配置 imagePullPolicy, 將容器鏡像的版本號(hào)設(shè)置為
:latest鲁捏，即images: nginx:latest； - 不配置 imagePullPolicy萧芙，也不配置容器版本號(hào)给梅，如
images: nginx
實(shí)驗(yàn)測試
測試準(zhǔn)備
nginx-deployment.yaml 配置 container 對(duì)應(yīng)的 imagePullPolicy: Always
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
minReadySeconds: 30
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.15.4
imagePullPolicy: Always
ports:
- containerPort: 80
測試方法
- 創(chuàng)建好對(duì)應(yīng)的 Deployment, 觀察某個(gè) Pod 的事件
- 通過某一種手段，不對(duì) Pod 進(jìn)行操作双揪，只是使得 Pod 里的 Container 退出（模擬 OOM場景）动羽；
- 觀測 Pod 元數(shù)據(jù)改變情況，觀測 docker image 拉取情況渔期。
測試過程
root@kmaster135:/home/chenjiaxi01/yaml/controllers# kubectl get pods -o wide| grep nginx
nginx-deployment-57f495d87b-k6g48 1/1 Running 2 173m 10.244.2.158 dnode137 <none>
nginx-deployment-57f495d87b-rcvlg 1/1 Running 0 173m 10.244.2.156 dnode137 <none>
nginx-deployment-57f495d87b-tnmrq 1/1 Running 0 173m 10.244.1.141 dnode136 <none>
定位到宿主上:
root@dnode137:~# docker ps | grep nginx-deployment-57f495d87b-rcvlg
9947530bc668 nginx@sha256:e8ab8d42e0c34c104ac60b43ba60b19af08e19a0e6d50396bdfd4cef0347ba83 "nginx -g 'daemon ..." 2 hours ago Up 2 hours k8s_nginx_nginx-deployment-57f495d87b-rcvlg_default_40cc107c-ee33-11e9-8ba3-000c290b4cc5_0
6436b335a958 k8s.gcr.io/pause:3.1 "/pause" 2 hours ago Up 2 hours k8s_POD_nginx-deployment-57f495d87b-rcvlg_default_40cc107c-ee33-11e9-8ba3-000c290b4cc5_0
停掉 Nginx 容器 9947530bc668:
root@dnode137:~# docker stop 9947530bc668
9947530bc668
觀察 Pod 情況:
root@kmaster135:/home/chenjiaxi01/yaml/controllers# kubectl describe pods nginx-deployment-57f495d87b-rcvlg
Name: nginx-deployment-57f495d87b-rcvlg
Namespace: default
Priority: 0
PriorityClassName: <none>
Node: dnode137/192.168.77.137
Start Time: Sun, 13 Oct 2019 20:32:30 -0700
Labels: app=nginx
pod-template-hash=57f495d87b
Annotations: <none>
Status: Running
IP: 10.244.2.156
Controlled By: ReplicaSet/nginx-deployment-57f495d87b
Containers:
nginx:
Container ID: docker://29a227a4831cce1f02bd83512dfcc377f703f5663ed5e6409a3db2f0884ba374
Image: nginx:1.15.4
Image ID: docker-pullable://nginx@sha256:e8ab8d42e0c34c104ac60b43ba60b19af08e19a0e6d50396bdfd4cef0347ba83
Port: 80/TCP
Host Port: 0/TCP
State: Running
Started: Sun, 13 Oct 2019 23:27:29 -0700
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Sun, 13 Oct 2019 20:32:46 -0700
Finished: Sun, 13 Oct 2019 23:27:23 -0700
Ready: True
Restart Count: 1
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-zh48z (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Pulling 32s (x2 over 175m) kubelet, dnode137 pulling image "nginx:1.15.4"
Normal Pulled 27s (x2 over 175m) kubelet, dnode137 Successfully pulled image "nginx:1.15.4"
Normal Created 27s (x2 over 175m) kubelet, dnode137 Created container
Normal Started 27s (x2 over 175m) kubelet, dnode137 Started container
測試結(jié)論
- Pod
Start Time保持不變运吓，所以可以認(rèn)為是 Pod 重啟，而不是刪掉 Pod 重建擎场； - Node 不會(huì)發(fā)生變化羽德，也就是不會(huì)重新調(diào)度；
- IP 是否會(huì)發(fā)生變化迅办？停掉 App Container 的時(shí)候沒有發(fā)生變化宅静，如果停掉的是 Infra Container 呢？猜想應(yīng)該是有可能變化的站欺，因?yàn)槭侵亟?Sandbox 時(shí)由 CNI 負(fù)責(zé)的姨夹；
-
Containers字段里由于容器發(fā)生了重建纤垂，所以都會(huì)有變化；可以看到State里有本次容器啟動(dòng)的情況磷账，Last State保留了上一次容器運(yùn)行的基本情況峭沦； -
ContainersRestart Count從 0 變?yōu)?1，表示容器的重啟次數(shù)逃糟，這個(gè)數(shù)值也會(huì)體現(xiàn)在kubectl get pod里吼鱼； - 從 Events 中可以看到，
kubelet, dnode137 pulling image "nginx:1.15.4"重新拉取了容器鏡像用于拉起新容器绰咽；如果是默認(rèn)的配置imagePullPolicy: IfNotPresent則該信息為Container image "nginx:1.15.4" already present on machine菇肃，如果 image 已經(jīng)存在的話。
源碼分析
在 kublet 的代碼里取募，在主循環(huán)SyncPod有相應(yīng)處理邏輯負(fù)責(zé) image 進(jìn)行拉取:
-
func SyncPod():pkg/kubelet/kuberuntime/kuberuntime_manager.go:578
// SyncPod syncs the running pod into the desired pod by executing following steps:
//
// 1. Compute sandbox and container changes.
// 2. Kill pod sandbox if necessary.
// 3. Kill any containers that should not be running.
// 4. Create sandbox if necessary.
// 5. Create init containers.
// 6. Create normal containers.
func (m *kubeGenericRuntimeManager) SyncPod(pod *v1.Pod, _ v1.PodStatus, podStatus *kubecontainer.PodStatus, pullSecrets []v1.Secret, backOff *flowcontrol.Backoff) (result kubecontainer.PodSyncResult) {
...
glog.V(4).Infof("Creating init container %+v in pod %v", container, format.Pod(pod))
if msg, err := m.startContainer(podSandboxID, podSandboxConfig, container, pod, podStatus, pullSecrets, podIP, kubecontainer.ContainerTypeInit); err != nil {
startContainerResult.Fail(err, msg)
utilruntime.HandleError(fmt.Errorf("init container start failed: %v: %s", err, msg))
return
}
...
}
-
func StartContainer:pkg/kubelet/kuberuntime/kuberuntime_container.go:89
// startContainer starts a container and returns a message indicates why it is failed on error.
// It starts the container through the following steps:
// * pull the image
// * create the container
// * start the container
// * run the post start lifecycle hooks (if applicable)
func (m *kubeGenericRuntimeManager) startContainer(podSandboxID string, podSandboxConfig *runtimeapi.PodSandboxConfig, container *v1.Container, pod *v1.Pod, podStatus *kubecontainer.PodStatus, pullSecrets []v1.Secret, podIP string, containerType kubecontainer.ContainerType) (string, error) {
// Step 1: pull the image.
imageRef, msg, err := m.imagePuller.EnsureImageExists(pod, container, pullSecrets)
if err != nil {
m.recordContainerEvent(pod, container, "", v1.EventTypeWarning, events.FailedToCreateContainer, "Error: %v", grpc.ErrorDesc(err))
return msg, err
}
...
}
-
type ImageManager interface:pkg/kubelet/images/types.go:50
// ImageManager provides an interface to manage the lifecycle of images.
// Implementations of this interface are expected to deal with pulling (downloading),
// managing, and deleting container images.
// Implementations are expected to abstract the underlying runtimes.
// Implementations are expected to be thread safe.
type ImageManager interface {
// EnsureImageExists ensures that image specified in `container` exists.
EnsureImageExists(pod *v1.Pod, container *v1.Container, pullSecrets []v1.Secret) (string, string, error)
// TODO(ronl): consolidating image managing and deleting operation in this interface
}
-
func EnsureImageExist:pkg/kubelet/images/image_manager.go:86
// EnsureImageExists pulls the image for the specified pod and container, and returns
// (imageRef, error message, error).
func (m *imageManager) EnsureImageExists(pod *v1.Pod, container *v1.Container, pullSecrets []v1.Secret) (string, string, error) {
...
present := imageRef != ""
if !shouldPullImage(container, present) {
if present {
msg := fmt.Sprintf("Container image %q already present on machine", container.Image)
m.logIt(ref, v1.EventTypeNormal, events.PulledImage, logPrefix, msg, glog.Info)
return imageRef, "", nil
} else {
msg := fmt.Sprintf("Container image %q is not present with pull policy of Never", container.Image)
m.logIt(ref, v1.EventTypeWarning, events.ErrImageNeverPullPolicy, logPrefix, msg, glog.Warning)
return "", msg, ErrImageNeverPull
}
}
...
}
-
func shouldPullImage:pkg/kubelet/images/image_manager.go:62
// shouldPullImage returns whether we should pull an image according to
// the presence and pull policy of the image.
func shouldPullImage(container *v1.Container, imagePresent bool) bool {
if container.ImagePullPolicy == v1.PullNever {
return false
}
if container.ImagePullPolicy == v1.PullAlways ||
(container.ImagePullPolicy == v1.PullIfNotPresent && (!imagePresent)) {
return true
}
return false
}
新的問題: 當(dāng)容器配置形如image: nginx:latest時(shí)琐谤，imagePullPolicy默認(rèn)配置為Always，管理默認(rèn)配置的代碼在哪里玩敏？
pkg/apis/core/v1/defaults.go:77:
func SetDefaults_Container(obj *v1.Container) {
if obj.ImagePullPolicy == "" {
// Ignore error and assume it has been validated elsewhere
_, tag, _, _ := parsers.ParseImageName(obj.Image)
// Check image tag
if tag == "latest" {
obj.ImagePullPolicy = v1.PullAlways
} else {
obj.ImagePullPolicy = v1.PullIfNotPresent
}
}
if obj.TerminationMessagePath == "" {
obj.TerminationMessagePath = v1.TerminationMessagePathDefault
}
if obj.TerminationMessagePolicy == "" {
obj.TerminationMessagePolicy = v1.TerminationMessageReadFile
}
}
