Table of Contents
1. 背景
client-go實際只是一個客戶端，list-watch我們經常聽到张肾。但實際上是apisever的實現(xiàn)坦喘。在apisever注冊資源對象的create, update, delete等等hanlder時贷屎，也注冊了List-watch的實現(xiàn)。
所以在研究client-go是如果處理list watch之前轧拄，先了解一個apiserver的list watch機制
2. list watch機制
List-watch是K8S統(tǒng)一的異步消息處理機制诫肠，保證了消息的實時性泄私，可靠性，順序性袜硫，性能等等氯葬，為聲明式風格的API奠定了良好的基礎，它是優(yōu)雅的通信方式婉陷，是K8S 架構的精髓帚称。
2.1 如何實現(xiàn)實時性
一般客戶端和服務器端的同步官研，無非就是兩種大類：一種是客戶端輪訓服務器端。第二種就是服務器端主動發(fā)起通知闯睹。
k8s采用的是第二種戏羽，主動發(fā)起通知。這里具體就是使用了watch機制楼吃。
list, watch其實都是特殊的get接口始花。
get default命名空間所有的pod url如下： curl http://xxx:port/api/v1/namespaces/default/pods
watch default命名空間所有的pod url如下: curl http://XXX/api/v1/namespaces/default/pods?watch=true 就是多了一個watch=true的參數(shù)
root:/# curl http://XXX/api/v1/namespaces/default/pods?watch=true
{"type":"ADDED","object":{"kind":"Pod","apiVersion":"v1","metadata":{"name":"zx-vpa-786d4b8bb5-xv5zw","generateName":"zx-vpa-786d4b8bb5-","namespace":"default","selfLink":"/api/v1/namespaces/default/pods/zx-vpa-786d4b8bb5-xv5zw","uid":"639944b7-3495-4fbb-a21d-cbc7f4d6f7a5","resourceVersion":"157197390","creationTimestamp":"2021-11-12T10:59:39Z","labels":{"app":"zx-vpa-test","pod-template-hash":"786d4b8bb5"},"annotations":{"v2-fixed-ip":"","v2-subnet":"faf7c8b0-55c3-42c7-ba27-ad90290a9cd9","v2-tenant":"","v2-vpc":"6af350be-c456-44bc-909d-4b92c48b3b54","vpaObservedContainers":"zx-vpa, zx-vpa2","vpaUpdates":"Pod resources updated by hamster-vpa: container 0: memory request, cpu request, memory limit, cpu limit; container 1: cpu request, memory request, cpu limit, memory limit"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"zx-vpa-786d4b8bb5","uid":"8199639c-40fc-4dc5-81c3-d3faff7f6b4c","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"default-token-dbxf8","secret":{"secretName":"default-token-dbxf8","defaultMode":420}}],"containers":[{"name":"zx-vpa","image":"dockerhub.nie.netease.com/fanqihong/ubuntu:stress","command":["sleep","36000"],"resources":{"limits":{"cpu":"12m","memory":"131072k"},"requests":{"cpu":"12m","memory":"131072k"}},"volumeMounts":[{"name":"default-token-dbxf8","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"},{"name":"zx-vpa2","image":"ncr.nie.netease.com/zouxiang/testcpu:v1","command":["sleep","36000"],"resources":{"limits":{"cpu":"12m","memory":"131072k"},"requests":{"cpu":"12m","memory":"131072k"}},"volumeMounts":[{"name":"default-token-dbxf8","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Always","terminationGracePeriodSeconds":5,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"7.34.19.14","hostNetwork":true,"securityContext":{},"schedulerName":"default-scheduler","enableServiceLinks":true},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-11-12T10:59:39Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-11-14T02:59:47Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-11-14T02:59:47Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-11-12T10:59:39Z"}],"hostIP":"7.34.19.14","podIP":"7.34.19.14","podIPs":[{"ip":"7.34.19.14"}],"startTime":"2021-11-12T10:59:39Z","containerStatuses":[{"name":"zx-vpa","state":{"running":{"startedAt":"2021-11-14T02:59:46Z"}},"lastState":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2021-11-13T16:59:45Z","finishedAt":"2021-11-14T02:59:45Z","containerID":"docker://87a70d2061b7fb37b0f97be3a4f9d44b345fbd54be3dcc4d8a61879dd5c6a127"}},"ready":true,"restartCount":4,"image":"dockerhub.nie.netease.com/fanqihong/ubuntu:stress","imageID":"docker-pullable://dockerhub.nie.netease.com/fanqihong/ubuntu@sha256:ac49d16f9686c2acd351d436ed7154311e4dba50ed8c18b6abaa578dde696440","containerID":"docker://bc586f53f363e9afb08c7a214eef06c8c1202f72439fc972d4c7d6177cfb8e63","started":true},{"name":"zx-vpa2","state":{"running":{"startedAt":"2021-11-14T02:59:47Z"}},"lastState":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2021-11-13T16:59:46Z","finishedAt":"2021-11-14T02:59:46Z","containerID":"docker://37d8dd54be6d27ed9f055049e700f12fa4aa30ec29f2fd16fd5176218b2acce9"}},"ready":true,"restartCount":4,"image":"ncr.nie.netease.com/zouxiang/testcpu:v1","imageID":"docker-pullable://ncr.nie.netease.com/zouxiang/testcpu@sha256:4560824247d61f92c0d4b62224fdb3efc47560339ff05c92f73d6c731eba2717","containerID":"docker://9ccb7968bee2c155c472e03d56a5987c9cf7e6833a4cb125084ceb19158474ed","started":true}],"qosClass":"Guaranteed"}}}
{"type":"ADDED","object":{"kind":"Pod","apiVersion":"v1","metadata":{"name":"zx-vpa-786d4b8bb5-mw6mr","generateName":"zx-vpa-786d4b8bb5-","namespace":"default","selfLink":"/api/v1/namespaces/default/pods/zx-vpa-786d4b8bb5-mw6mr","uid":"4e7f3a44-7483-434d-a917-52b37c0eae33","resourceVersion":"157192079","creationTimestamp":"2021-11-12T10:52:37Z","labels":{"app":"zx-vpa-test","pod-template-hash":"786d4b8bb5"},"annotations":{"v2-fixed-ip":"","v2-subnet":"faf7c8b0-55c3-42c7-ba27-ad90290a9cd9","v2-tenant":"","v2-vpc":"6af350be-c456-44bc-909d-4b92c48b3b54","vpaObservedContainers":"zx-vpa, zx-vpa2","vpaUpdates":"Pod resources updated by hamster-vpa: container 0: cpu request, memory request, cpu limit, memory limit; container 1: memory request, cpu request, cpu limit, memory limit"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"zx-vpa-786d4b8bb5","uid":"8199639c-40fc-4dc5-81c3-d3faff7f6b4c","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"default-token-dbxf8","secret":{"secretName":"default-token-dbxf8","defaultMode":420}}],"containers":[{"name":"zx-vpa","image":"dockerhub.nie.netease.com/fanqihong/ubuntu:stress","command":["sleep","36000"],"resources":{"limits":{"cpu":"12m","memory":"131072k"},"requests":{"cpu":"12m","memory":"131072k"}},"volumeMounts":[{"name":"default-token-dbxf8","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"},{"name":"zx-vpa2","image":"ncr.nie.netease.com/zouxiang/testcpu:v1","command":["sleep","36000"],"resources":{"limits":{"cpu":"12m","memory":"131072k"},"requests":{"cpu":"12m","memory":"131072k"}},"volumeMounts":[{"name":"default-token-dbxf8","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Always","terminationGracePeriodSeconds":5,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"10.90.67.175","hostNetwork":true,"securityContext":{},"schedulerName":"default-scheduler","enableServiceLinks":true},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-11-12T10:52:37Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-11-14T02:52:54Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-11-14T02:52:54Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-11-12T10:52:37Z"}],"hostIP":"10.90.67.175","podIP":"10.90.67.175","podIPs":[{"ip":"10.90.67.175"}],"startTime":"2021-11-12T10:52:37Z","containerStatuses":[{"name":"zx-vpa","state":{"running":{"startedAt":"2021-11-14T02:52:50Z"}},"lastState":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2021-11-13T16:52:47Z","finishedAt":"2021-11-14T02:52:47Z","containerID":"docker://ddf625ee9c90ba70ba5f1d27caa4d61ded938143a724dccbfada898271ac7fd0"}},"ready":true,"restartCount":4,"image":"dockerhub.nie.netease.com/fanqihong/ubuntu:stress","imageID":"docker-pullable://dockerhub.nie.netease.com/fanqihong/ubuntu@sha256:ac49d16f9686c2acd351d436ed7154311e4dba50ed8c18b6abaa578dde696440","containerID":"docker://854091543fc0ba88d3dc4a839f8014d21ecbaecf4e40221d2ce9d6a343ddbe29","started":true},{"name":"zx-vpa2","state":{"running":{"startedAt":"2021-11-14T02:52:53Z"}},"lastState":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2021-11-13T16:52:50Z","finishedAt":"2021-11-14T02:52:50Z","containerID":"docker://0241d05357e1f6b8eec73810341bddf14479a168aaa7958ee580855eb2f4300f"}},"ready":true,"restartCount":4,"image":"ncr.nie.netease.com/zouxiang/testcpu:v1","imageID":"docker-pullable://ncr.nie.netease.com/zouxiang/testcpu@sha256:4560824247d61f92c0d4b62224fdb3efc47560339ff05c92f73d6c731eba2717","containerID":"docker://ad0d09158c0db8b10d2c282f2749c1c1e97fdb707e10392b9033aa619b162450","started":true}],"qosClass":"Guaranteed"}}}
// 一旦有對象改變就會收到事件。type有三種孩锡，modifyed, added ,deleted
{"type":"MODIFIED","object":{"kind":"Pod","apiVersion":"v1","metadata":{"name":"zx-vpa-786d4b8bb5-xv5zw","generateName":"zx-vpa-786d4b8bb5-","namespace":"default","selfLink":"/api/v1/namespaces/default/pods/zx-vpa-786d4b8bb5-xv5zw","uid":"639944b7-3495-4fbb-a21d-cbc7f4d6f7a5","resourceVersion":"157401529","creationTimestamp":"2021-11-12T10:59:39Z","labels":{"app":"zx-vpa-test","pod-template-hash":"786d4b8bb5"},"annotations":{"v2-fixed-ip":"","v2-subnet":"faf7c8b0-55c3-42c7-ba27-ad90290a9cd9","v2-tenant":"","v2-vpc":"6af350be-c456-44bc-909d-4b92c48b3b54","vpaObservedContainers":"zx-vpa, zx-vpa2","vpaUpdates":"11111111111Pod resources updated by hamster-vpa: container 0: memory request, cpu request, memory limit, cpu limit; container 1: cpu request, memory request, cpu limit, memory limit"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"zx-vpa-786d4b8bb5","uid":"8199639c-40fc-4dc5-81c3-d3faff7f6b4c","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"default-token-dbxf8","secret":{"secretName":"default-token-dbxf8","defaultMode":420}}],"containers":[{"name":"zx-vpa","image":"dockerhub.nie.netease.com/fanqihong/ubuntu:stress","command":["sleep","36000"],"resources":{"limits":{"cpu":"12m","memory":"131072k"},"requests":{"cpu":"12m","memory":"131072k"}},"volumeMounts":[{"name":"default-token-dbxf8","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"},{"name":"zx-vpa2","image":"ncr.nie.netease.com/zouxiang/testcpu:v1","command":["sleep","36000"],"resources":{"limits":{"cpu":"12m","memory":"131072k"},"requests":{"cpu":"12m","memory":"131072k"}},"volumeMounts":[{"name":"default-token-dbxf8","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Always","terminationGracePeriodSeconds":5,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"7.34.19.14","hostNetwork":true,"securityContext":{},"schedulerName":"default-scheduler","enableServiceLinks":true},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-11-12T10:59:39Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-11-14T02:59:47Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-11-14T02:59:47Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-11-12T10:59:39Z"}],"hostIP":"7.34.19.14","podIP":"7.34.19.14","podIPs":[{"ip":"7.34.19.14"}],"startTime":"2021-11-12T10:59:39Z","containerStatuses":[{"name":"zx-vpa","state":{"running":{"startedAt":"2021-11-14T02:59:46Z"}},"lastState":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2021-11-13T16:59:45Z","finishedAt":"2021-11-14T02:59:45Z","containerID":"docker://87a70d2061b7fb37b0f97be3a4f9d44b345fbd54be3dcc4d8a61879dd5c6a127"}},"ready":true,"restartCount":4,"image":"dockerhub.nie.netease.com/fanqihong/ubuntu:stress","imageID":"docker-pullable://dockerhub.nie.netease.com/fanqihong/ubuntu@sha256:ac49d16f9686c2acd351d436ed7154311e4dba50ed8c18b6abaa578dde696440","containerID":"docker://bc586f53f363e9afb08c7a214eef06c8c1202f72439fc972d4c7d6177cfb8e63","started":true},{"name":"zx-vpa2","state":{"running":{"startedAt":"2021-11-14T02:59:47Z"}},"lastState":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2021-11-13T16:59:46Z","finishedAt":"2021-11-14T02:59:46Z","containerID":"docker://37d8dd54be6d27ed9f055049e700f12fa4aa30ec29f2fd16fd5176218b2acce9"}},"ready":true,"restartCount":4,"image":"ncr.nie.netease.com/zouxiang/testcpu:v1","imageID":"docker-pullable://ncr.nie.netease.com/zouxiang/testcpu@sha256:4560824247d61f92c0d4b62224fdb3efc47560339ff05c92f73d6c731eba2717","containerID":"docker://9ccb7968bee2c155c472e03d56a5987c9cf7e6833a4cb125084ceb19158474ed","started":true}],"qosClass":"Guaranteed"}}}
//刪除一個pod衙荐，發(fā)現(xiàn)會進入
MODIFIED （設置deletiontimestamp）->
ADDED "status":{"phase":"Pending","qosClass":"Guaranteed"}}} 新pod pending
MODIFIED podScheduled
MODIFIED ContainerCreating
MODIFIED.. 到pod running
DELETED 刪除舊Pod
curl http://7.34.19.44:58201/api/v1/watch/namespaces/default/pods
看起來也是一樣的效果
通過上面的實踐可以發(fā)現(xiàn)：
（1）watch其實就是一種特殊的get
（2）可以看到刪除操作后，對象的整個變化過程
（3）watch每次都會返回type浮创，和完整的對象信息
2.2 如何實現(xiàn)順序性
K8S在每個資源的事件中都帶一個resourceVersion的標簽忧吟，這個標簽是遞增的數(shù)字，所以當客戶端并發(fā)處理同一個資源的事件時斩披，它就可以對比resourceVersion來保證最終的狀態(tài)和最新的事件所期望的狀態(tài)保持一致溜族。
2.3 如何實現(xiàn)消息可靠性
list和watch一起保證了消息的可靠性，避免因消息丟失而造成狀態(tài)不一致場景垦沉。具體而言煌抒，list API可以查詢當前的資源及其對應的狀態(tài)(即期望的狀態(tài))，客戶端通過拿期望的狀態(tài)和實際的狀態(tài)進行對比厕倍，糾正狀態(tài)不一致的資源寡壮。Watch API和apiserver保持一個長鏈接，接收資源的狀態(tài)變更事件并做相應處理讹弯。如果僅調用watch API况既，若某個時間點連接中斷，就有可能導致消息丟失组民，所以需要通過list API解決消息丟失的問題棒仍。從另一個角度出發(fā)，我們可以認為list API獲取全量數(shù)據(jù)臭胜，watch API獲取增量數(shù)據(jù)莫其。雖然僅僅通過輪詢list API，也能達到同步資源狀態(tài)的效果耸三，但是存在開銷大乱陡，實時性不足的問題。
2.4 如何解決性能問題
（1）list-watch機制的結合就已經在apiserver做了性能優(yōu)化仪壮。（是不是可以watch的時候憨颠，只傳遞更新了的字段，而不是全量數(shù)據(jù)）
（2）client-go的 tool.cache做了客戶端的性能優(yōu)化問題
3.總結
本節(jié)主要從apiserver端探究了以下list-watch睛驳。接下來從client-go端源碼看看具體是如何實現(xiàn)的
