阿里云Cloud-Provider

系開源項(xiàng)目铐炫，見github

主要提供Service的LoadBalance功能

源碼解析

源碼相對比較簡單逛裤，就是一個(gè)簡單的自定義控制器笛求，默認(rèn)包括了這幾個(gè)控制器：

• 節(jié)點(diǎn)控制器：負(fù)責(zé)維護(hù)節(jié)點(diǎn)信息

• 路由控制器：負(fù)責(zé)添加POD網(wǎng)段到VPC路由表

• Service控制器：負(fù)責(zé)申請/釋放LoadBalance類型Service的負(fù)載均衡器

• NLB控制器：類似CLB碍庵，不過控制的是網(wǎng)絡(luò)負(fù)載均衡瞬沦，對應(yīng)阿里云文檔

節(jié)點(diǎn)控制器

當(dāng)使用云廠商的cloud-provider時(shí)醒颖，Kubelet 必須配置啟動參數(shù)--cloud-provider=external

設(shè)置后kubelet會給節(jié)點(diǎn)添加污點(diǎn)朱躺，參考kubelet代碼

if kl.externalCloudProvider {
        taint := v1.Taint{
            Key:    cloudproviderapi.TaintExternalCloudProvider,
            Value:  "true",
            Effect: v1.TaintEffectNoSchedule,
        }

        nodeTaints = append(nodeTaints, taint)
    }

而這里的節(jié)點(diǎn)控制器就是為了從云廠商獲取節(jié)點(diǎn)IP刁赖、狀態(tài)等信息，然后去除這個(gè)污點(diǎn)长搀，使節(jié)點(diǎn)可用

func (m *ReconcileNode) syncCloudNode(node *corev1.Node) error {
        var cloudTaint *v1.Taint
    for _, taint := range taints {
        if taint.Key == api.TaintExternalCloudProvider {
            cloudTaint = &taint
        }
    }
    if cloudTaint == nil {
        klog.V(5).Infof("node %s is registered without cloud taint. return ok", node.Name)
        return nil
    }


    err := m.doAddCloudNode(node)
    
    return nil
}

這里可用看到如果污點(diǎn)已經(jīng)沒了宇弛，則節(jié)點(diǎn)控制器就不再處理這個(gè)節(jié)點(diǎn)了；否則就需要進(jìn)行初始化了

• 首先查找這個(gè)節(jié)點(diǎn)對應(yīng)的ECS

func findCloudECS(ins prvd.IInstance, node *v1.Node) (*prvd.NodeAttribute, error) {
    if node.Spec.ProviderID != "" {
        return findCloudECSById(ins, node)
    } else {
        return findCloudECSByIp(ins, node)
    }
}

其中providerId的格式是cn-hangzhou.i-v98dklsmnxkkgiiil7：即REGION.NODEID

如果指定了providerId源请，則會查詢r(jià)egion下是否存在這個(gè)nodeid對應(yīng)的ECS

如果沒有指定枪芒，則會查詢VPC下這個(gè)節(jié)點(diǎn)的內(nèi)網(wǎng)IP對應(yīng)的ECS

var internalIP []string
    for _, addr := range node.Status.Addresses {
        if addr.Type == v1.NodeInternalIP {
            internalIP = append(internalIP, addr.Address)
        }
    }

• 找到ECS后，就能更新節(jié)點(diǎn)信息了谁尸，主要是更新節(jié)點(diǎn)的標(biāo)簽信息和污點(diǎn)信息

func setFields(node *v1.Node, ins *prvd.NodeAttribute, cfgRoute bool) {

    var modifiers []nodeModifier
    if ins.InstanceType != "" {
        modify := func(n *v1.Node) {
            n.Labels["beta.kubernetes.io/instance-type"] = ins.InstanceType
            n.Labels["node.kubernetes.io/instance-type"] = ins.InstanceType
        }
        modifiers = append(modifiers, modify)
    }

    if ins.Zone != "" {
        modify := func(n *v1.Node) {
            n.Labels["failure-domain.beta.kubernetes.io/zone"] = ins.Zone
            n.Labels["topology.kubernetes.io/zone"] = ins.Zone
        }
        modifiers = append(modifiers, modify)
    }

    if ins.Region != "" {
        modify := func(n *v1.Node) {
            n.Labels["failure-domain.beta.kubernetes.io/region"] = ins.Region
            n.Labels["topology.kubernetes.io/region"] = ins.Region
        }
        modifiers = append(modifiers, modify)
    }

    if node.Spec.ProviderID == "" && ins.InstanceID != "" {
        prvdId := fmt.Sprintf("%s.%s", ins.Region, ins.InstanceID)
        modify := func(n *v1.Node) {
            n.Spec.ProviderID = prvdId
        }
        modifiers = append(modifiers, modify)
    }

    modifiers = append(modifiers, removeCloudTaints)

    for _, modify := range modifiers {
        modify(node)
    }
}

主要包括這些標(biāo)簽舅踪，可以看到主要就是ECS對應(yīng)的信息

beta.kubernetes.io/instance-type
node.kubernetes.io/instance-type
failure-domain.beta.kubernetes.io/zone
topology.kubernetes.io/zone
failure-domain.beta.kubernetes.io/region
topology.kubernetes.io/region

然后移除污點(diǎn)、并且設(shè)置node.Spec.ProviderID為REGION.NODEID

• 最后再更新節(jié)點(diǎn)的status字段良蛮，主要是設(shè)置status.address字段

func findAddress(instance *ecs.Instance) []v1.NodeAddress {
    var addrs []v1.NodeAddress

    if len(instance.PublicIpAddress.IpAddress) > 0 {
        for _, ipaddr := range instance.PublicIpAddress.IpAddress {
            addrs = append(addrs, v1.NodeAddress{Type: v1.NodeExternalIP, Address: ipaddr})
        }
    }

    if instance.EipAddress.IpAddress != "" {
        addrs = append(addrs, v1.NodeAddress{Type: v1.NodeExternalIP, Address: instance.EipAddress.IpAddress})
    }

    if len(instance.InnerIpAddress.IpAddress) > 0 {
        for _, ipaddr := range instance.InnerIpAddress.IpAddress {
            addrs = append(addrs, v1.NodeAddress{Type: v1.NodeInternalIP, Address: ipaddr})
        }
    }

    if len(instance.VpcAttributes.PrivateIpAddress.IpAddress) > 0 {
        for _, ipaddr := range instance.VpcAttributes.PrivateIpAddress.IpAddress {
            addrs = append(addrs, v1.NodeAddress{Type: v1.NodeInternalIP, Address: ipaddr})
        }
    }

    return addrs
}

這里會統(tǒng)計(jì)ECS上的公網(wǎng)抽碌、內(nèi)網(wǎng)IP；其中公網(wǎng)IP作為status.address中的ExternalIP决瞳，內(nèi)網(wǎng)IP作為status.address中的InternalIP

然后再拼接kubelet默認(rèn)為節(jié)點(diǎn)設(shè)置的hostname货徙，作為status.address中的Hostname

func setHostnameAddress(node *v1.Node, addrs []v1.NodeAddress) []v1.NodeAddress {
    // Check if a hostname address exists in the cloud provided addresses
    hostnameExists := false
    for i := range addrs {
        if addrs[i].Type == v1.NodeHostName {
            hostnameExists = true
        }
    }
    // If hostname was not present in cloud provided addresses, use the hostname
    // from the existing node (populated by kubelet)
    if !hostnameExists {
        for _, addr := range node.Status.Addresses {
            if addr.Type == v1.NodeHostName {
                addrs = append(addrs, addr)
            }
        }
    }
    return addrs
}

拼接完成后左权，使用patch接口將信息更新到節(jié)點(diǎn)上

diff := func(copy runtime.Object) (client.Object, error) {
            nins := copy.(*corev1.Node)
            nins.Status.Addresses = cloudNode.Addresses
            return nins, nil
        }
        err := helper.PatchM(m.client, node, diff, helper.PatchStatus)

路由控制器

路由控制器主要是為了解決POD跨主訪問的問題

通過cloud-provider的配置參數(shù)啟動，默認(rèn)就是會配置路由信息

fs.BoolVar(&cfg.ConfigureCloudRoutes, flagConfigureCloudRoutes, true, "Should CIDRs allocated by allocate-node-cidrs be configured on the cloud provider.")

配置的是節(jié)點(diǎn)的POD CIDR

func getIPv4RouteForNode(node *v1.Node) (*net.IPNet, string, error) {
    var (
        ipv4CIDR    *net.IPNet
        ipv4CIDRStr string
        err         error
    )
    for _, podCidr := range append(node.Spec.PodCIDRs, node.Spec.PodCIDR) {
        if podCidr != "" {
            _, ipv4CIDR, err = net.ParseCIDR(podCidr)
            
            ipv4CIDRStr = ipv4CIDR.String()
            if len(ipv4CIDR.Mask) == net.IPv4len {
                ipv4CIDRStr = ipv4CIDR.String()
                break
            }
        }
    }
    return ipv4CIDR, ipv4CIDRStr, nil
}

然后獲取VPC路由表痴颊，vpc id可以直接通過阿里云的metadata服務(wù)器獲取赏迟，參考terway系列文章中的metadata獲取

func getRouteTables(ctx context.Context, providerIns prvd.Provider) ([]string, error) {
    vpcId, err := providerIns.VpcID()
    
    tables, err := providerIns.ListRouteTables(ctx, vpcId)
    
    if len(tables) > 1 {
        return nil, fmt.Errorf("multiple route tables found by vpc id[%s], length(tables)=%d", ctrlCfg.CloudCFG.Global.VpcID, len(tables))
    }
    if len(tables) == 0 {
        return nil, fmt.Errorf("no route tables found by vpc id[%s]", ctrlCfg.CloudCFG.Global.VpcID)
    }
    return tables, nil
}

最后就是添加POD CIDR到路由表上

func (r *VPCProvider) CreateRoute(
    ctx context.Context, table string, provideID string, destinationCIDR string,
) (*model.Route, error) {
    createRouteEntryRequest := vpc.CreateCreateRouteEntryRequest()
    createRouteEntryRequest.RouteTableId = table
    createRouteEntryRequest.DestinationCidrBlock = destinationCIDR
    createRouteEntryRequest.NextHopType = model.RouteNextHopTypeInstance
    _, instance, err := util.NodeFromProviderID(provideID)
    if err != nil {
        return nil, fmt.Errorf("invalid provide id: %v, err: %v", provideID, err)
    }
    createRouteEntryRequest.NextHopId = instance
    _, err = r.auth.VPC.CreateRouteEntry(createRouteEntryRequest)
    if err != nil {
        return nil, fmt.Errorf("error create route entry for %s, %s, error: %v", provideID, destinationCIDR, err)
    }
    return &model.Route{
        Name:            fmt.Sprintf("%s-%s", provideID, destinationCIDR),
        DestinationCIDR: destinationCIDR,
        ProviderId:      provideID,
    }, nil
}

可以看到這里的路由信息：目標(biāo)網(wǎng)段是POD CIDR的，設(shè)置下一跳為當(dāng)前ECS的nodeid蠢棱，即所有到POD CIDR的數(shù)據(jù)包下一跳都是當(dāng)前ECS锌杀，這樣就能打通POD跨主訪問的網(wǎng)絡(luò)了，都是通過VPC路由來實(shí)現(xiàn)的

Service控制器

Service是主要控制對象泻仙，而且支持的注解很多糕再，參考阿里云CLB文檔

下面主要分兩個(gè)主要流程分析下

創(chuàng)建CLB

這個(gè)場景對應(yīng)的是在創(chuàng)建Service時(shí)指定type: LoadBalancer

• 首先給Service添加一個(gè)特殊的Finalizer，方便刪除CLB的時(shí)候只處理有這個(gè)Finalizer的Service

if err := m.finalizerManager.AddFinalizers(req.Ctx, req.Service, helper.ServiceFinalizer); err != nil {
        
    }

• 然后解析Service的注解饰豺，獲取LoadBalance的基本信息亿鲜，如公網(wǎng)還是內(nèi)網(wǎng)、收費(fèi)方式冤吨、帶寬等信息

func (mgr *LoadBalancerManager) BuildLocalModel(reqCtx *svcCtx.RequestContext, mdl *model.LoadBalancer) error {
    mdl.LoadBalancerAttribute.AddressType = model.AddressType(reqCtx.Anno.Get(annotation.AddressType))
    mdl.LoadBalancerAttribute.InternetChargeType = model.InternetChargeType(reqCtx.Anno.Get(annotation.ChargeType))
    mdl.LoadBalancerAttribute.InstanceChargeType = model.InstanceChargeType(reqCtx.Anno.Get(annotation.InstanceChargeType))
    mdl.LoadBalancerAttribute.LoadBalancerSpec = model.LoadBalancerSpecType(reqCtx.Anno.Get(annotation.Spec))
    bandwidth := reqCtx.Anno.Get(annotation.Bandwidth)
    if bandwidth != "" {
        i, err := strconv.Atoi(bandwidth)
        mdl.LoadBalancerAttribute.Bandwidth = i
    }
    mdl.LoadBalancerAttribute.LoadBalancerSpec = model.LoadBalancerSpecType(reqCtx.Anno.Get(annotation.Spec))

    if reqCtx.Anno.Get(annotation.LoadBalancerId) != "" {
        mdl.LoadBalancerAttribute.LoadBalancerId = reqCtx.Anno.Get(annotation.LoadBalancerId)
        mdl.LoadBalancerAttribute.IsUserManaged = true
    }
    mdl.LoadBalancerAttribute.LoadBalancerName = reqCtx.Anno.Get(annotation.LoadBalancerName)
    mdl.LoadBalancerAttribute.VSwitchId = reqCtx.Anno.Get(annotation.VswitchId)
    mdl.LoadBalancerAttribute.MasterZoneId = reqCtx.Anno.Get(annotation.MasterZoneID)
    mdl.LoadBalancerAttribute.SlaveZoneId = reqCtx.Anno.Get(annotation.SlaveZoneID)
    mdl.LoadBalancerAttribute.ResourceGroupId = reqCtx.Anno.Get(annotation.ResourceGroupId)
    mdl.LoadBalancerAttribute.AddressIPVersion = model.AddressIPVersionType(reqCtx.Anno.Get(annotation.IPVersion))
    mdl.LoadBalancerAttribute.DeleteProtection = model.FlagType(reqCtx.Anno.Get(annotation.DeleteProtection))
    mdl.LoadBalancerAttribute.ModificationProtectionStatus =
        model.ModificationProtectionType(reqCtx.Anno.Get(annotation.ModificationProtection))
    mdl.LoadBalancerAttribute.Tags = reqCtx.Anno.GetLoadBalancerAdditionalTags()
    mdl.LoadBalancerAttribute.Address = reqCtx.Anno.Get(annotation.IP)
    return nil
}

• 然后構(gòu)建后端負(fù)載均衡池蒿柳，主要是解析Service對應(yīng)的Endpoints，主要是解析Endpoints中所有的容器IP和容器端口

func setBackendsFromEndpoints(candidates *backend.EndpointWithENI, vgroup model.VServerGroup) []model.BackendAttribute {
    var backends []model.BackendAttribute

    if len(candidates.Endpoints.Subsets) == 0 {
        return nil
    }
    for _, ep := range candidates.Endpoints.Subsets {
        var backendPort int
        if vgroup.ServicePort.TargetPort.Type == intstr.Int {
            backendPort = vgroup.ServicePort.TargetPort.IntValue()
        } else {
            for _, p := range ep.Ports {
                if p.Name == vgroup.ServicePort.Name {
                    backendPort = int(p.Port)
                    break
                }
            }
            if backendPort == 0 {
                klog.Warningf("%s cannot find port according port name: %s", vgroup.VGroupName, vgroup.ServicePort.Name)
            }
        }

        for _, addr := range ep.Addresses {
            backends = append(backends, model.BackendAttribute{
                NodeName: addr.NodeName,
                ServerIp: addr.IP,
                // set backend port to targetPort by default
                // if backend type is ecs, update backend port to nodePort
                Port:        backendPort,
                Description: vgroup.VGroupName,
            })
        }
    }
    return backends
}

構(gòu)建后端負(fù)載均衡池的時(shí)候會分三種類型處理
1）如果Service指定了externalTrafficPolicy: Local

我們知道這種類型只會將POD所在的那些節(jié)點(diǎn)加入到負(fù)載均衡池里面漩蟆，因此只需要將Endpoints中的那些POD加入到負(fù)載均衡池中即可

這里的ServerId對應(yīng)的就是ECS的nodeid垒探，然后置空ServerIp，通過NodePort方式訪問

for _, backend := range initBackends {
        backend.ServerId = id
        backend.ServerIp = ""
        backend.Type = model.ECSBackendType
        // for ECS backend type, port should be set to NodePort
        backend.Port = int(vgroup.ServicePort.NodePort)
        ecsBackends = append(ecsBackends, backend)
    }

最后在設(shè)置后端負(fù)載均衡池中的各個(gè)后端服務(wù)器的權(quán)重怠李，節(jié)點(diǎn)上存在的POD個(gè)數(shù)就是當(dāng)前節(jié)點(diǎn)的權(quán)重

func podNumberAlgorithm(mode helper.TrafficPolicy, backends []model.BackendAttribute) []model.BackendAttribute {
    
    // LocalTrafficPolicy
    ecsPods := make(map[string]int)
    for _, b := range backends {
        ecsPods[b.ServerId] += 1
    }
    for i := range backends {
        backends[i].Weight = ecsPods[backends[i].ServerId]
    }
    return backends
}

2）如果Service指定了externalTrafficPolicy: Cluster

我們知道這種類型會將所有節(jié)點(diǎn)都加入到負(fù)載均衡池里面圾叼，因此這種模式會將所有節(jié)點(diǎn)都加入到負(fù)載均衡池里

func (mgr *VGroupManager) buildClusterBackends(reqCtx *svcCtx.RequestContext, candidates *backend.EndpointWithENI, vgroup model.VServerGroup) ([]model.BackendAttribute, error) {

    // 1. add ecs backends. add all cluster nodes.
    for _, node := range candidates.Nodes {

        ecsBackends = append(
            ecsBackends,
            model.BackendAttribute{
                ServerId:    id,
                Weight:      DefaultServerWeight,
                Port:        int(vgroup.ServicePort.NodePort),
                Type:        model.ECSBackendType,
                Description: vgroup.VGroupName,
            },
        )
    }

    return setWeightBackends(helper.ClusterTrafficPolicy, backends, vgroup.VGroupWeight), nil
}

然后各個(gè)后端服務(wù)器的權(quán)重都是一樣的

func podNumberAlgorithm(mode helper.TrafficPolicy, backends []model.BackendAttribute) []model.BackendAttribute {
    if mode == helper.ClusterTrafficPolicy {
        for i := range backends {
            backends[i].Weight = 100
        }
        return backends
    }
}

3）還一種是直接使用EIP，通過注解service.beta.kubernetes.io/backend-type: eni

這種對應(yīng)的是Terway網(wǎng)絡(luò)插件直接分配EIP給POD捺癞，然后直接將POD加入到負(fù)載均衡池中

func updateENIBackends(reqCtx *svcCtx.RequestContext, mgr *VGroupManager, backends []model.BackendAttribute, ipVersion model.AddressIPVersionType) (
    []model.BackendAttribute, error) {
    vpcId, err := mgr.cloud.VpcID()

    vpcCIDRs, err := mgr.cloud.DescribeVpcCIDRBlock(context.TODO(), vpcId, ipVersion)

    var ips []string
    for _, b := range backends {
        ips = append(ips, b.ServerIp)
    }
    result, err := mgr.cloud.DescribeNetworkInterfaces(vpcId, ips, ipVersion)

    var skipIPs []string
    for i := range backends {
        eniid, ok := result[backends[i].ServerIp]
        // for ENI backend type, port should be set to targetPort (default value), no need to update
        backends[i].ServerId = eniid
        backends[i].Type = model.ENIBackendType
    }

    return backends, nil
}

然后各個(gè)后端服務(wù)器的權(quán)重也都是一樣的

func podNumberAlgorithm(mode helper.TrafficPolicy, backends []model.BackendAttribute) []model.BackendAttribute {
    if mode == helper.ENITrafficPolicy{
        for i := range backends {
            backends[i].Weight = 100
        }
        return backends
    }
}

• 最后是構(gòu)建Load Balance的監(jiān)聽器夷蚊，也是通過解析Service的注解，支持的注解同樣參考阿里云CLB文檔

func (mgr *ListenerManager) BuildLocalModel(reqCtx *svcCtx.RequestContext, mdl *model.LoadBalancer) error {
    for _, port := range reqCtx.Service.Spec.Ports {
        listener, err := mgr.buildListenerFromServicePort(reqCtx, port)
        mdl.Listeners = append(mdl.Listeners, listener)
    }
    return nil
}

至此髓介，LoadBalance從Service上解析完成了惕鼓，可以看到主要包括三個(gè)方面

• 基本信息，從注解里解析

• 負(fù)載均衡池唐础，從Endpoints中解析

• 監(jiān)聽器箱歧，從注解里解析

LoadBalance解析完成后，然后就要看是否需要創(chuàng)建這個(gè)LoadBalance了一膨，怎么看呢呀邢？

首先需要確定這個(gè)Service是否已經(jīng)關(guān)聯(lián)了已有的LoadBalance，主要是看是否通過注解設(shè)置了service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id

如果指定了這個(gè)注解豹绪，則會先去SLB查詢這個(gè)LoadBalance

func (mgr *LoadBalancerManager) Find(reqCtx *svcCtx.RequestContext, mdl *model.LoadBalancer) error {
    // 1. set load balancer id
    if reqCtx.Anno.Get(annotation.LoadBalancerId) != "" {
        mdl.LoadBalancerAttribute.LoadBalancerId = reqCtx.Anno.Get(annotation.LoadBalancerId)
    }

    // 2. set default loadbalancer name
    // it's safe to set loadbalancer name which will be overwritten in FindLoadBalancer func
    mdl.LoadBalancerAttribute.LoadBalancerName = reqCtx.Anno.GetDefaultLoadBalancerName()

    // 3. set default loadbalancer tag
    // filter tags using logic operator OR, so only TAGKEY tag can be added
    mdl.LoadBalancerAttribute.Tags = []tag.Tag{
        {
            Key:   "kubernetes.do.not.delete",
            Value: reqCtx.Anno.GetDefaultLoadBalancerName(),
        },
    }
    return mgr.cloud.FindLoadBalancer(reqCtx.Ctx, mdl)
}

如果沒有指定LoadBalance价淌，那么就會根據(jù)上面解析出來的LoadBalance信息自動創(chuàng)建一個(gè)

if remote.LoadBalancerAttribute.LoadBalancerId == "" {
        if err := m.slbMgr.Create(reqCtx, local); err != nil {
            
        }
    }

自動創(chuàng)建的LoadBalance名稱和Service的UID有關(guān)

func (n *AnnotationRequest) GetDefaultLoadBalancerName() string {
    //GCE requires that the name of a load balancer starts with a lower case letter.
    ret := "a" + string(n.Service.UID)
    ret = strings.Replace(ret, "-", "", -1)
    //AWS requires that the name of a load balancer is shorter than 32 bytes.
    if len(ret) > 32 {
        ret = ret[:32]
    }
    return ret
}

最終，不管是自動創(chuàng)建的LoadBalance還是使用已有的LoadBalance，到現(xiàn)在是已經(jīng)有了一個(gè)可用的LoadBalance了

但是還需要給這個(gè)LoadBalance設(shè)置負(fù)載均衡池和監(jiān)聽器

而負(fù)載均衡池和監(jiān)聽器就來自上面解析出來的信息了

需要注意的是输钩，如果使用的是已有的LoadBalance豺型，那么會對比已有LoadBalance中的負(fù)載均衡池及監(jiān)聽器 和 上面解析出來的是負(fù)載均衡池及監(jiān)聽器仲智，并且以解析出來的 負(fù)載均衡池及監(jiān)聽器 為準(zhǔn)來更新已有的LoadBalance

比如對負(fù)載均衡池的更新买乃，會以local（解析出來的）中的負(fù)載均衡池為基準(zhǔn)，在remote（SLB中的LoadBalancer）中找對應(yīng)的后端服務(wù)器钓辆，如果找到了剪验，就以local的為準(zhǔn)更新remote中的后端服務(wù)器信息；如果沒有找到前联，就先創(chuàng)建這個(gè)后端服務(wù)器功戚，然后加入到remote中

func (m *ModelApplier) applyVGroups(reqCtx *svcCtx.RequestContext, local *model.LoadBalancer, remote *model.LoadBalancer) error {
    var errs []error
    for i := range local.VServerGroups {
        found := false
        var old model.VServerGroup
        for _, rv := range remote.VServerGroups {
            // for reuse vgroup case, find by vgroup id first
            if local.VServerGroups[i].VGroupId != "" &&
                local.VServerGroups[i].VGroupId == rv.VGroupId {
                found = true
                old = rv
                break
            }
            // find by vgroup name
            if local.VServerGroups[i].VGroupId == "" &&
                local.VServerGroups[i].VGroupName == rv.VGroupName {
                found = true
                local.VServerGroups[i].VGroupId = rv.VGroupId
                old = rv
                break
            }
        }

        // update
        if found {
            if err := m.vGroupMgr.UpdateVServerGroup(reqCtx, local.VServerGroups[i], old); err != nil {
                
            }
        }

        // create
        if !found {
            
            err := m.vGroupMgr.CreateVServerGroup(reqCtx, &local.VServerGroups[i], remote.LoadBalancerAttribute.LoadBalancerId)
            
            if err := m.vGroupMgr.BatchAddVServerGroupBackendServers(reqCtx, local.VServerGroups[i],
                local.VServerGroups[i].Backends); err != nil {
            }
            remote.VServerGroups = append(remote.VServerGroups, local.VServerGroups[i])
        }
    }

}

LoadBalance設(shè)置完成后，會添加Service的標(biāo)簽信息似嗤，其中HASH是根據(jù)Service中的所有字段求出來的

service.beta.kubernetes.io/hash
service.k8s.alibaba/loadbalancer-id

然后再把LoadBalance的地址更新到Service的status里

if len(newStatus.Ingress) == 0 {
        newStatus.Ingress = append(newStatus.Ingress,
            v1.LoadBalancerIngress{
                IP: lb.LoadBalancerAttribute.Address,
            })
    }

刪除CLB

這個(gè)主要對應(yīng)的就是刪除Service或者修改Service的類型

func NeedDeleteLoadBalancer(svc *v1.Service) bool {
    return svc.DeletionTimestamp != nil || svc.Spec.Type != v1.ServiceTypeLoadBalancer
}

這時(shí)候就只會處理那些帶有Finalizer的啸臀，而且是自動創(chuàng)建的LoadBalancer，將其自動刪除

if helper.NeedDeleteLoadBalancer(reqCtx.Service) {
        if !local.LoadBalancerAttribute.IsUserManaged {
            err := m.slbMgr.Delete(reqCtx, remote)
            
            remote.LoadBalancerAttribute.LoadBalancerId = ""
            remote.LoadBalancerAttribute.Address = ""
            return nil
        }
    }

然后再移除Service的標(biāo)簽烁落、status乘粒、Finalizer信息

NLB控制器

最后一個(gè)默認(rèn)會啟用的控制器，對應(yīng)的是阿里云的NLB負(fù)載均衡伤塌，代碼流程和CLB幾乎是一模一樣

只是由于NLB的自身產(chǎn)品實(shí)現(xiàn)和CLB的不同灯萍，所以最終的效果才不同