Bare Metal Service Load Balancers

AKA "how to set up a bank of haproxy for platforms that don't have load balancers".

Disclaimer:

• This is a work in progress.
• A better way to achieve this will probably emerge once discussions on (#260, #561) converge.
• Backends are pluggable, but Haproxy is the only loadbalancer with a working implementation.
• I have never deployed haproxy to production, so contributions are welcome (see wishlist for ideas).
• For fault tolerant load balancing of ingress traffic, you need:
  1. Multiple hosts running load balancers
  2. Multiple A records for each hostname in a DNS service.

This module will not help with the latter

Overview

Ingress

There are 2 ways to expose a service to ingress traffic in the current kubernetes service model:

• Create a cloud load balancer.
• Allocate a port (the same port) on every node in your cluster and proxy ingress traffic through that port to the endpoints.

The service-loadbalancer aims to give you 1 on bare metal, making 2 unnecessary for the common case. The replication controller manifest in this directly creates a service-loadbalancer pod on all nodes with the role=loadbalancer label. Each service-loadbalancer pod contains:

• A load balancer controller that watches the kubernetes api for services and endpoints.
• A load balancer manifest. This is used to bootstrap the load balancer. The load balancer itself is pluggable, so you can easily swap
  haproxy for something like f5 or pound.
• A template used to write load balancer rules. This is tied to the loadbalancer used in the manifest, since each one has a different config format.

L7 load balancing of Http services: The load balancer controller automatically exposes http services to ingress traffic on all nodes with a role=loadbalancer label. It assumes all services are http unless otherwise instructed. Each http service gets a loadbalancer forwarding rule, such that requests received on http://loadbalancer-node/serviceName:port balanced between its endpoints according to the algorithm specified in the loadbalacer.json manifest. You do not need more than a single loadbalancer pod to balance across all your http services (you can scale the rc to increase capacity).

L4 loadbalancing of Tcp services: Since one needs to specify ports at pod creation time (kubernetes doesn't currently support port ranges), a single loadbalancer is tied to a set of preconfigured node ports, and hence a set of TCP services it can expose. The load balancer controller will dynamically add rules for each configured TCP service as it pops into existence. However, each "new" (unspecified in the tcpServices section of the loadbalancer.json) service will need you to open up a new container-host port pair for traffic. You can achieve this by creating a new loadbalancer pod with the targetPort set to the name of your service, and that service specified in the tcpServices map of the new loadbalancer.

Cross-cluster loadbalancing

On cloud providers that offer a private ip range for all instances on a network, you can setup multiple clusters in different availability zones, on the same network, and loadbalancer services across these zones. On GCE for example, every instance is a member of a single network. A network performs the same function that a router does: it defines the network range and gateway IP address, handles communication between instances, and serves as a gateway between instances and other networks. On such networks the endpoints of a service in one cluster are visible in all other clusters in the same network, so you can setup an edge loadbalancer that watches a kubernetes master of another cluster for services. Such a deployment allows you to fallback to a different AZ during times of duress or planned downtime (eg: database update).

Examples

Initial cluster state:

$ kubectl get svc --all-namespaces -o yaml  | grep -i "selfLink"
    selfLink: /api/v1/namespaces/default/services/kubernetes
    selfLink: /api/v1/namespaces/default/services/nginxsvc
    selfLink: /api/v1/namespaces/kube-system/services/elasticsearch-logging
    selfLink: /api/v1/namespaces/kube-system/services/kibana-logging
    selfLink: /api/v1/namespaces/kube-system/services/kube-dns
    selfLink: /api/v1/namespaces/kube-system/services/kube-ui
    selfLink: /api/v1/namespaces/kube-system/services/monitoring-grafana
    selfLink: /api/v1/namespaces/kube-system/services/monitoring-heapster
    selfLink: /api/v1/namespaces/kube-system/services/monitoring-influxdb

These are all the cluster addon services in namespace=kube-system.

Create a loadbalancer

• Loadbalancers are created via a ReplicationController.
• Load balancers will only run on nodes with the role=loadbalancer label.

$ kubectl create -f ./rc.yaml
replicationcontrollers/service-loadbalancer
$  kubectl get pods -l app=service-loadbalancer
NAME                         READY     STATUS    RESTARTS   AGE
service-loadbalancer-dapxv   0/2       Pending   0          1m
$ kubectl describe pods -l app=service-loadbalancer
Events:
  FirstSeen                                    From            Reason                  Message
  Tue, 21 Jul 2015 11:19:22 -0700              {scheduler }    failedScheduling        Failed for reason MatchNodeSelector and possibly others

Notice that the pod hasn't started because the scheduler is waiting for you to tell it which nodes to use as a load balancer.

$ kubectl label node e2e-test-beeps-minion-c9up role=loadbalancer
NAME                         LABELS                                                                STATUS
e2e-test-beeps-minion-c9up   kubernetes.io/hostname=e2e-test-beeps-minion-c9up,role=loadbalancer   Ready

Expose services

Let's create 3 services (HTTP, HTTPS and TCP) to test the loadbalancer.

HTTP

You can use the https-nginx example to create some new HTTP/HTTPS services.

$ cd ../../examples/https-nginx
$ make keys secret KEY=/tmp/nginx.key CERT=/tmp/nginx.crt SECRET=/tmp/secret.json
$ kubectl create -f /tmp/secret.json
$ kubectl get secrets
NAME                  TYPE                                  DATA
default-token-vklfs   kubernetes.io/service-account-token   2
nginxsecret           Opaque                                2

Lets introduce a small twist. The nginx-app example exposes the nginx service using NodePort, which means it opens up a random port on every node in your cluster and exposes the service on that. Delete the type: NodePort line before creating it.

$ kubectl create -f nginx-app.yaml
$ kubectl get svc
NAME         LABELS                                    SELECTOR    IP(S)         PORT(S)
kubernetes   component=apiserver,provider=kubernetes   <none>      10.0.0.1      443/TCP
nginxsvc     app=nginx                                 app=nginx   10.0.79.131   80/TCP
                                                                                 443/TCP
$ curl http://104.197.63.17/nginxsvc

HTTPS

HTTPS services are handled at L4 (see wishlist)

$ curl https://104.197.63.17:8080 -k

A couple of points to note:

• The nginxsvc is specified in the tcpServices of the loadbalancer.json manifest.
• The https service is accessible directly on the specified port, which matches the service port.
• You need to take care of ensuring there is no collision between these service ports on the node.

SSL Termination

To terminate SSL for a service you just need to annotate the service with serviceloadbalancer/lb.sslTerm: "true" as seen below. This will cause your service to be served behind /{service-name} or /{service-name}:{port} if not running on port 80. This mimics the standard http functionality.

metadata:
  name: myservice
  annotations:
    serviceloadbalancer/lb.sslTerm: "true"
  labels:

• Create a secret with one of your favorite tool

• Add secrets to your loadbalancer pod

        ports:
        # All http services
        - containerPort: 80
          hostPort: 80
          protocol: TCP
        # ssl term
        - containerPort: 443
          hostPort: 443
          protocol: TCP
        # haproxy stats
        - containerPort: 1936
          hostPort: 1936
          protocol: TCP
        resources: {}
        volumeMounts:
        - mountPath: "/ssl"
          name: secret-volume
    volumes:
    - name: secret-volume
      secret:
        secretName: my-secret
       

• Add your SSL configuration to loadbalancer pod

      args:
      - --ssl-cert=/ssl/crt.pem
      - --ssl-ca-cert=/ssl/ca.crt
      - --namespace=default

Custom ACL

• Adding the aclMatch annotation will allow you to serve the service on a specific path although URLs will not be rewritten back to root. The following will cause your service to be available at /test and your web service will be passed the url with /test on the front.

 metadata:
   name: myservice
   annotations:
     serviceloadbalancer/lb.sslTerm: "true"
     serviceloadbalancer/lb.aclMatch: "-i /test"
   labels:

TCP

$ cat mysql-app.yaml
apiVersion: v1
kind: Pod
metadata:
  name: mysql
  labels:
    name: mysql
spec:
  containers:
  - image: mysql
    name: mysql
    env:
    - name: MYSQL_ROOT_PASSWORD
      # Use secrets instead of env for passwords
      value: password
    ports:
    - containerPort: 3306
      name: mysql
    volumeMounts:
    # name must match the volume name below
    - name: mysql-storage
      # mount path within the container
      mountPath: /var/lib/mysql
  volumes:
  - name: mysql-storage
    emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  labels:
    name: mysql
  name: mysql
spec:
  type: NodePort
  ports:
    # the port that this service should serve on
    - port: 3306
  # label keys and values that must match in order to receive traffic for this service
  selector:
    name: mysql

We'll create the service and access mysql from outside the cluster:

$ kubectl create -f mysql-app.yaml
$ kubeclt get svc
NAME         LABELS                                    SELECTOR    IP(S)         PORT(S)
kubernetes   component=apiserver,provider=kubernetes   <none>      10.0.0.1      443/TCP
nginxsvc     app=nginx                                 app=nginx   10.0.79.131   80/TCP
                                                                                 443/TCP
mysql        app=mysql                                 app=mysql   10.0.63.72    3306/TCP

$ mysql -u root -ppassword --host 104.197.63.17 --port 3306 -e 'show databases;'
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
+--------------------+

Cross-namespace loadbalancing

By default, the loadbalancer only listens for services in the default namespace. You can list available namespaces via:

$ kubectl get namespaces
NAME          LABELS    STATUS    AGE
default       <none>    Active    1d
kube-system   <none>    Active    1d

You can tell it to expose services on a different namespace through a command line argument. Currently, each namespace needs a different loadbalancer (see wishlist). Modify the rc.yaml file to supply the namespace argument by adding the following lines to the bottom of the loadbalancer spec:

args:
  - --tcp-services=mysql:3306,nginxsvc:443
  - --namespace=kube-system

Though the loadbalancer can watch services across namespaces you can't start 2 loadbalancers with the same name in a single namespace. So if you already have a loadbalancer running, either change the name of the rc, or change the namespace in rc.yaml:

$ kubectl create -f rc.yaml
$ kubectl get pods -o wide
NAME                         READY     STATUS    RESTARTS   AGE       NODE
service-loadbalancer-yofyv   1/1       Running   0          1m        e2e-test-beeps-minion-c9up

$ kubectl get nodes e2e-test-beeps-minion-c9up -o json | grep -i externalip -A 1
                "type": "ExternalIP",
                "address": "104.197.63.17"
$ curl http://104.197.63.17/kube-ui

Cross-cluster loadbalancing

First setup your 2 clusters, and a kubeconfig secret as described in the [sharing clusters example] (../../examples/sharing-clusters/README.md). We will create a loadbalancer in our first cluster (US) and have it publish the services from the second cluster (EU). This is the entire modified loadbalancer manifest:

apiVersion: v1
kind: ReplicationController
metadata:
  name: service-loadbalancer
  labels:
    app: service-loadbalancer
    version: v1
spec:
  replicas: 1
  selector:
    app: service-loadbalancer
    version: v1
  template:
    metadata:
      labels:
        app: service-loadbalancer
        version: v1
    spec:
      volumes:
      # token from the eu cluster, must already exist
      # and match the name of the volume using in container
      - name: eu-config
        secret:
          secretName: kubeconfig
      nodeSelector:
        role: loadbalancer
      containers:
      - image: gcr.io/google_containers/servicelb:0.4
        imagePullPolicy: Always
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8081
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        name: haproxy
        ports:
        # All http services
        - containerPort: 80
          hostPort: 80
          protocol: TCP
        # nginx https
        - containerPort: 443
          hostPort: 8080
          protocol: TCP
        # mysql
        - containerPort: 3306
          hostPort: 3306
          protocol: TCP
        # haproxy stats
        - containerPort: 1936
          hostPort: 1936
          protocol: TCP
        resources: {}
        args:
        - --tcp-services=mysql:3306,nginxsvc:443
        - --use-kubernetes-cluster-service=false
        # use-kubernetes-cluster-service=false in conjunction with the
        # kube/config will force the service-loadbalancer to watch for
        # services form the eu cluster.
        volumeMounts:
        - mountPath: /.kube
          name: eu-config
        env:
        - name: KUBECONFIG
          value: /.kube/config

Note that it is essentially the same as the rc.yaml checked into the service-loadbalancer directory expect that it consumes the kubeconfig secret as an extra KUBECONFIG environment variable.

$ kubectl config use-context <us-clustername>
$ kubectl create -f rc.yaml
$ kubectl get pods -o wide
service-loadbalancer-5o2p4   1/1       Running   0          13m       kubernetes-minion-5jtd
$ kubectl get node kubernetes-minion-5jtd -o json | grep -i externalip -A 2
                "type": "ExternalIP",
                "address": "104.197.81.116"
$ curl http://104.197.81.116/nginxsvc
Europe

Advanced features

• Sticky sessions: Currently undocumented but possible via annotations.
• Name based virtual hosting: Currently undocumented but possible via annotations.
• Configurable algorithms: Currently undocumented but possible via annotations.

Troubleshooting:

• If you can curl or netcat the endpoint from the pod (with kubectl exec) and not from the node, you have not specified hostport and containerport.
• If you can hit the ips from the node but not from your machine outside the cluster, you have not opened firewall rules for the right network.
• If you can't hit the ips from within the container, either haproxy or the service_loadbalacer script is not running.
  1. Use ps in the pod
  2. sudo restart haproxy in the pod
  3. cat /etc/haproxy/haproxy.cfg in the pod
  4. try kubectl logs haproxy
  5. run the service_loadbalancer with --dry
• Check http://<node_ip>:1936 for the stats page. It requires the password used in the template file.
• Try talking to haproxy on the stats socket directly on the container using kubectl exec, eg: echo “show info” | socat unix-connect:/tmp/haproxy stdio
• Run the service_loadbalancer with the flag --syslog to append the haproxy log as part of the pod stdout. Use kubectl logs to check the
  status of the services or stats about the traffic

Wishlist:

• Allow services to specify their url routes (see openshift routes)

• Scrape :1926 and scale replica count of the loadbalancer rc from a helper pod (this is basically ELB)

• Scrape :1936/;csv and autoscale services

• Better https support. 3 options to handle ssl:

  1. Pass Through: Load balancer drops down to L4 balancing and forwards TCP encrypted packets to destination.
  2. Redirect: All traffic is https. HTTP connections are encrypted using load balancer certs.

  Currently you need to trigger TCP loadbalancing for your https service by specifying it in loadbalancer.json. Support for the other 2 would be nice.

• Multinamespace support: Currently the controller only watches a single namespace for services.

• Support for external services (eg: amazon rds)

• Dynamically modify loadbalancer.json. Will become unnecessary when we have a loadbalancer resource.

• Headless services: I just didn't think people would care enough about this.

Analytics