在 K8s 中身隐，kube-apiserver 使用 etcd 對 REST object 資源進(jìn)行持久化存儲廷区，本文介紹如何配置生成自簽 https 證書，搭建 etcd 集群給 apiserver 使用贾铝，并附相關(guān)坑點記錄隙轻。

1. 安裝 cfssl 工具

cd /data/work

wget https://github.com/cloudflare/cfssl/releases/download/v1.6.0/cfssl_1.6.0_linux_amd64 -O cfssl
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.0/cfssljson_1.6.0_linux_amd64 -O cfssljson
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.0/cfssl-certinfo_1.6.0_linux_amd64 -O cfssl-certinfo

chmod +x cfssl*
mv cfssl* /usr/local/bin/


chmod +x cfssl*
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo

2. 創(chuàng)建 ca 證書

cat > ca-csr.json <<EOF
{
  "CN": "etcd-ca",
  "key": {
      "algo": "rsa",
      "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "etcd-ca",
      "OU": "etcd-ca"
    }
  ],
  "ca": {
          "expiry": "87600h"
  }
}
EOF


cfssl gencert -initca ca-csr.json | cfssljson -bare ca

=> 會生成：ca-key.pem, ca.csr, ca.pem

3. 配置 ca 證書策略

cat > ca-config.json <<EOF
{
  "signing": {
      "default": {
          "expiry": "87600h"
        },
      "profiles": {
          "etcd-ca": {
              "usages": [
                  "signing",
                  "key encipherment",
                  "server auth",
                  "client auth"
              ],
              "expiry": "87600h"
          }
      }
  }
}
EOF

4. 配置 etcd 請求 csr

cat > etcd-csr.json <<EOF
{
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
    "etcd0-0.etcd",
    "etcd1-0.etcd",
    "etcd2-0.etcd"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [{
    "C": "CN",
    "ST": "Beijing",
    "L": "Beijing",
    "O": "etcd",
    "OU": "etcd"
  }]
}
EOF

5. 生成 etcd 證書

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=etcd-ca etcd-csr.json | cfssljson  -bare etcd

=> 會生成：etcd-key.pem, etcd.csr, etcd.pem

6. 創(chuàng)建 etcd cluster

yaml 文件：https://github.com/k8s-club/etcd-operator

kubectl apply -f etcd-cluster.yaml

7. 查看 DNS 解析

dnsutils 安裝：https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/

kubectl exec -it -n etcd dnsutils -- nslookup etcd

Server:         9.165.x.x
Address:        9.165.x.x#53

Name:   etcd.etcd.svc.cluster.local
Address: 9.165.x.x
Name:   etcd.etcd.svc.cluster.local
Address: 9.165.x.x
Name:   etcd.etcd.svc.cluster.local
Address: 9.165.x.x

8. 查看 etcd 集群狀態(tài)

kubectl exec -it -n etcd etcd0-0 -- sh

/usr/local/bin/etcdctl --write-out=table --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints=https://etcd0-0.etcd:2379,https://etcd1-0.etcd:2379,https://etcd2-0.etcd:2379 endpoint health

+---------------------------+--------+-------------+-------+
|         ENDPOINT          | HEALTH |    TOOK     | ERROR |
+---------------------------+--------+-------------+-------+
| https://etcd0-0.etcd:2379 |   true | 13.551982ms |       |
| https://etcd1-0.etcd:2379 |   true | 13.540498ms |       |
| https://etcd2-0.etcd:2379 |   true | 23.119639ms |       |
+---------------------------+--------+-------------+-------+

/usr/local/bin/etcdctl --write-out=table --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints=https://etcd0-0.etcd:2379,https://etcd1-0.etcd:2379,https://etcd2-0.etcd:2379 endpoint status

+--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|         ENDPOINT         |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| http://etcd0-0.etcd:2379 | 4dde210279eea33a |  3.4.13 |   20 kB |      true |      false |         2 |          9 |                  9 |        |
| http://etcd1-0.etcd:2379 | 20669865d12a473b |  3.4.13 |   20 kB |     false |      false |         2 |          9 |                  9 |        |
| http://etcd2-0.etcd:2379 | 3f17922d1ed63113 |  3.4.13 |   20 kB |     false |      false |         2 |          9 |                  9 |        |
+--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

9. 驗證 etcd 讀寫

kubectl exec -it -n etcd etcd0-0 -- sh

/usr/local/bin/etcdctl --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem put hello world
OK

/usr/local/bin/etcdctl --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem get hello
hello
world

查看所有 keys:
/usr/local/bin/etcdctl --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem get "" --keys-only --prefix
hello

查看所有 key-val:
/usr/local/bin/etcdctl --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem get "" --prefix
hello
world

10. 配置 apiserver 請求 csr

cat > apiserver-csr.json <<EOF
{
  "CN": "apiserver",
  "hosts": [
    "*.etcd"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [{
    "C": "CN",
    "ST": "Beijing",
    "L": "Beijing",
    "O": "apiserver",
    "OU": "apiserver"
  }]
}
EOF

11. 生成 apiserver 證書

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=etcd-ca apiserver-csr.json | cfssljson  -bare apiserver

=> 會生成：apiserver-key.pem, apiserver.csr, apiserver.pem

12. 創(chuàng)建 extension-apiserver

apiserver.yaml：通過 ConfigMap 將生成的 *.pem 證書掛載給 apiserver 使用

containers:
  - image: xxxxx:latest
    args:
    - --etcd-servers=https://etcd0-0.etcd:2379
    - --etcd-cafile=/etc/kubernetes/certs/kube-apiserver-etcd-ca.crt
    - --etcd-certfile=/etc/kubernetes/certs/kube-apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/certs/kube-apiserver-etcd-client.key

kubectl apply -f apiserver.yaml

13. 坑點記錄

13.1 證書 hosts 不對

log:
etcd0-0:
{"level":"warn","ts":"2021-08-19T11:55:07.755Z","caller":"embed/config_logging.go:279","msg":"rejected connection","remote-addr":"127.0.0.1:41226","server-name":"","error":"tls: first record does not look like a TLS handshake"}


etcd1-0:
{"level":"info","ts":"2021-08-19T11:54:16.830Z","caller":"embed/serve.go:191","msg":"serving client traffic securely","address":"[::]:2379"}
{"level":"info","ts":"2021-08-19T11:54:16.838Z","caller":"etcdserver/server.go:716","msg":"initialized peer connections; fast-forwarding election ticks","local-member-id":"30dd90df9a304e97","forward-ticks":18,"forward-duration":"4.5s","election-ticks":20,"election-timeout":"5s","active-remote-members":2}
{"level":"info","ts":"2021-08-19T11:54:16.867Z","caller":"membership/cluster.go:558","msg":"set initial cluster version","cluster-id":"80c7f1f6c2848777","local-member-id":"30dd90df9a304e97","cluster-version":"3.4"}
{"level":"info","ts":"2021-08-19T11:54:16.867Z","caller":"api/capability.go:76","msg":"enabled capabilities for version","cluster-version":"3.4"}


etcd2-0:
{"level":"warn","ts":"2021-08-19T11:54:17.782Z","caller":"embed/config_logging.go:270","msg":"rejected connection","remote-addr":"9.165.x.x:40180","server-name":"etcd2-0.etcd","ip-addresses":["0.0.0.0","127.0.0.1"],"dns-names":["etcd0-0.etcd","etcd1-0.etcd","etcd2-0.etcd"],"error":"tls: \"9.165.x.x\" does not match any of DNSNames [\"etcd0-0.etcd\" \"etcd1-0.etcd\" \"etcd2-0.etcd\"] (lookup etcd1-0.etcd on 9.165.x.x:53: no such host)"}

解決：重新配置正確的 hosts 域名

13.2 證書 hosts 配置坑點

"hosts": [
    "127.0.0.1",
    "etcd0-0.etcd",
    "*.etcd" // 允許 * 泛域名，但不能為空 "" 或 *
  ],

13.3 dns 設(shè)置參考

推薦設(shè)置 *.xxx.ns.svc垢揩，這樣擴(kuò)容后也不需要重簽證書

參考：https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/

Go 代碼參考如下：

func genEtcdWildcardDnsName(namespace, serviceName string) []string {
    return []string{
        fmt.Sprintf("%s.%s.%s", serviceName, namespace, "svc"),
        fmt.Sprintf("*.%s.%s.%s", serviceName, namespace, "svc"),
        fmt.Sprintf("%s.%s.%s", serviceName, namespace, DnsBase),
        fmt.Sprintf("*.%s.%s.%s", serviceName, namespace, DnsBase),
    }
}

13.4 leader/follower 已經(jīng)建立成功了玖绿，但訪問報錯

# /usr/local/bin/etcdctl put hello world
{"level":"warn","ts":"2021-08-19T12:32:11.200Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-05ed1825-e70f-492a-af94-03c633d0affc/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: all SubConns are in TransientFailure, latest connection error: connection closed"}
Error: context deadline exceeded

解決：etcdctl 需要帶證書訪問

/usr/local/bin/etcdctl --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem put hello world

13.5 http 與 https 之間不能切換

先通過 http 建立了 cluster，然后再用自簽證書 https 來建立叁巨，這樣就會報錯：

tls: first record does not look like a TLS handshake

經(jīng)過驗證：無論是從 http => https斑匪，還是從 https => http 的切換都會報這個錯，因為一旦建立 cluster 成功锋勺，則把連接的協(xié)議(http/https) 寫入到 etcd 存儲里了蚀瘸，不能再更改連接協(xié)議。

解決：如果真正遇到需要切換協(xié)議庶橱，可嘗試下面方式

• 允許刪除數(shù)據(jù)：刪除后重新建立 cluster
• 不允許刪數(shù)據(jù)：可以嘗試采用 snapshot & restore 進(jìn)行快照與恢復(fù)操作

13.6 apiserver 可直接使用第 5 步生成的 etcd 證書嗎贮勃？

經(jīng)過驗證，是可以直接使用 etcd 證書的苏章，但生產(chǎn)上不建議這樣使用衙猪。

生產(chǎn)上建議對 apiserver(或其他應(yīng)用) 單獨生成證書，可使用泛域名(*.xx.xx)布近、不同過期時間等方式靈活配置，也更有利于集群管控丝格。