- 1， 生成 TLS 秘鑰對(duì)
- 2也物，拷貝密鑰對(duì)到所有節(jié)點(diǎn)
- 3宫屠，配置 etcd 使用證書
- 4，測(cè)試 etcd 是否正常
- 5滑蚯，配置 kube-apiserver 使用 CA 連接 etcd
- 6浪蹂，測(cè)試 kube-apiserver
- 7，未解決的問(wèn)題
SSL/TSL 認(rèn)證分單向認(rèn)證和雙向認(rèn)證兩種方式告材。簡(jiǎn)單說(shuō)就是單向認(rèn)證只是客戶端對(duì)服務(wù)端的身份進(jìn)行驗(yàn)證坤次，雙向認(rèn)證是客戶端和服務(wù)端互相進(jìn)行身份認(rèn)證。就比如斥赋，我們登錄淘寶買東西缰猴，為了防止我們登錄的是假淘寶網(wǎng)站，此時(shí)我們通過(guò)瀏覽器打開(kāi)淘寶買東西時(shí)疤剑，瀏覽器會(huì)驗(yàn)證我們登錄的網(wǎng)站是否是真的淘寶的網(wǎng)站滑绒，而淘寶網(wǎng)站不關(guān)心我們是否“合法”闷堡，這就是單向認(rèn)證。而雙向認(rèn)證是服務(wù)端也需要對(duì)客戶端做出認(rèn)證疑故。
因?yàn)榇蟛糠?kubernetes 基于內(nèi)網(wǎng)部署杠览，而內(nèi)網(wǎng)應(yīng)該都會(huì)采用私有 IP 地址通訊，權(quán)威 CA 好像只能簽署域名證書纵势，對(duì)于簽署到 IP 可能無(wú)法實(shí)現(xiàn)倦零。所以我們需要預(yù)先自建 CA 簽發(fā)證書。
Generate self-signed certificates 官方參考文檔
官方推薦使用 cfssl 來(lái)自建 CA 簽發(fā)證書吨悍，當(dāng)然你也可以用眾人熟知的 OpenSSL 或者 easy-rsa扫茅。以下步驟遵循官方文檔：
1， 生成 TLS 秘鑰對(duì)
生成步驟：
- 1育瓜，下載 cfssl
- 2葫隙，初始化證書頒發(fā)機(jī)構(gòu)
- 3，配置 CA 選項(xiàng)
- 4躏仇，生成服務(wù)器端證書
- 5恋脚，生成對(duì)等證書
- 6，生成客戶端證書
想深入了解 HTTPS 的看這里：
- 聊聊HTTPS和SSL/TLS協(xié)議
- 數(shù)字證書CA及掃盲
- 互聯(lián)網(wǎng)加密及OpenSSL介紹和簡(jiǎn)單使用
- SSL雙向認(rèn)證和單向認(rèn)證的區(qū)別
1焰手，下載 cfssl
mkdir ~/bin
curl -s -L -o ~/bin/cfssl https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
curl -s -L -o ~/bin/cfssljson https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
chmod +x ~/bin/{cfssl,cfssljson}
export PATH=$PATH:~/bin
2糟描，初始化證書頒發(fā)機(jī)構(gòu)
mkdir ~/cfssl
cd ~/cfssl
cfssl print-defaults config > ca-config.json
cfssl print-defaults csr > ca-csr.json
證書類型介紹：
- client certificate 用于通過(guò)服務(wù)器驗(yàn)證客戶端。例如etcdctl书妻，etcd proxy船响，fleetctl或docker客戶端。
- server certificate 由服務(wù)器使用躲履，并由客戶端驗(yàn)證服務(wù)器身份见间。例如docker服務(wù)器或kube-apiserver。
- peer certificate 由 etcd 集群成員使用工猜，供它們彼此之間通信使用米诉。
3，配置 CA 選項(xiàng)
$ cat << EOF > ca-config.json
{
"signing": {
"default": {
"expiry": "43800h"
},
"profiles": {
"server": {
"expiry": "43800h",
"usages": [
"signing",
"key encipherment",
"server auth"
]
},
"client": {
"expiry": "43800h",
"usages": [
"signing",
"key encipherment",
"client auth"
]
},
"peer": {
"expiry": "43800h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
$ cat << EOF > ca-csr.json
{
"CN": "My own CA",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "US",
"L": "CA",
"O": "My Company Name",
"ST": "San Francisco",
"OU": "Org Unit 1",
"OU": "Org Unit 2"
}
]
}
生成 CA 證書：
$ cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
將會(huì)生成以下幾個(gè)文件：
ca-key.pem
ca.csr
ca.pem
請(qǐng)務(wù)必保證 ca-key.pem 文件的安全篷帅，*.csr 文件在整個(gè)過(guò)程中不會(huì)使用史侣。
4，生成服務(wù)器端證書
$ echo '{"CN":"coreos1","hosts":["10.93.81.17","127.0.0.1"],"key":{"algo":"rsa","size":2048}}' | cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server -hostname="10.93.81.17,127.0.0.1,server" - | cfssljson -bare server
hosts 字段需要自定義魏身。
然后將得到以下幾個(gè)文件：
server-key.pem
server.csr
server.pem
5惊橱，生成對(duì)等證書
$ echo '{"CN":"member1","hosts":["10.93.81.17","127.0.0.1"],"key":{"algo":"rsa","size":2048}}' | cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer -hostname="10.93.81.17,127.0.0.1,server,member1" - | cfssljson -bare member1
hosts 字段需要自定義。
然后將得到以下幾個(gè)文件：
member1-key.pem
member1.csr
member1.pem
如果有多個(gè) etcd 成員叠骑，重復(fù)此步為每個(gè)成員生成對(duì)等證書李皇。
6，生成客戶端證書
$ echo '{"CN":"client","hosts":["10.93.81.17","127.0.0.1"],"key":{"algo":"rsa","size":2048}}' | cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client - | cfssljson -bare client
hosts 字段需要自定義。
然后將得到以下幾個(gè)文件：
client-key.pem
client.csr
client.pem
至此掉房，所有證書都已生成完畢茧跋。
2，拷貝密鑰對(duì)到所有節(jié)點(diǎn)
- 1卓囚，拷貝密鑰對(duì)到所有節(jié)點(diǎn)
- 2瘾杭，更新系統(tǒng)證書庫(kù)
1，拷貝密鑰對(duì)到所有節(jié)點(diǎn)
$ mkdir -pv /etc/ssl/etcd/
$ cp ~/cfssl/* /etc/ssl/etcd/
$ chown -R etcd:etcd /etc/ssl/etcd
$ chmod 600 /etc/ssl/etcd/*-key.pem
$ cp ~/cfssl/ca.pem /etc/ssl/certs/
2哪亿，更新系統(tǒng)證書庫(kù)
$ yum install ca-certificates -y
$ update-ca-trust
3粥烁，配置 etcd 使用證書
$ etcdctl version
etcdctl version: 3.1.3
API version: 3.1
$ cat /etc/etcd/etcd.conf
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
#監(jiān)聽(tīng)URL，用于與其他節(jié)點(diǎn)通訊
ETCD_LISTEN_PEER_URLS="https://10.93.81.17:2380"
#告知客戶端的URL, 也就是服務(wù)的URL
ETCD_LISTEN_CLIENT_URLS="https://10.93.81.17:2379,https://10.93.81.17:4001"
#表示監(jiān)聽(tīng)其他節(jié)點(diǎn)同步信號(hào)的地址
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.93.81.17:2380"
#–advertise-client-urls 告知客戶端的URL, 也就是服務(wù)的URL蝇棉，tcp2379端口用于監(jiān)聽(tīng)客戶端請(qǐng)求
ETCD_ADVERTISE_CLIENT_URLS="https://10.93.81.17:2379"
#啟動(dòng)參數(shù)配置
ETCD_NAME="node1"
ETCD_INITIAL_CLUSTER="node1=https://10.93.81.17:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
#[security]
ETCD_CERT_FILE="/etc/ssl/etcd/server.pem"
ETCD_KEY_FILE="/etc/ssl/etcd/server-key.pem"
ETCD_TRUSTED_CA_FILE="/etc/ssl/etcd/ca.pem"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_PEER_CERT_FILE="/etc/ssl/etcd/member1.pem"
ETCD_PEER_KEY_FILE="/etc/ssl/etcd/member1-key.pem"
ETCD_PEER_TRUSTED_CA_FILE="/etc/ssl/etcd/ca.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
#[logging]
ETCD_DEBUG="true"
ETCD_LOG_PACKAGE_LEVELS="etcdserver=WARNING,security=DEBUG"
4讨阻，測(cè)試 etcd 是否正常
$ systemctl restart etcd
如果報(bào)錯(cuò)，使用 journalctl -f -t etcd 和 journalctl -u etcd 來(lái)定位問(wèn)題篡殷。
$ curl --cacert /etc/ssl/etcd/ca.pem --cert /etc/ssl/etcd/client.pem --key /etc/ssl/etcd/client-key.pem https://10.93.81.17:2379/health
{"health": "true"}
$ etcdctl --endpoints=[10.93.81.17:2379] --cacert=/etc/ssl/etcd/ca.pem --cert=/etc/ssl/etcd/client.pem --key=/etc/ssl/etcd/client-key.pem member list
$ etcdctl --endpoints=[10.93.81.17:2379] --cacert=/etc/ssl/etcd/ca.pem --cert=/etc/ssl/etcd/client.pem --key=/etc/ssl/etcd/client-key.pem put /foo/bar "hello world"
$ etcdctl --endpoints=[10.93.81.17:2379] --cacert=/etc/ssl/etcd/ca.pem --cert=/etc/ssl/etcd/client.pem --key=/etc/ssl/etcd/client-key.pem get /foo/bar
5钝吮，配置 kube-apiserver 使用 CA 連接 etcd
$ cp /etc/ssl/etcd/* /var/run/kubernetes/
$ chown -R kube.kube /var/run/kubernetes/
在 /etc/kubernetes/apiserver 中 KUBE_API_ARGS 新加一下幾個(gè)參數(shù)：
--cert-dir='/var/run/kubernetes/' --etcd-cafile='/var/run/kubernetes/ca.pem' --etcd-certfile='/var/run/kubernetes/client.pem' --etcd-keyfile='/var/run/kubernetes/client-key.pem'
6，測(cè)試 kube-apiserver
$ systemctl restart kube-apiserver kube-controller-manager kube-scheduler kubelet kube-proxy
$ systemctl status -l kube-apiserver kube-controller-manager kube-scheduler kubelet kube-proxy
$ kubectl get node
$ kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Unhealthy Get https://10.93.81.17:2379/health: remote error: tls: bad certificate
$ ./version.sh
etcdctl version: 3.1.3
API version: 3.1
Kubernetes v1.6.0-beta.1
7板辽，未解決的問(wèn)題
1奇瘦，使用 kubectl get cs 查看會(huì)出現(xiàn)如上面所示的報(bào)錯(cuò)：
etcd-0 Unhealthy Get https://10.93.81.17:2379/health: remote error: tls: bad certificate
此問(wèn)題有人提交 pr 但尚未被 merge，etcd component status check should include credentials
2劲弦，使用以下命令查看到的 2380 端口是未加密的
$ etcdctl --endpoints=[10.93.81.17:2379] --cacert=/etc/ssl/etcd/ca.pem --cert=/etc/ssl/etcd/client.pem --key=/etc/ssl/etcd/client-key.pem member list
2017-03-15 15:02:05.611564 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
145b401ad8709f51, started, node1, http://10.93.81.17:2380, https://10.93.81.17:2379
參考文檔：
