安裝kafka
安裝zookeeper
安裝zookeeper可以采用k8s的StatefulSet方式,非常方便,在安裝kafka過程中遇到很多問題,但是zookeeper非常文檔
apiVersion: v1
kind: Service
metadata:
name: zk-hs
labels:
app: zk
namespace: kafka
spec:
selector:
app: zk
clusterIP: None
ports:
- name: server
port: 2888
- name: leader-election
port: 3888
---
apiVersion: v1
kind: Service
metadata:
name: zk-cs
labels:
app: zk
namespace: kafka
spec:
selector:
app: zk
type: NodePort
ports:
- name: client
port: 2181
nodePort: 31811
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: zk-pdb
namespace: kafka
spec:
selector:
matchLabels:
app: zk
maxUnavailable: 1
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: zk
namespace: kafka
spec:
serviceName: "zk-hs"
replicas: 3 # by default is 1
selector:
matchLabels:
app: zk # has to match .spec.template.metadata.labels
updateStrategy:
type: RollingUpdate
podManagementPolicy: Parallel
template:
metadata:
labels:
app: zk # has to match .spec.selector.matchLabels
spec:
containers:
- name: zk
imagePullPolicy: Always
image: leolee32/kubernetes-library:kubernetes-zookeeper1.0-3.4.10
resources:
requests:
memory: "2G"
cpu: "1"
ports:
- containerPort: 2181
name: client
- containerPort: 2888
name: server
- containerPort: 3888
name: leader-election
command:
- sh
- -c
- "start-zookeeper \
--servers=3 \
--data_dir=/var/lib/zookeeper/data \
--data_log_dir=/var/lib/zookeeper/data/log \
--conf_dir=/opt/zookeeper/conf \
--client_port=2181 \
--election_port=3888 \
--server_port=2888 \
--tick_time=2000 \
--init_limit=10 \
--sync_limit=5 \
--heap=4G \
--max_client_cnxns=60 \
--snap_retain_count=3 \
--purge_interval=12 \
--max_session_timeout=40000 \
--min_session_timeout=4000 \
--log_level=INFO"
readinessProbe:
exec:
command:
- sh
- -c
- "zookeeper-ready 2181"
initialDelaySeconds: 10
timeoutSeconds: 5
livenessProbe:
exec:
command:
- sh
- -c
- "zookeeper-ready 2181"
initialDelaySeconds: 10
timeoutSeconds: 5
volumeMounts:
- name: datadir
mountPath: /var/lib/zookeeper
volumeClaimTemplates:
- metadata:
name: datadir
annotations:
volume.beta.kubernetes.io/storage-class: managed-nfs-storage
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 1Gi
安裝kafka
采用helm的方式安裝kafka最簡單,真的不要太方便了,只需要簡單的配置即可
借鑒以下文章的方式,鏡像采用的是bitnami/kafka
https://artifacthub.io/packages/helm/bitnami/kafka
https://www.shuzhiduo.com/A/n2d9qvb0dD/
下載kafka的chart：bitnami/kafka
下載成功后當前目錄有個壓縮包：kafka-0.20.8.tgz孟岛，解壓：tar -zxvf kafka-0.20.8.tgz
進入解壓后的kafka目錄走诞，編輯values.yaml文件，下面是具體的修改點：
校驗安裝是否成功
- kubectl exec -ti zk-1 -n kafka -- zkCli.sh -server zk-cs:2181
出現(xiàn)
Welcome to ZooKeeper!
JLine support is enabled
即安裝成功
- kubectl exec -ti kafka-0 -n kafka -- /bin/bash
登錄到pod里面
執(zhí)行獲取topic : kafka-topics.sh --list --zookeeper zk-cs:2181
kafka-topics.sh --list --zookeeper 10.45.2.13:31811
創(chuàng)建topic : kafka-topics.sh --create --topic testtopic --zookeeper zk-cs:2181 --partitions 1 --replication-factor 1
執(zhí)行獲取topic : kafka-topics.sh --list --zookeeper zk-cs:2181
往topic中生產(chǎn)數(shù)據(jù): kafka-console-producer.sh --broker-list nodeport:31092 --topic wisecotest
kafka-2.kafka-cs.kafka.svc.cluster.local:31092
// 對外暴露的地址端口10.233.137.116:9092 32092
往topic中生產(chǎn)數(shù)據(jù): kafka-console-producer.sh --broker-list kafka-cs:9092 --topic wisecotest
從topic中獲取數(shù)據(jù): kafka-console-consumer.sh --bootstrap-server kafka-2.kafka-cs.kafka.svc.cluster.local:31092 --topic wisecotest --from-beginning
安裝問題解答
- 出現(xiàn): 0/3 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 2 node(s) didn't match pod anti-affinity rules.
原因是設(shè)定了親和性,導(dǎo)致每個節(jié)點上只能安裝一個類型,master節(jié)點又不允許安裝其他的節(jié)點,導(dǎo)致失敗
- 注意事項,采用stateful set的方式安裝kafka在集群內(nèi)部是可以訪問的,但是集群外是不能訪問
stateful set 安裝kafka后
通過service對外暴露的ip:port能訪問到zookeeper,zookeeper返回的是注冊的集群內(nèi)部的地址(這里應(yīng)該可以修改),導(dǎo)致當前鏈接不上返回的地址
最后采用了helm的安裝方式,等于是對外暴露了三個service,每個service對應(yīng)pod
然后在zookeeper中看到的注冊地址中有個external的存在
