1.安裝chi-operator
ClickHouse Operator creates, configures and manages ClickHouse clusters running on Kubernetes.
kubectl apply -f https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/deploy/operator/clickhouse-operator-install.yaml
1.1 查看chi-operator
kubectl -n kube-system get pod | grep clickhouse-operator
如果pod的狀態(tài)是running的虎眨，說明chi-operator部署成功透且≡奥妫可通過下面的命令查看其日志奏司。
kubectl -n kube-system logs -f clickhouse-operator-5b45484748-kpg6t clickhouse-operator
2. 部署集群
2.1 部署架構(gòu)
按照下圖，將要部署一個2shard，2replica的一個集群，即需要四個pod抚太。每個pod的存儲使用loca pv的方式。也就是需要四臺機器昔案。
image.png
2.2 部署集群
下面的代碼包含兩個部分
- local pv的部署yaml,注意此處選定了四臺機器尿贫。
- chi的部署yaml。
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: clickhouse-local-volume
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-clickhouse-0
spec:
capacity:
storage: 100Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: clickhouse-local-volume
hostPath:
path: /mnt/data/clickhouse
type: DirectoryOrCreate
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- "clickhouse1"
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-clickhouse-1
spec:
capacity:
storage: 100Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: clickhouse-local-volume
hostPath:
path: /mnt/data/clickhouse
type: DirectoryOrCreate
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- "clickhouse2"
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-clickhouse-2
spec:
capacity:
storage: 100Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: clickhouse-local-volume
hostPath:
path: /mnt/data/clickhouse
type: DirectoryOrCreate
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- "clickhouse3"
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-clickhouse-3
spec:
capacity:
storage: 100Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: clickhouse-local-volume
hostPath:
path: /mnt/data/clickhouse
type: DirectoryOrCreate
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- "clickhouse4"
---
apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
name: "aibee"
spec:
defaults:
templates:
serviceTemplate: service-template
podTemplate: pod-template
dataVolumeClaimTemplate: volume-claim
configuration:
settings:
compression/case/method: zstd
disable_internal_dns_cache: 1
timezone: Asia/Shanghai
zookeeper:
nodes:
- host: zk-svc
port: 2181
session_timeout_ms: 30000
operation_timeout_ms: 10000
clusters:
- name: "clickhouse"
layout:
shardsCount: 2
replicasCount: 2
templates:
serviceTemplates:
- name: service-template
spec:
ports:
- name: http
port: 8123
- name: tcp
port: 9000
type: LoadBalancer
podTemplates:
- name: pod-template
spec:
containers:
- name: clickhouse
imagePullPolicy: Always
image: yandex/clickhouse-server:latest
volumeMounts:
# 掛載數(shù)據(jù)文件路徑
- name: volume-claim
mountPath: /var/lib/clickhouse
# 掛載數(shù)據(jù)文件路徑
- name: volume-claim
mountPath: /var/log/clickhouse-server
resources:
# 配置cpu和內(nèi)存大小
limits:
memory: "1Gi"
cpu: "1"
requests:
memory: "1Gi"
cpu: "1"
volumeClaimTemplates:
- name: volume-claim
reclaimPolicy: Retain
spec:
storageClassName: "clickhouse-local-volume"
accessModes:
- ReadWriteOnce
resources:
# pv的存儲大小
requests:
storage: 100Gi
注意: volumeClaimTemplates的reclaimPolicy必須是Retain踏揣，這樣即使刪除集群庆亡，數(shù)據(jù)會保留下來。否則在刪除集群的時候會刪除所有以"Replica*"開頭的table捞稿。我被這個坑了很久又谋。源碼如下：
// hostGetDropTables returns set of 'DROP TABLE ...' SQLs
func (s *Schemer) hostGetDropTables(host *chop.ChiHost) ([]string, []string, error) {
// There isn't a separate query for deleting views. To delete a view, use DROP TABLE
// See https://clickhouse.yandex/docs/en/query_language/create/
sql := heredoc.Doc(`
SELECT
distinct name,
concat('DROP TABLE IF EXISTS "', database, '"."', name, '"') AS drop_db_query
FROM system.tables
WHERE engine like 'Replicated%'`,
)
names, sqlStatements, _ := s.getObjectListFromClickHouse([]string{CreatePodFQDN(host)}, sql)
return names, sqlStatements, nil
部署成功后的pod的分布情況：
chi-aibee-clickhouse-0-0-0 1/1 Running 0 20m 192.168.35.196 clickhouse3 <none> <none>
chi-aibee-clickhouse-0-1-0 1/1 Running 0 20m 192.168.132.103 clickhouse2 <none> <none>
chi-aibee-clickhouse-1-0-0 1/1 Running 0 20m 192.168.13.41 clickhouse4 <none> <none>
chi-aibee-clickhouse-1-1-0 1/1 Running 0 19m 192.168.133.164 clickhouse1 <none> <none>
2.3 查看svc的地址
kubectl get svc clickhouse-aibee
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
clickhouse-aibee LoadBalancer 10.100.185.34 <pending> 8123:30745/TCP,9000:32346/TCP 22m
2.4 連接集群
使用上面的svc的ClusterIP，默認(rèn)賬戶密碼：clickhouse_operator/clickhouse_operator_password
clickhouse-client -h 10.100.185.34 -u clickhouse_operator --password clickhouse_operator_password
更多自定義的情況請參考這個地址：https://github.com/Altinity/clickhouse-operator/blob/master/docs/custom_resource_explained.md
2.4 內(nèi)置的宏
Operator provides set of macros, which are:
-
{installation}-- ClickHouse Installation name -
{cluster}-- primary cluster name -
{replica}-- replica name in the cluster, maps to pod service name -
{shard}-- shard id
ClickHouse also supports internal macros {database} and {table} that maps to current database and table respectively.
下面的代碼展示的是當(dāng)前集群自動創(chuàng)建的macros娱局，我們可以在創(chuàng)建表的時候使用彰亥。
<yandex>
<macros>
<installation>aibee</installation>
<all-sharded-shard>0</all-sharded-shard>
<cluster>clickhouse</cluster>
<shard>0</shard>
<replica>chi-aibee-clickhouse-0-0</replica>
</macros>
</yandex>
3 創(chuàng)建表
CREATE TABLE events_local on cluster '{cluster}' (
event_date Date,
event_type Int32,
article_id Int32,
title String
) engine=ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/{database}/{table}', '{replica}', event_date, (event_type, article_id), 8192);
CREATE TABLE events on cluster '{cluster}' AS events_local
ENGINE = Distributed('{cluster}', default, events_local, rand());
3.1 插入數(shù)據(jù)
INSERT INTO events SELECT today(), rand()%3, number, 'my title' FROM numbers(100);
3.2 查看數(shù)據(jù)
SELECT count() FROM events;
SELECT count() FROM events_local;
4 集群監(jiān)控
chi-operator已經(jīng)集成了metrics-operator。下面的命令查看監(jiān)控地址
kubectl get service clickhouse-operator-metrics -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
clickhouse-operator-metrics ClusterIP 10.102.111.74 <none> 8888/TCP 48d
Prometheus可以使用這個地址抓取metrics衰齐。
http://<service/clickhouse-operator-metrics>:8888/metrics
Grafana Dashbord
https://github.com/Altinity/clickhouse-operator/blob/master/grafana-dashboard/Altinity_ClickHouse_Operator_dashboard.json
更多請參考 https://github.com/Altinity/clickhouse-operator/blob/master/docs/prometheus_setup.md
