??Prometheus是一個集數(shù)據(jù)收集存儲靡狞、數(shù)據(jù)查詢和數(shù)據(jù)圖表顯示于一身的開源監(jiān)控組件褪那。本文主要講解如何搭建Prometheus半醉，并使用它監(jiān)控Kubernetes集群疚俱。
準備工作
- Kubernetes集群，如果你還沒有搭建好Kubernetes集群缩多，可以參考這篇文章－Kubernetes-離線部署Kubernetes 1.9.0
- Prometheus的docker鏡像呆奕，截至筆者寫這篇文章之時养晋，最新的版本是2.3.2，如果你的Kubernetes集群不能訪問Docker Hub梁钾，請將鏡像pull下來放到自己的私有鏡像倉庫绳泉。
image
部署Prometheus
-
創(chuàng)建命名空間
新建一個yaml文件命名為monitor-namespace.yaml，寫入如下內(nèi)容
apiVersion: v1
kind: Namespace
metadata:
name: monitoring
執(zhí)行如下命令創(chuàng)建monitoring命名空間
kubectl create -f monitor-namespace.yaml
create namespace
-
創(chuàng)建ClusterRole
??你需要對上面創(chuàng)建的命名空間分配集群的讀取權(quán)限姆泻，以便Prometheus可以通過Kubernetes的API獲取集群的資源指標零酪。
新建一個yaml文件命名為cluster-role.yaml，寫入如下內(nèi)容：
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: prometheus
rules:
- apiGroups: [""]
resources:
- nodes
- nodes/proxy
- services
- endpoints
- pods
verbs: ["get", "list", "watch"]
- apiGroups:
- extensions
resources:
- ingresses
verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: prometheus
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: prometheus
subjects:
- kind: ServiceAccount
name: default
namespace: monitoring
執(zhí)行如下命令創(chuàng)建
kubectl create -f cluster-role.yaml
create role
-
創(chuàng)建Config Map
??我們需要創(chuàng)建一個Config Map保存后面創(chuàng)建Prometheus容器用到的一些配置拇勃，這些配置包含了從Kubernetes集群中動態(tài)發(fā)現(xiàn)pods和運行中的服務(wù)四苇。
新建一個yaml文件命名為config-map.yaml，寫入如下內(nèi)容：
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-server-conf
labels:
name: prometheus-server-conf
namespace: monitoring
data:
prometheus.yml: |-
global:
scrape_interval: 5s
evaluation_interval: 5s
scrape_configs:
- job_name: 'kubernetes-apiservers'
kubernetes_sd_configs:
- role: endpoints
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
relabel_configs:
- source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
action: keep
regex: default;kubernetes;https
- job_name: 'kubernetes-nodes'
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- target_label: __address__
replacement: kubernetes.default.svc:443
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}/proxy/metrics
- job_name: 'kubernetes-pods'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
target_label: __address__
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: kubernetes_pod_name
- job_name: 'kubernetes-cadvisor'
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- target_label: __address__
replacement: kubernetes.default.svc:443
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
- job_name: 'kubernetes-service-endpoints'
kubernetes_sd_configs:
- role: endpoints
relabel_configs:
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
action: replace
target_label: __scheme__
regex: (https?)
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
action: replace
target_label: __address__
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
- action: labelmap
regex: __meta_kubernetes_service_label_(.+)
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace
- source_labels: [__meta_kubernetes_service_name]
action: replace
target_label: kubernetes_name
執(zhí)行如下命令進行創(chuàng)建
kubectl create -f config-map.yaml -n monitoring
create config map
-
創(chuàng)建Deployment模式的Prometheus
新建一個yaml文件命名為prometheus-deployment.yaml方咆，寫入如下內(nèi)容月腋，鏡像那里注意一下，我寫的是我私有倉庫的路徑瓣赂，如果kubernetes集群能訪問Docker Hub請修改為【prom/prometheus:v2.3.2】：
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: prometheus-deployment
namespace: monitoring
spec:
replicas: 1
template:
metadata:
labels:
app: prometheus-server
spec:
containers:
- name: prometheus
image: registry.docker.uih/library/prometheus:2.3.2
args:
- "--config.file=/etc/prometheus/prometheus.yml"
- "--storage.tsdb.path=/prometheus/"
ports:
- containerPort: 9090
volumeMounts:
- name: prometheus-config-volume
mountPath: /etc/prometheus/
- name: prometheus-storage-volume
mountPath: /prometheus/
volumes:
- name: prometheus-config-volume
configMap:
defaultMode: 420
name: prometheus-server-conf
- name: prometheus-storage-volume
emptyDir: {}
使用如下命令部署
kubectl create -f prometheus-deployment.yaml --namespace=monitoring
部署完成后通過dashboard能夠看到如下的界面：
Prometheus deployment
連接Prometheus
這里有兩種方式
- 通過kubectl命令進行端口代理
- 針對Prometheus的POD暴露一個服務(wù)榆骚，推薦此種方式
首先新建一個yaml文件命名為prometheus-service.yaml，寫入如下內(nèi)容：
apiVersion: v1
kind: Service
metadata:
name: prometheus-service
spec:
selector:
app: prometheus-server
type: NodePort
ports:
- port: 9090
targetPort: 9090
nodePort: 30909
執(zhí)行如下命令創(chuàng)建服務(wù)
kubectl create -f prometheus-service.yaml --namespace=monitoring
create service
通過dashboard可以查看到部署成功的服務(wù)
service
現(xiàn)在可以通過瀏覽器訪問【http://10.3.14.193:30909】钩述，看到如下界面
Prometheus UI
現(xiàn)在可以點擊 status –> Targets寨躁，馬上就可以看到所有Kubernetes集群上的Endpoint通過服務(wù)發(fā)現(xiàn)的方式自動連接到了Prometheus。
target
targets
我們還可以通過圖形化界面查看內(nèi)存
memory rss
