1.1.1.5 內(nèi)網(wǎng)服務(wù)器 k8s master(修改默認(rèn)告警系統(tǒng))
1.1.1.1 外部機(jī)器(搭建郵件轉(zhuǎn)發(fā)nginx)

1. nginx搭建

cat nginx.conf

stream{

    server{
        listen 4650;
        proxy_pass exmail.xxx.com:587;

          }
      }

2. 告警系統(tǒng)

此配置只適用企業(yè)郵箱辑舷，qq網(wǎng)易郵箱得自行網(wǎng)上尋找配置

2.1 alertmanager.yaml

mkdir -p /home/kubernetes/k8s/app/alertmanager_alert
cd /home/kubernetes/k8s/app/alertmanager_alert
touch alertmanager.yaml

#cat alertmanager.yaml
global:
  resolve_timeout: 5m
  smtp_smarthost: '1.1.1.1:4650'
  smtp_from: '123@xxx.com'
  smtp_auth_username: '123@xxx.com'
  smtp_auth_password: 'password'
  smtp_hello: 'xxx.com'
  smtp_require_tls: true

templates:
  - "./*.tmpl"

receivers:
- name: "email"
  email_configs:
  - to: '456@xxx.com'
    html: '{{ template "email.html" . }}'
    send_resolved: true
    tls_config:
      insecure_skip_verify: true

route:
  group_by:
  - job
  group_interval: 30s
  group_wait: 5s
  receiver: "email"
  repeat_interval: 12h

2.2 新增告警模板

cd /home/kubernetes/k8s/app/alertmanager_alert
mkdir template
touch template/email.tmpl

#cat touch template/email.tmpl
#抄的 http://www.manongjc.com/detail/26-ysorykovlufnizt.html(http://www.manongjc.com/detail/26-ysorykovlufnizt.html)


{{ define "email.html" }}
{{- if gt (len .Alerts.Firing) 0 -}}
{{- range $index, $alert := .Alerts -}}

========= <span style=color:red;font-size:36px;font-weight:bold;> 監(jiān)控告警 </span>=========<br>

<span style=font-size:20px;font-weight:bold;> 告警程序:</span>     Alertmanager <br>
<span style=font-size:20px;font-weight:bold;> 告警類型:</span>    {{ $alert.Labels.alertname }} <br>
<span style=font-size:20px;font-weight:bold;> 告警級(jí)別:</span>    {{ $alert.Labels.severity }} 級(jí) <br>
<span style=font-size:20px;font-weight:bold;> 告警狀態(tài):</span>    {{ .Status }} <br>
<span style=font-size:20px;font-weight:bold;> 故障主機(jī):</span>    {{ $alert.Labels.instance }} {{ $alert.Labels.device }} <br>
<span style=font-size:20px;font-weight:bold;> 告警主題:</span>    {{ .Annotations.summary }} <br>
<span style=font-size:20px;font-weight:bold;> 告警詳情:</span>    {{ $alert.Annotations.message }}{{ $alert.Annotations.description}} <br>
<span style=font-size:20px;font-weight:bold;> 主機(jī)標(biāo)簽:</span>    {{ range .Labels.SortedPairs  }} <br> [{{ .Name }}: {{ .Value  | html }} ]{{ end }}<br>
<span style=font-size:20px;font-weight:bold;> 故障時(shí)間:</span>    {{ ($alert.StartsAt.Add 28800e9).Format "2006-01-02 15:04:05" }}<br>

========= = end =  =========<br>
<br>
<br>
<br>
<br>

<div>
    <div style=margin:40px>
        <p style=font-size:20px>xxxx</p>
        <p style=color:red;font-size:14px>
        (這是一封自動(dòng)發(fā)送的郵件，請(qǐng)勿回復(fù)。)
        </p>
    </div>


    <div align=right style="margin:40px;border-top:solid 1px gray" id=bottomTime>
        <p style=margin-right:20px>
            xxx
        </p>
        <label style=margin-right:20px>
            {{ ($alert.StartsAt.Add 28800e9).Format "2006-01-02 " }}<br>
        </label>
    </div>
</div>


{{- end }}
{{- end }}
{{- if gt (len .Alerts.Resolved) 0 -}}
{{- range $index, $alert := .Alerts -}}
========= <span style=color:#00FF00;font-size:24px;font-weight:bold;> 告警恢復(fù) </span>=========<br>
<span style=font-size:20px;font-weight:bold;> 告警程序:</span>    Alertmanager <br>
<span style=font-size:20px;font-weight:bold;> 告警主題:</span>    {{ $alert.Annotations.summary }}<br>
<span style=font-size:20px;font-weight:bold;> 告警主機(jī):</span>    {{ .Labels.instance }} <br>
<span style=font-size:20px;font-weight:bold;> 告警類型:</span>    {{ .Labels.alertname }}<br>
<span style=font-size:20px;font-weight:bold;> 告警級(jí)別:</span>    {{ $alert.Labels.severity }} 級(jí) <br>
<span style=font-size:20px;font-weight:bold;> 告警狀態(tài):</span>    {{   .Status }}<br>
<span style=font-size:20px;font-weight:bold;> 告警詳情:</span>    {{ $alert.Annotations.message }}{{ $alert.Annotations.description}}<br>
<span style=font-size:20px;font-weight:bold;> 故障時(shí)間:</span>    {{ ($alert.StartsAt.Add 28800e9).Format "2006-01-02 15:04:05" }}<br>
<span style=font-size:20px;font-weight:bold;> 恢復(fù)時(shí)間:</span>    {{ ($alert.EndsAt.Add 28800e9).Format "2006-01-02 15:04:05" }}<br>
========= = end =  =========
<br>
<br>
<br>
<br>

<div>
    <div style=margin:40px>
        <p style=font-size:20px>xxxx</p>
        <p style=color:red;font-size:14px>
        (這是一封自動(dòng)發(fā)送的郵件，請(qǐng)勿回復(fù)迹冤。)
        </p>
    </div>


    <div align=right style="margin:40px;border-top:solid 1px gray" id=bottomTime>
        <p style=margin-right:20px>
            xxx
        </p>
        <label style=margin-right:20px>
            {{ ($alert.StartsAt.Add 28800e9).Format "2006-01-02 " }}<br>
        </label>
    </div>
</div>
{{- end }}
{{- end }}
{{- end }}

2.3 刪除默認(rèn)配置膜蠢，換上新配置

cd /home/kubernetes/k8s/app/alertmanager_alert
kubectl delete -n monitoring secrets alertmanager-main
kubectl  create   secret generic  alertmanager-main --from-file=alertmanager.yaml --from-file=./template/email.tmpl -n monitoring