為了滿足一些業(yè)務(wù)上的特定場(chǎng)景，這時(shí)就需要定制化開(kāi)發(fā)一些功能，在我們的業(yè)務(wù)代碼里加入少許代碼精堕，就能實(shí)現(xiàn)和我們自身業(yè)務(wù)相關(guān)的一些監(jiān)控功能，比如追蹤日志里加入一些特殊的信息锌唾、對(duì)訂單數(shù)量的變化進(jìn)行監(jiān)控锄码、對(duì)用戶數(shù)量變化進(jìn)行監(jiān)控等。
一晌涕、Trace
自定義一個(gè)跟蹤方法很簡(jiǎn)單滋捶，只需在要跟蹤的方法上添加@Trace注解即可，當(dāng)然它也需要 activations/apm-toolkit-trace-activation-8.6.0.jar插件的支持
- 在springboot的pom.xml中引入
<dependency>
<groupId>org.apache.skywalking</groupId>
<artifactId>apm-toolkit-trace</artifactId>
<version>${skywalking.version}</version>
</dependency>
- 定義一個(gè)Controller余黎，添加下面請(qǐng)求
@GetMapping("tractAnnotation")
public User traceAnnotation(@RequestParam("name") String name) {
log.info("參數(shù):[{}]", name);
User user = trace(name);
ActiveSpan.tag("user-tag", user.toString());
log.info("tractId:[{}]", TraceContext.traceId());
return user;
}
@Trace(operationName = "myTrace")
@Tags({
@Tag(key = "參數(shù)", value = "arg[0]"),
@Tag(key = "返回值", value = "returnedObj.name")
})
private User trace(String name) {
User user = new User();
user.setName(name);
return user;
}
- 請(qǐng)求 http://localhost:9000/tractAnnotation?name=xxx 后重窟，在UI的追蹤面板中查看記錄。
二惧财、Meter
skywalking 從8.0開(kāi)始引入了指標(biāo)監(jiān)控巡扇，同時(shí)也可以支持 micrometer，這樣就可以在自己的業(yè)務(wù)系統(tǒng)中自定義一些指標(biāo)垮衷，比如訪問(wèn)總數(shù)厅翔，訂單總數(shù)等，增強(qiáng)了擴(kuò)展性搀突。下面我們以一個(gè)實(shí)例來(lái)演示這個(gè)功能刀闷。
修改OAP配置
- 首先在服務(wù)器端增加一個(gè)自定義指標(biāo)文件 spring-meter.yaml，并且要遵從MAL語(yǔ)法。
!!! 將spring-meter.yaml文件放到config/meter-analyzer-config下
expSuffix: instance(['service'], ['instance'])
metricPrefix: meter_order
metricsRules:
- name: new_increase_count
exp: new_increase_count.increase("PT1M")
- 修改config/application.yml 第280行左右找到 meterAnalyzerActiveFiles甸昏，配置為上面文件名spring-meter.yaml（去掉后綴）
agent-analyzer:
selector: ${SW_AGENT_ANALYZER:default}
default:
....
meterAnalyzerActiveFiles: ${SW_METER_ANALYZER_ACTIVE_FILES:spring-meter}
如果存儲(chǔ)用的是mysql顽分，服務(wù)啟動(dòng)后，會(huì)生成一張 meter_order_new_increase_count 的表施蜜，說(shuō)明服務(wù)端配置成功卒蘸。
應(yīng)用端開(kāi)發(fā)
在springboot應(yīng)用中引入meter依賴
<dependency>
<groupId>org.apache.skywalking</groupId>
<artifactId>apm-toolkit-meter</artifactId>
<version>${project.version}</version>
</dependency>
編寫(xiě)一個(gè)Controller，多次請(qǐng)求meter 來(lái)模擬訂單數(shù)量變化翻默，并查看meter_order_new_increase_count 表是否有新增記錄
@GetMapping("meter")
public void meter() {
Counter counter = MeterFactory.counter(new MeterId ("new_increase_count",MeterId.MeterType.COUNTER)).tag("Order Count", "100").mode(Counter.Mode.INCREMENT).build();
counter.increment(Math.random()*10);
log.info("{}:{}", counter.getName(),counter.get());
}
注意!!!：?jiǎn)?dòng)springboot時(shí)別忘了在VM Option中添加javaagent參數(shù)
-javaagent:skywalking-agent\skywalking-agent.jar -Dskywalking.agent.service_name=myapp -Dskywalking.agent.instance_name=myapp -Dskywalking.collector.backend_service=localhost:11800
關(guān)于 micrometer 的使用大概 這個(gè)樣子缸沃，這個(gè)我沒(méi)有實(shí)踐，感興趣的可以測(cè)試下修械。
<dependency>
<groupId>org.apache.skywalking</groupId>
<artifactId>apm-toolkit-micrometer-registry</artifactId>
<version>${skywalking.version}</version>
</dependency>
@GetMapping("micrometer")
public void micrometer() {
// If you has some counter want to rate by agent side
SkywalkingConfig config = new SkywalkingConfig(Arrays.asList("test_rate_counter"));
SkywalkingMeterRegistry registry = new SkywalkingMeterRegistry(config);
io.micrometer.core.instrument.Counter counter = registry.counter("order.count.total","china","beijing");
counter.increment();
log.info("Midrometer-{}:{}",registry.getMeters(),counter.measure());
}
UI圖表
編輯UI和泌，添加一個(gè)item，指標(biāo)輸入meter_order_new_increase_count（就是上面在OAP服務(wù)端定義的那個(gè)指標(biāo)）祠肥，選擇read all values in..
注意： UI中添加的指標(biāo)必須是在OAP服務(wù)端提前編寫(xiě)好的，否則這里無(wú)法添加
三梯皿、Log
skywalking可以將應(yīng)用日志收集到oap服務(wù)端方便在調(diào)用鏈中查看某個(gè)請(qǐng)求的相關(guān)日志仇箱。
- 在springboot應(yīng)用中添加logback配置：logback-spring.xml
<?xml version="1.0" encoding="UTF-8"?>
<!--
~ Licensed to the Apache Software Foundation (ASF) under one or more
~ contributor license agreements. See the NOTICE file distributed with
~ this work for additional information regarding copyright ownership.
~ The ASF licenses this file to You under the Apache License, Version 2.0
~ (the "License"); you may not use this file except in compliance with
~ the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
-->
<configuration scan="true" scanPeriod=" 5 seconds">
<appender name="stdout" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
<layout class="org.apache.skywalking.apm.toolkit.log.logback.v1.x.mdc.TraceIdMDCPatternLogbackLayout">
<Pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%X{tid}] [%thread] %-5level %logger{36} -%msg%n</Pattern>
</layout>
</encoder>
</appender>
<appender name="grpc-log" class="org.apache.skywalking.apm.toolkit.log.logback.v1.x.log.GRPCLogClientAppender">
<encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
<layout class="org.apache.skywalking.apm.toolkit.log.logback.v1.x.mdc.TraceIdMDCPatternLogbackLayout">
<Pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%X{tid}] [%thread] %-5level %logger{36} -%msg%n</Pattern>
</layout>
</encoder>
</appender>
<appender name="fileAppender" class="ch.qos.logback.core.FileAppender">
<file>d:/temp/e2e-service-provider.log</file>
<encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
<layout class="org.apache.skywalking.apm.toolkit.log.logback.v1.x.TraceIdPatternLogbackLayout">
<Pattern>[%sw_ctx] [%level] %d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %logger:%line - %msg%n</Pattern>
</layout>
</encoder>
</appender>
<root level="INFO">
<appender-ref ref="grpc-log"/>
<appender-ref ref="stdout"/>
</root>
<logger name="fileLogger" level="INFO">
<appender-ref ref="fileAppender"/>
</logger>
</configuration>
- 修改agent.config ，添加如下配置：
plugin.toolkit.log.grpc.reporter.server_host=${SW_GRPC_LOG_SERVER_HOST:192.168.x.x}
plugin.toolkit.log.grpc.reporter.server_port=${SW_GRPC_LOG_SERVER_PORT:11800}
plugin.toolkit.log.grpc.reporter.max_message_size=${SW_GRPC_LOG_MAX_MESSAGE_SIZE:10485760}
plugin.toolkit.log.grpc.reporter.upstream_timeout=${SW_GRPC_LOG_GRPC_UPSTREAM_TIMEOUT:30}
- 當(dāng)訪問(wèn)應(yīng)用時(shí)东羹，會(huì)在Skywalking中產(chǎn)生日志
四剂桥、node-exporter
Skywalking 也支持 Prometheus node-exporter導(dǎo)入指標(biāo)，從而可以監(jiān)控操作系統(tǒng)級(jí)別的指標(biāo)属提。在Skywalking中類似這類的指標(biāo)是通過(guò)OpenTelemetry Collector來(lái)收集权逗，通過(guò) OpenTelemetry receiver 來(lái)接收。因此要支持 node-exporter 需要分為三個(gè)步驟：
在要監(jiān)控的操作系統(tǒng)上啟動(dòng)一個(gè) node-exporter
安裝并啟動(dòng)一個(gè) OpenTelemetry Collector .
在SkyWalking中配置 OpenTelemetry receiver.
- 在vm01冤议、vm02上斟薇，分別啟動(dòng) node_exporter
$ tar -xzvf node_exporter-1.0.1.linux-amd64.tar.gz && cd node_exporter-1.0.1.linux-amd64
$ nohup ./node_exporter &
-
安裝OpenTelemetry Collector
使用docker-compose方式啟動(dòng)一個(gè)otel-collector
version: "2"
services:
# Collector
otel-collector:
# Specify the image to start the container from
image: otel/opentelemetry-collector:0.19.0
# Set the otel-collector configfile
command: ["--config=/etc/otel-collector-config.yaml"]
# Mapping the configfile to host directory
volumes:
- ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
ports:
- "13133:13133" # health_check extension
- "55678:55678" # OpenCensus receiver
修改 otel-collector-config.yaml配置，vm01恕酸、vm02為啟動(dòng)了node_exporter的機(jī)器IP堪滨，將oap替換成OAP服務(wù)地址。
注意：logging 級(jí)別不要設(shè)成debug蕊温，否則磁盤會(huì)被日志爆滿
receivers:
prometheus:
config:
scrape_configs:
- job_name: 'otel-collector'
scrape_interval: 1s
static_configs:
- targets: ['vm01:9100']
- targets: ['vm02:9100']
processors:
batch:
exporters:
opencensus:
endpoint: "oap:11800" # The OAP Server address
insecure: true
# Exports data to the console
logging:
# 注意這里的日志級(jí)別不要設(shè)的太高袱箱，否則會(huì)磁盤爆滿
logLevel: error
service:
pipelines:
metrics:
receivers: [prometheus]
processors: [batch]
exporters: [opencensus,logging]
如果采用k8s來(lái)部署opentelemetry-collector，請(qǐng)參考下面
# otel-collector-k8s.yaml
---
apiVersion: v1
kind: ConfigMap
metadata:
name: otel-agent-conf
labels:
app: opentelemetry
component: otel-agent-conf
data:
otel-agent-config: |
receivers:
prometheus:
config:
scrape_configs:
- job_name: 'otel-collector'
scrape_interval: 1s
static_configs:
- targets: ['vm-1:9100']
- targets: ['vm-2:9100']
processors:
batch:
exporters:
opencensus:
endpoint: "oap.skywalking.svc.cluster.local:11800" # The OAP Server address
insecure: true
# Exports data to the console
#logging:
# logLevel: debug
service:
pipelines:
metrics:
receivers: [prometheus]
processors: [batch]
exporters: [opencensus]
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: otel-agent
labels:
app: opentelemetry
component: otel-agent
spec:
serviceName: otel-agent
selector:
matchLabels:
app: opentelemetry
component: otel-agent
template:
metadata:
labels:
app: opentelemetry
component: otel-agent
spec:
containers:
- command:
- "/otelcol"
- "--config=/conf/otel-agent-config.yaml"
# Memory Ballast size should be max 1/3 to 1/2 of memory.
- "--mem-ballast-size-mib=165"
image: otel/opentelemetry-collector:0.19.0
name: otel-agent
resources:
limits:
cpu: 500m
memory: 500Mi
requests:
cpu: 100m
memory: 100Mi
ports:
- containerPort: 55679 # ZPages endpoint.
- containerPort: 4317 # Default OpenTelemetry receiver port.
- containerPort: 8888 # Metrics.
volumeMounts:
- name: otel-agent-config-vol
mountPath: /conf
# 這里不能開(kāi)啟探針檢查义矛，否則容器會(huì)自動(dòng)退出
#livenessProbe:
# httpGet:
# path: /
# port: 13133 # Health Check extension default port.
#readinessProbe:
# httpGet:
# path: /
# port: 13133 # Health Check extension default port.
volumes:
- configMap:
name: otel-agent-conf
items:
- key: otel-agent-config
path: otel-agent-config.yaml
name: otel-agent-config-vol
-
修改OAP的配置文件config/application.yml发笔，激活vm規(guī)則，這些規(guī)則配置存放在otel-oc-rules目錄下凉翻，如果配置多個(gè)規(guī)則了讨，以逗號(hào)分隔。如果要定制指標(biāo)就修改 vm.yaml文件。
按照官方的文檔一步步操作完量蕊，發(fā)現(xiàn)UI上根本不顯示铺罢。這里就要注意了，默認(rèn)receiver-otel的selector是
-残炮，因此receiver-otel插件根本不會(huì)加載的韭赘，所以需要將selector配置成default。
receiver-otel:
selector: ${SW_OTEL_RECEIVER:default}
default:
enabledHandlers: ${SW_OTEL_RECEIVER_ENABLED_HANDLERS:"oc"}
enabledOcRules: ${SW_OTEL_RECEIVER_ENABLED_OC_RULES:"vm,oap"}
- 查看UI中VM已經(jīng)抓取到機(jī)器的指標(biāo)势就，但貌似和真實(shí)值有些出入泉瞻，暫先不管了
