OAL 基礎(chǔ)知識(shí)

基本介紹

OAL(Observability Analysis Language) 是一門(mén)用來(lái)分析流式數(shù)據(jù)的語(yǔ)言。

因?yàn)?OAL 聚焦于度量 Service 码党、 Service Instance 和 Endpoint 的指標(biāo)德崭，所以它學(xué)習(xí)和使用起來(lái)非常簡(jiǎn)單。

OAL 基于 altlr 與 javassist 將 oal 腳本轉(zhuǎn)化為動(dòng)態(tài)生成的類(lèi)文件揖盘。

自從 6.3 版本后眉厨， OAL 引擎內(nèi)置在 OAP 服務(wù)器中，可以看做 oal-rt(OAL Runtime) 兽狭。 OAL 腳本位置 OAL 配置目錄下( /config/oal )憾股，使用者能夠更改腳本并重啟生效鹿蜀。注意： OAL 腳本仍然是一門(mén)編譯語(yǔ)言， oal-rt 動(dòng)態(tài)的生成 Java 代碼服球。

如果你配置了環(huán)境變量 SW_OAL_ENGINE_DEBUG=Y茴恰，能在工作目錄下的 oal-rt 目錄下找到生成的 Class 文件。

語(yǔ)法

// 聲明一個(gè)指標(biāo)
METRICS_NAME = from(SCOPE.(* | [FIELD][,FIELD ...])) // 從某一個(gè)SCOPE中獲取數(shù)據(jù)
[.filter(FIELD OP [INT | STRING])] // 可以過(guò)濾掉部分?jǐn)?shù)據(jù)
.FUNCTION([PARAM][, PARAM ...]) // 使用某個(gè)聚合函數(shù)將數(shù)據(jù)聚合

// 禁用一個(gè)指標(biāo)
disable(METRICS_NAME);

語(yǔ)法案例

oap-server/server-bootstrap/src/main/resources/oal/java-agent.oal

// 從ServiceInstanceJVMMemory的used獲取數(shù)據(jù)斩熊，只需要 heapStatus 為 true的數(shù)據(jù)往枣，并取long型的平均值
instance_jvm_memory_heap = from(ServiceInstanceJVMMemory.used).filter(heapStatus == true).longAvg();

org.apache.skywalking.oap.server.core.source.ServiceInstanceJVMMemory

@ScopeDeclaration(id = SERVICE_INSTANCE_JVM_MEMORY, name = "ServiceInstanceJVMMemory", catalog = SERVICE_INSTANCE_CATALOG_NAME)
@ScopeDefaultColumn.VirtualColumnDefinition(fieldName = "entityId", columnName = "entity_id", isID = true, type = String.class)
public class ServiceInstanceJVMMemory extends Source {
    @Override
    public int scope() {
        return DefaultScopeDefine.SERVICE_INSTANCE_JVM_MEMORY;
    }

    @Override
    public String getEntityId() {
        return String.valueOf(id);
    }

    @Getter @Setter
    private String id;
    @Getter @Setter @ScopeDefaultColumn.DefinedByField(columnName = "name", requireDynamicActive = true)
    private String name;
    @Getter @Setter @ScopeDefaultColumn.DefinedByField(columnName = "service_name", requireDynamicActive = true)
    private String serviceName;
    @Getter @Setter @ScopeDefaultColumn.DefinedByField(columnName = "service_id")
    private String serviceId;
    @Getter @Setter
    private boolean heapStatus;
    @Getter @Setter
    private long init;
    @Getter @Setter
    private long max;
    @Getter @Setter
    private long used;
    @Getter @Setter
    private long committed;
}

可供參考的官方文檔：Observability Analysis Language

從一個(gè)案例開(kāi)始分析 OAL 原理

缺少的類(lèi)加載信息監(jiān)控

默認(rèn)的 APM/Instance 頁(yè)面，缺少關(guān)于 JVM Class 的信息（如下圖所示）粉渠，故這次將相關(guān)信息補(bǔ)齊分冈。由這次案例來(lái)分析 OAL 的原理。

file

在 Skywalking-04：擴(kuò)展Metric監(jiān)控信息 中霸株，講到了如何在已有 Source 類(lèi)的情況下丈秩，增加一些指標(biāo)。

這次直接連 Source 類(lèi)以及 OAL 詞法語(yǔ)法關(guān)鍵字都自己定義淳衙。

可供參考的官方文檔：Source and Scope extension for new metrics

確定增加的指標(biāo)

通過(guò)Java ManagementFactory解析這篇文章，可以確定監(jiān)控指標(biāo)為“當(dāng)前加載類(lèi)的數(shù)量”饺著、“已卸載類(lèi)的數(shù)量”箫攀、“一共加載類(lèi)的數(shù)量”三個(gè)指標(biāo)

ClassLoadingMXBean classLoadingMXBean = ManagementFactory.getClassLoadingMXBean();
// 當(dāng)前加載類(lèi)的數(shù)量
int loadedClassCount = classLoadingMXBean.getLoadedClassCount();
// 已卸載類(lèi)的數(shù)量
long unloadedClassCount = classLoadingMXBean.getUnloadedClassCount();
// 一共加載類(lèi)的數(shù)量
long totalLoadedClassCount = classLoadingMXBean.getTotalLoadedClassCount();

定義 agent 與 oap server 通訊類(lèi)

在 apm-protocol/apm-network/src/main/proto/language-agent/JVMMetric.proto 協(xié)議文件中增加如下定義。

在 apm-protocol/apm-network 目錄下執(zhí)行 mvn clean package -DskipTests=true 會(huì)生成新的相關(guān) Java 類(lèi)幼衰，org.apache.skywalking.apm.network.language.agent.v3.Class 該類(lèi)就是我們?cè)诖a中實(shí)際操作的類(lèi)靴跛。

message Class {
  int64 loadedClassCount = 1;
  int64 unloadedClassCount = 3;
  int64 totalLoadedClassCount = 2;
}

message JVMMetric {
    int64 time = 1;
    CPU cpu = 2;
    repeated Memory memory = 3;
    repeated MemoryPool memoryPool = 4;
    repeated GC gc = 5;
    Thread thread = 6;
    // 在JVM指標(biāo)中添加Class的定義
    Class clazz = 7;
}

收集 agent 的信息后，將信息發(fā)送至 oap server

收集 Class 相關(guān)的指標(biāo)信息

package org.apache.skywalking.apm.agent.core.jvm.clazz;

import org.apache.skywalking.apm.network.language.agent.v3.Class;

import java.lang.management.ClassLoadingMXBean;
import java.lang.management.ManagementFactory;

public enum ClassProvider {
    /**
     * instance
     */
    INSTANCE;

    private final ClassLoadingMXBean classLoadingMXBean;

    ClassProvider() {
        this.classLoadingMXBean = ManagementFactory.getClassLoadingMXBean();
    }
    
    // 構(gòu)建class的指標(biāo)信息
    public Class getClassMetrics() {
        int loadedClassCount = classLoadingMXBean.getLoadedClassCount();
        long unloadedClassCount = classLoadingMXBean.getUnloadedClassCount();
        long totalLoadedClassCount = classLoadingMXBean.getTotalLoadedClassCount();
        return Class.newBuilder().setLoadedClassCount(loadedClassCount)
                .setUnloadedClassCount(unloadedClassCount)
                .setTotalLoadedClassCount(totalLoadedClassCount)
                .build();
    }

}

在 org.apache.skywalking.apm.agent.core.jvm.JVMService#run 方法中渡嚣，將 class 相關(guān)指標(biāo)設(shè)置到 JVM 指標(biāo)類(lèi)中

    @Override
    public void run() {
        long currentTimeMillis = System.currentTimeMillis();
        try {
            JVMMetric.Builder jvmBuilder = JVMMetric.newBuilder();
            jvmBuilder.setTime(currentTimeMillis);
            jvmBuilder.setCpu(CPUProvider.INSTANCE.getCpuMetric());
            jvmBuilder.addAllMemory(MemoryProvider.INSTANCE.getMemoryMetricList());
            jvmBuilder.addAllMemoryPool(MemoryPoolProvider.INSTANCE.getMemoryPoolMetricsList());
            jvmBuilder.addAllGc(GCProvider.INSTANCE.getGCList());
            jvmBuilder.setThread(ThreadProvider.INSTANCE.getThreadMetrics());
            // 設(shè)置class的指標(biāo)
            jvmBuilder.setClazz(ClassProvider.INSTANCE.getClassMetrics());
            // 將JVM的指標(biāo)放在阻塞隊(duì)列中
            // org.apache.skywalking.apm.agent.core.jvm.JVMMetricsSender#run方法，會(huì)將相關(guān)信息發(fā)送至oap server
            sender.offer(jvmBuilder.build());
        } catch (Exception e) {
            LOGGER.error(e, "Collect JVM info fail.");
        }
    }

創(chuàng)建 Source 類(lèi)

public class DefaultScopeDefine {
    public static final int SERVICE_INSTANCE_JVM_CLASS = 11000;

    /** Catalog of scope, the metrics processor could use this to group all generated metrics by oal rt. */
    public static final String SERVICE_INSTANCE_CATALOG_NAME = "SERVICE_INSTANCE";
}

package org.apache.skywalking.oap.server.core.source;

import lombok.Getter;
import lombok.Setter;

import static org.apache.skywalking.oap.server.core.source.DefaultScopeDefine.SERVICE_INSTANCE_CATALOG_NAME;
import static org.apache.skywalking.oap.server.core.source.DefaultScopeDefine.SERVICE_INSTANCE_JVM_CLASS;

@ScopeDeclaration(id = SERVICE_INSTANCE_JVM_CLASS, name = "ServiceInstanceJVMClass", catalog = SERVICE_INSTANCE_CATALOG_NAME)
@ScopeDefaultColumn.VirtualColumnDefinition(fieldName = "entityId", columnName = "entity_id", isID = true, type = String.class)
public class ServiceInstanceJVMClass extends Source {
    @Override
    public int scope() {
        return SERVICE_INSTANCE_JVM_CLASS;
    }

    @Override
    public String getEntityId() {
        return String.valueOf(id);
    }

    @Getter @Setter
    private String id;
    @Getter @Setter @ScopeDefaultColumn.DefinedByField(columnName = "name", requireDynamicActive = true)
    private String name;
    @Getter @Setter @ScopeDefaultColumn.DefinedByField(columnName = "service_name", requireDynamicActive = true)
    private String serviceName;
    @Getter @Setter @ScopeDefaultColumn.DefinedByField(columnName = "service_id")
    private String serviceId;
    @Getter @Setter
    private long loadedClassCount;
    @Getter @Setter
    private long unloadedClassCount;
    @Getter @Setter
    private long totalLoadedClassCount;
}

將從 agent 獲取到的信息识椰，發(fā)送至 SourceReceive

在 org.apache.skywalking.oap.server.analyzer.provider.jvm.JVMSourceDispatcher 進(jìn)行如下修改

    public void sendMetric(String service, String serviceInstance, JVMMetric metrics) {
        long minuteTimeBucket = TimeBucket.getMinuteTimeBucket(metrics.getTime());

        final String serviceId = IDManager.ServiceID.buildId(service, NodeType.Normal);
        final String serviceInstanceId = IDManager.ServiceInstanceID.buildId(serviceId, serviceInstance);

        this.sendToCpuMetricProcess(
            service, serviceId, serviceInstance, serviceInstanceId, minuteTimeBucket, metrics.getCpu());
        this.sendToMemoryMetricProcess(
            service, serviceId, serviceInstance, serviceInstanceId, minuteTimeBucket, metrics.getMemoryList());
        this.sendToMemoryPoolMetricProcess(
            service, serviceId, serviceInstance, serviceInstanceId, minuteTimeBucket, metrics.getMemoryPoolList());
        this.sendToGCMetricProcess(
            service, serviceId, serviceInstance, serviceInstanceId, minuteTimeBucket, metrics.getGcList());
        this.sendToThreadMetricProcess(
                service, serviceId, serviceInstance, serviceInstanceId, minuteTimeBucket, metrics.getThread());
        // class指標(biāo)處理
        this.sendToClassMetricProcess(
                service, serviceId, serviceInstance, serviceInstanceId, minuteTimeBucket, metrics.getClazz());
    }

    private void sendToClassMetricProcess(String service,
            String serviceId,
            String serviceInstance,
            String serviceInstanceId,
            long timeBucket,
            Class clazz) {
        // 拼裝Source對(duì)象
        ServiceInstanceJVMClass serviceInstanceJVMClass = new ServiceInstanceJVMClass();
        serviceInstanceJVMClass.setId(serviceInstanceId);
        serviceInstanceJVMClass.setName(serviceInstance);
        serviceInstanceJVMClass.setServiceId(serviceId);
        serviceInstanceJVMClass.setServiceName(service);
        serviceInstanceJVMClass.setLoadedClassCount(clazz.getLoadedClassCount());
        serviceInstanceJVMClass.setUnloadedClassCount(clazz.getUnloadedClassCount());
        serviceInstanceJVMClass.setTotalLoadedClassCount(clazz.getTotalLoadedClassCount());
        serviceInstanceJVMClass.setTimeBucket(timeBucket);
        // 將Source對(duì)象發(fā)送至SourceReceive進(jìn)行處理
        sourceReceiver.receive(serviceInstanceJVMClass);
    }

在 OAL 詞法定義和語(yǔ)法定義中加入 Source 相關(guān)信息

在 oap-server/oal-grammar/src/main/antlr4/org/apache/skywalking/oal/rt/grammar/OALLexer.g4 定義 Class 關(guān)鍵字

// Keywords

FROM: 'from';
FILTER: 'filter';
DISABLE: 'disable';
SRC_ALL: 'All';
SRC_SERVICE: 'Service';
SRC_SERVICE_INSTANCE: 'ServiceInstance';
SRC_ENDPOINT: 'Endpoint';
SRC_SERVICE_RELATION: 'ServiceRelation';
SRC_SERVICE_INSTANCE_RELATION: 'ServiceInstanceRelation';
SRC_ENDPOINT_RELATION: 'EndpointRelation';
SRC_SERVICE_INSTANCE_JVM_CPU: 'ServiceInstanceJVMCPU';
SRC_SERVICE_INSTANCE_JVM_MEMORY: 'ServiceInstanceJVMMemory';
SRC_SERVICE_INSTANCE_JVM_MEMORY_POOL: 'ServiceInstanceJVMMemoryPool';
SRC_SERVICE_INSTANCE_JVM_GC: 'ServiceInstanceJVMGC';
SRC_SERVICE_INSTANCE_JVM_THREAD: 'ServiceInstanceJVMThread';
SRC_SERVICE_INSTANCE_JVM_CLASS:'ServiceInstanceJVMClass'; // 在OAL詞法定義中添加Class的關(guān)鍵字
SRC_DATABASE_ACCESS: 'DatabaseAccess';
SRC_SERVICE_INSTANCE_CLR_CPU: 'ServiceInstanceCLRCPU';
SRC_SERVICE_INSTANCE_CLR_GC: 'ServiceInstanceCLRGC';
SRC_SERVICE_INSTANCE_CLR_THREAD: 'ServiceInstanceCLRThread';
SRC_ENVOY_INSTANCE_METRIC: 'EnvoyInstanceMetric';

在 oap-server/oal-grammar/src/main/antlr4/org/apache/skywalking/oal/rt/grammar/OALParser.g4 添加 Class 關(guān)鍵字

source
    : SRC_ALL | SRC_SERVICE | SRC_DATABASE_ACCESS | SRC_SERVICE_INSTANCE | SRC_ENDPOINT |
      SRC_SERVICE_RELATION | SRC_SERVICE_INSTANCE_RELATION | SRC_ENDPOINT_RELATION |
      SRC_SERVICE_INSTANCE_JVM_CPU | SRC_SERVICE_INSTANCE_JVM_MEMORY | SRC_SERVICE_INSTANCE_JVM_MEMORY_POOL | 
      SRC_SERVICE_INSTANCE_JVM_GC | SRC_SERVICE_INSTANCE_JVM_THREAD | SRC_SERVICE_INSTANCE_JVM_CLASS |// 在OAL語(yǔ)法定義中添加詞法定義中定義的關(guān)鍵字
      SRC_SERVICE_INSTANCE_CLR_CPU | SRC_SERVICE_INSTANCE_CLR_GC | SRC_SERVICE_INSTANCE_CLR_THREAD |
      SRC_ENVOY_INSTANCE_METRIC |
      SRC_BROWSER_APP_PERF | SRC_BROWSER_APP_PAGE_PERF | SRC_BROWSER_APP_SINGLE_VERSION_PERF |
      SRC_BROWSER_APP_TRAFFIC | SRC_BROWSER_APP_PAGE_TRAFFIC | SRC_BROWSER_APP_SINGLE_VERSION_TRAFFIC
    ;

在 oap-server/oal-grammar 目錄下執(zhí)行 mvn clean package -DskipTests=true 會(huì)生成新的相關(guān) Java 類(lèi)

定義 OAL 指標(biāo)

在 oap-server/server-bootstrap/src/main/resources/oal/java-agent.oal 中添加基于 OAL 語(yǔ)法的 Class 相關(guān)指標(biāo)定義

// 當(dāng)前加載類(lèi)的數(shù)量
instance_jvm_class_loaded_class_count = from(ServiceInstanceJVMClass.loadedClassCount).longAvg();
// 已卸載類(lèi)的數(shù)量
instance_jvm_class_unloaded_class_count = from(ServiceInstanceJVMClass.unloadedClassCount).longAvg();
// 一共加載類(lèi)的數(shù)量
instance_jvm_class_total_loaded_class_count = from(ServiceInstanceJVMClass.totalLoadedClassCount).longAvg();

配置 UI 面板

將如下界面配置導(dǎo)入 APM 面板中

{
  "name": "Instance",
  "children": [{
      "width": "3",
      "title": "Service Instance Load",
      "height": "250",
      "entityType": "ServiceInstance",
      "independentSelector": false,
      "metricType": "REGULAR_VALUE",
      "metricName": "service_instance_cpm",
      "queryMetricType": "readMetricsValues",
      "chartType": "ChartLine",
      "unit": "CPM - calls per minute"
    },
    {
      "width": 3,
      "title": "Service Instance Throughput",
      "height": "250",
      "entityType": "ServiceInstance",
      "independentSelector": false,
      "metricType": "REGULAR_VALUE",
      "metricName": "service_instance_throughput_received,service_instance_throughput_sent",
      "queryMetricType": "readMetricsValues",
      "chartType": "ChartLine",
      "unit": "Bytes"
    },
    {
      "width": "3",
      "title": "Service Instance Successful Rate",
      "height": "250",
      "entityType": "ServiceInstance",
      "independentSelector": false,
      "metricType": "REGULAR_VALUE",
      "metricName": "service_instance_sla",
      "queryMetricType": "readMetricsValues",
      "chartType": "ChartLine",
      "unit": "%",
      "aggregation": "/",
      "aggregationNum": "100"
    },
    {
      "width": "3",
      "title": "Service Instance Latency",
      "height": "250",
      "entityType": "ServiceInstance",
      "independentSelector": false,
      "metricType": "REGULAR_VALUE",
      "metricName": "service_instance_resp_time",
      "queryMetricType": "readMetricsValues",
      "chartType": "ChartLine",
      "unit": "ms"
    },
    {
      "width": 3,
      "title": "JVM CPU (Java Service)",
      "height": "250",
      "entityType": "ServiceInstance",
      "independentSelector": false,
      "metricType": "REGULAR_VALUE",
      "metricName": "instance_jvm_cpu",
      "queryMetricType": "readMetricsValues",
      "chartType": "ChartLine",
      "unit": "%",
      "aggregation": "+",
      "aggregationNum": ""
    },
    {
      "width": 3,
      "title": "JVM Memory (Java Service)",
      "height": "250",
      "entityType": "ServiceInstance",
      "independentSelector": false,
      "metricType": "REGULAR_VALUE",
      "metricName": "instance_jvm_memory_heap, instance_jvm_memory_heap_max,instance_jvm_memory_noheap, instance_jvm_memory_noheap_max",
      "queryMetricType": "readMetricsValues",
      "chartType": "ChartLine",
      "unit": "MB",
      "aggregation": "/",
      "aggregationNum": "1048576"
    },
    {
      "width": 3,
      "title": "JVM GC Time",
      "height": "250",
      "entityType": "ServiceInstance",
      "independentSelector": false,
      "metricType": "REGULAR_VALUE",
      "metricName": "instance_jvm_young_gc_time, instance_jvm_old_gc_time",
      "queryMetricType": "readMetricsValues",
      "chartType": "ChartLine",
      "unit": "ms"
    },
    {
      "width": 3,
      "title": "JVM GC Count",
      "height": "250",
      "entityType": "ServiceInstance",
      "independentSelector": false,
      "metricType": "REGULAR_VALUE",
      "queryMetricType": "readMetricsValues",
      "chartType": "ChartBar",
      "metricName": "instance_jvm_young_gc_count, instance_jvm_old_gc_count"
    },
    {
      "width": 3,
      "title": "JVM Thread Count (Java Service)",
      "height": "250",
      "entityType": "ServiceInstance",
      "independentSelector": false,
      "metricType": "REGULAR_VALUE",
      "queryMetricType": "readMetricsValues",
      "chartType": "ChartLine",
      "metricName": "instance_jvm_thread_live_count, instance_jvm_thread_daemon_count, instance_jvm_thread_peak_count,instance_jvm_thread_deadlocked,instance_jvm_thread_monitor_deadlocked"
    },
    {
      "width": 3,
      "title": "JVM Thread State Count (Java Service)",
      "height": "250",
      "entityType": "ServiceInstance",
      "independentSelector": false,
      "metricType": "REGULAR_VALUE",
      "metricName": "instance_jvm_thread_new_thread_count,instance_jvm_thread_runnable_thread_count,instance_jvm_thread_blocked_thread_count,instance_jvm_thread_wait_thread_count,instance_jvm_thread_time_wait_thread_count,instance_jvm_thread_terminated_thread_count",
      "queryMetricType": "readMetricsValues",
      "chartType": "ChartBar"
    },
    {
      "width": 3,
      "title": "JVM Class Count (Java Service)",
      "height": "250",
      "entityType": "ServiceInstance",
      "independentSelector": false,
      "metricType": "REGULAR_VALUE",
      "metricName": "instance_jvm_class_loaded_class_count,instance_jvm_class_unloaded_class_count,instance_jvm_class_total_loaded_class_count",
      "queryMetricType": "readMetricsValues",
      "chartType": "ChartArea"
    },
    {
      "width": 3,
      "title": "CLR CPU  (.NET Service)",
      "height": "250",
      "entityType": "ServiceInstance",
      "independentSelector": false,
      "metricType": "REGULAR_VALUE",
      "metricName": "instance_clr_cpu",
      "queryMetricType": "readMetricsValues",
      "chartType": "ChartLine",
      "unit": "%"
    },
    {
      "width": 3,
      "title": "CLR GC (.NET Service)",
      "height": "250",
      "entityType": "ServiceInstance",
      "independentSelector": false,
      "metricType": "REGULAR_VALUE",
      "metricName": "instance_clr_gen0_collect_count, instance_clr_gen1_collect_count, instance_clr_gen2_collect_count",
      "queryMetricType": "readMetricsValues",
      "chartType": "ChartBar"
    },
    {
      "width": 3,
      "title": "CLR Heap Memory (.NET Service)",
      "height": "250",
      "entityType": "ServiceInstance",
      "independentSelector": false,
      "metricType": "REGULAR_VALUE",
      "metricName": "instance_clr_heap_memory",
      "queryMetricType": "readMetricsValues",
      "chartType": "ChartLine",
      "unit": "MB",
      "aggregation": "/",
      "aggregationNum": "1048576"
    },
    {
      "width": 3,
      "title": "CLR Thread (.NET Service)",
      "height": "250",
      "entityType": "ServiceInstance",
      "independentSelector": false,
      "metricType": "REGULAR_VALUE",
      "queryMetricType": "readMetricsValues",
      "chartType": "ChartLine",
      "metricName": "instance_clr_available_completion_port_threads,instance_clr_available_worker_threads,instance_clr_max_completion_port_threads,instance_clr_max_worker_threads"
    }
  ]
}

結(jié)果校驗(yàn)

可以看到導(dǎo)入的界面中绝葡，已經(jīng)有 Class 相關(guān)指標(biāo)了

file

代碼貢獻(xiàn)

• Add some new thread metric and class metric to JVMMetric #7230
• add some new thread metric and class metric to JVMMetric #52
• Remove Terminated State and New State in JVMMetric (#7230) #53
• Add some new thread metric and class metric to JVMMetric (#7230) #7243

參考文檔

• Observability Analysis Language
• Source and Scope extension for new metrics
• Java ManagementFactory解析

分享并記錄所學(xué)所見(jiàn)