看了很多ELK日子分析系統(tǒng)的實現，多數是寫怎么搭建，至于怎么在不影響主項目的情況下汤踏、異步去記錄日志沒有過多說明，為此我以Logback為列舔腾，通過Logback底層原理溪胶，在不影響主代碼的情況下，實現ELK日志分析系統(tǒng)稳诚。
PS:至于安裝ELK哗脖，和ELK的說明的，可以參考一下連接：
https://www.cnblogs.com/kevingrace/p/5919021.html
https://www.cnblogs.com/yuhuLin/p/7018858.html
https://my.oschina.net/itblog/blog/547250

Logback:
Logback是由log4j創(chuàng)始人設計的另一個開源日志組件,官方網站： http://logback.qos.ch扳还。

看看（最）簡單的Logback配置logback.xml：

    <configuration>
    
        <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
            <!-- encoder 默認配置為PatternLayoutEncoder -->
            <encoder>
                <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
            </encoder>
        </appender>
    
        <root level="INFO">
            <appender-ref ref="STDOUT" />
        </root>
    
    </configuration>  

我們配置一個ConsoleAppender就可以在控制臺上打印出日志才避。

實現思路：
Logback（我用的是logback-classic-1.2.3版本）有個DBAppender，功能就是將日志插入數據庫氨距，這樣的話我們可以仿照它桑逝，實現將日志插入到elasticsearch的功能。
我們看看DBAppender所需要加的配置：

<appender name="db-classic-mysql" class="ch.qos.logback.classic.db.DBAppender">
<connectionSource class="ch.qos.logback.core.db.DataSourceConnectionSource">
    <dataSource class="com.mchange.v2.c3p0.ComboPooledDataSource">
        <driverClass>com.mysql.jdbc.Driver</driverClass>
        <jdbcUrl>jdbc:mysql://{$server ip}:3306/{$dbname}</jdbcUrl>
        <user>{$user}</user>
        <password>{$password}</password>
    </dataSource>
</connectionSource>
</appender>

從配置上看就是把參數傳到DBAppender里面俏让，我們ES的Appender只需仿照它的做法就可以楞遏。

目標：找出DBAppender的sql在哪里commit
我們再看看DBAppender：

image.png

其初始化參數：

protected String insertPropertiesSQL;
protected String insertExceptionSQL;
protected String insertSQL;
protected static final Method GET_GENERATED_KEYS_METHOD;
private DBNameResolver dbNameResolver;
static final int TIMESTMP_INDEX = 1;
static final int FORMATTED_MESSAGE_INDEX = 2;
static final int LOGGER_NAME_INDEX = 3;
static final int LEVEL_STRING_INDEX = 4;
static final int THREAD_NAME_INDEX = 5;
static final int REFERENCE_FLAG_INDEX = 6;
static final int ARG0_INDEX = 7;
static final int ARG1_INDEX = 8;
static final int ARG2_INDEX = 9;
static final int ARG3_INDEX = 10;
static final int CALLER_FILENAME_INDEX = 11;
static final int CALLER_CLASS_INDEX = 12;
static final int CALLER_METHOD_INDEX = 13;
static final int CALLER_LINE_INDEX = 14;
static final int EVENT_ID_INDEX = 15;
static final StackTraceElement EMPTY_CALLER_DATA = CallerData.naInstance();

涉及sql的就只有
insertPropertiesSQL茬暇，insertExceptionSQL，insertSQL寡喝。而我們順藤摸瓜而钞，發(fā)現最后他們都做這樣的操作：

super.start();

我們看看他的類繼承關系圖（idea自帶這個插件，eclipse可以安裝類似的插件）：

image.png

看看DBAppender繼承的DBAppenderBase：
protected abstract的就有getGeneratedKeysMethod();getInsertSQL();subAppend(E var1, Connection var2, PreparedStatement var3)拘荡；secondarySubAppend(E var1, Connection var2, long var3)臼节；
就說我們仿照DBAppender繼承DBAppenderBase就要實現上面4個方法；

看看DBAppenderBase繼承的UnsynchronizedAppenderBase：
protected abstract的就只有有append(E var1);

再看回DBAppenderBase的append(E eventObject)：

connection.commit();

Ｉ好蟆Ｍ臁！ｓā７垭！Ｊ欢怠６笾佟！３纭Ｍ佬住！Ｋ磷省４＠ⅰ！ＶＴ０隆！７咐纭Ｊ舴摺！Ｋ嵋邸Ｗ≈睢！４睾础Ｖ豢恰！Ｊ钏堋吼句！
！Ｊ赂瘛Ｌ柩蕖８阋！Ｔ短隆Ａ痈佟！Ｋⅰｑ尽！倘潜！就是在這里１疗狻！ｄ桃颉７夏馈！Ｑ荨Ｊ扰取！＠窖凇９号！！
！２琛＝柘！Ｓ旄健＠商印！ＭΨ荨０病！Ｔ炔础Ｓ叛怠！８髌浮４Х恰！６阋颉Ｔ缇础＜缮怠！８慵唷Ｋⅰ！Ｋ雎俊７帧！＞０擦啤！

這樣的話我們仿照DBAppenderBase寫一個ESAppender够委，繼承UnsynchronizedAppenderBase荐类，實現append方法，將日志插入ES里茁帽！

下面就是貼代碼玉罐、貼圖片時間：
項目結構圖：

image.png

主類ESAppender：

package com.elk.log.appender;

import ch.qos.logback.classic.spi.ILoggingEvent;
import ch.qos.logback.classic.spi.IThrowableProxy;
import ch.qos.logback.classic.spi.ThrowableProxyUtil;
import ch.qos.logback.core.UnsynchronizedAppenderBase;
import com.alibaba.fastjson.JSON;
import com.elk.log.utils.InitES;
import com.elk.log.vo.EsLogVo;
import com.elk.log.vo.Location;
import io.searchbox.client.JestClient;
import io.searchbox.client.JestResult;
import io.searchbox.core.Index;


import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.Map;
import java.util.Properties;

public class ESAppender  extends UnsynchronizedAppenderBase<ILoggingEvent> {

    private static JestClient jestClient ;

    //索引名稱
    String esIndex = "java-log-#date#";
    //索引類型
    String esType = "java-log";
    //是否打印行號
    boolean isLocationInfo = true;
    //運行環(huán)境
    String env = "";
    //es地址
    String esAddress = "";

    public String getEsIndex() {
        return esIndex;
    }

    public void setEsIndex(String esIndex) {
        this.esIndex = esIndex;
    }

    public String getEsType() {
        return esType;
    }

    public void setEsType(String esType) {
        this.esType = esType;
    }

    public boolean isLocationInfo() {
        return isLocationInfo;
    }

    public void setLocationInfo(boolean locationInfo) {
        isLocationInfo = locationInfo;
    }

    public String getEnv() {
        return env;
    }

    public void setEnv(String env) {
        this.env = env;
    }

    public String getEsAddress() {
        return esAddress;
    }

    public void setEsAddress(String esAddress) {
        this.esAddress = esAddress;
    }

    @Override
    protected void append(ILoggingEvent event) {
        EsLogVo esLogVo = new EsLogVo();
        esLogVo.setHost("HostName");
        esLogVo.setIp("127.0.0.1");
        esLogVo.setEnv(this.env);
        esLogVo.setLevel(event.getLevel().toString());
        Location location = new Location();
        StackTraceElement[] callerDataArray = event.getCallerData();
        if(callerDataArray != null && callerDataArray.length >0){
            StackTraceElement immediateCallerData = callerDataArray[0];
            location.setClassName(immediateCallerData.getClassName());
            location.setMethod(immediateCallerData.getMethodName());
            location.setFile(immediateCallerData.getFileName());
            location.setLine(Integer.toString(immediateCallerData.getLineNumber()));
        }
        IThrowableProxy tp = event.getThrowableProxy();
        if (tp != null){
            String throwable = ThrowableProxyUtil.asString(tp);
            esLogVo.setThrowable(throwable);
        }
        esLogVo.setLocation(location);
        esLogVo.setLogger(event.getLoggerName());
        esLogVo.setMessage(event.getFormattedMessage());

        SimpleDateFormat df = new SimpleDateFormat("yyyy.MM.dd");
        SimpleDateFormat df2 = new SimpleDateFormat("yyyy-MM-dd");
        esLogVo.setTimestamp(df2.format(new Date(event.getTimeStamp())));
        esLogVo.setThread(event.getThreadName());

        Map<String ,String > mdcPropertyMap = event.getMDCPropertyMap();
        esLogVo.setTraceId(mdcPropertyMap.get("traceId"));
        esLogVo.setRpcId(mdcPropertyMap.get("rpcId"));


        String jsonString = JSON.toJSONString(esLogVo);
        String esIndex_format = esIndex.replace("#date#",df.format(new Date(event.getTimeStamp())));
        Index index = new Index.Builder(esLogVo).index(esIndex_format).type(esType).build();
        try{
            JestResult result = jestClient.execute(index);
            System.out.println(result);
        }catch (Exception e){
            e.printStackTrace();
        }

    }

    @Override
    public void start() {
        super.start();
        Properties properties = new Properties();
        properties.put("es.hosts",esAddress);
        properties.put("es.username","zkpk");
        properties.put("es.password","123456");
        jestClient = InitES.jestClient(properties);
    }
    
    @Override
    public void stop() {
        super.stop();
        jestClient.shutdownClient();
    }
}

我們主要實現append方法，重寫start()潘拨，stop()吊输。
append主要是插入es索引，start和stop主要用來打開和關閉ES的鏈接铁追。

工具類InitES（用于初始化es連接用的）：

    package com.elk.log.utils;

import io.searchbox.client.JestClientFactory;
import io.searchbox.client.config.HttpClientConfig;
import io.searchbox.client.JestClient;


import java.util.ArrayList;
import java.util.LinkedHashSet;
import java.util.List;
import java.util.Properties;
public class InitES {

    private static io.searchbox.client.JestClient JestClient;

    public static JestClient jestClient(Properties properties) {
        JestClientFactory factory = new JestClientFactory();
        String userName = properties.getProperty("es.username");
        String password = properties.getProperty("es.password");
        String esHosts = properties.getProperty("es.hosts");
        List<String> serverList = new ArrayList <String>();
        for (String address:esHosts.split(",")) {
            serverList.add("http://"+address);
        }

        HttpClientConfig.Builder builder = new HttpClientConfig.Builder(serverList);
        builder.maxTotalConnection(20);
        builder.defaultMaxTotalConnectionPerRoute(5);
        builder.defaultCredentials(userName,password);
        builder.multiThreaded(true);

        factory.setHttpClientConfig(builder.build());

        if (JestClient == null) {
            JestClient = factory.getObject();
        }
        return JestClient;
    }
}

這里季蚂，我們用Jest方式連接ES。

兩個Javabean：EsLogVo琅束，Location：

public  class EsLogVo {

    private String host;
    private String ip;
    private String env;
    private String message;
    private String timestamp;
    private String logger;
    private String level;
    private String thread;
    private String throwable;
    private Location location;
    private String traceId;
    private String rpcId;
  ...set and get...
    
}

public class Location {
    private String className;
    private String method;
    private String file;
    private String line;

  ...set and get...
    
}

測試類ESLogTest：

package com.elk.log;

import org.junit.Test;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.util.concurrent.TimeUnit;

public class ESLogTest {
    private static final Logger logger = LoggerFactory.getLogger(ESLogTest.class);

    @Test
    public  void testLog() throws InterruptedException{
    logger.info("我是正常信息 test message info");
    logger.error("我一條異常的信息",new Exception("項目報錯了扭屁，加班吧！Ｉ鳌Ａ侠摹！"));
    logger.debug("debug消息 debug hello hi");
    logger.warn("警告警告");
    TimeUnit.SECONDS.sleep(10);
    }

}

配置文件logback.xml：

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <contextName>logback-api</contextName>
    <property name="logback.system" value="logback-system"/>
    <property name="logback.path" value="../logs/logback-system"/>
    <property name="logback.level" value="DEBUG"/>
    <property name="logback.pattern" value="%d{yyyy-MM-dd HH:mm:ss} [%t] %-5p - %m%n"/>
    <property name="logback.env" value="dev"/>
    <property name="logback.isLocation" value="true"/>
    <property name="logback.esAddress" value="192.168.0.128:9200,192.168.0.129:9200,192.168.0.130:9200"/>


    <appender name="stdout" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>[%-5p %d{yy-MM-dd HH:mm:ss} %m] %caller{1}</pattern>
        </encoder>
    </appender>


    <appender name="ES" class="com.elk.log.appender.ESAppender">
        <!--索引名字 date為通配符 艾船，會自動替換為yyyy.MM.dd格式-->
        <esIndex>java-log-#date#</esIndex>
        <!--索引類型-->
        <esType>${logback.system}</esType>
        <!--運行環(huán)境-->
        <env>${logback.env}</env>
        <!--ES地址-->
        <esAddress>${logback.esAddress}</esAddress>
    </appender>

    <appender name="ASYNC_ES" class="ch.qos.logback.classic.AsyncAppender">
        <!--默認情況下葵腹，當BlockingQueue還有20%容量，他將丟棄TRACE,DEBUG和INFO級別的event屿岂，只保留warn和error-->
        <discardingThreshold>0</discardingThreshold>
        <!--BlockingQueue的最大容量践宴，默認情況下，大小為256-->
        <queueSize>256</queueSize>
        <appender-ref ref="ES"/>
        <!--要是保留行號爷怀，需要開啟為true-->
        <includeCallerData>true</includeCallerData>
    </appender>

    <logger name="com.elk.log" additivity="true">
        <level value="${logback.level}"></level>
        <appender-ref ref="ASYNC_ES" />
    </logger>

    <root level="${logback.level}">
        <appender-ref ref="stdout" />
    </root>
</configuration>

這里我們定義自己的ESAppender阻肩，取名未ES，并將所需參數傳到ESAppender霉撵。
然后我們再把它放到AsyncAppender里面進行異步處理磺浙，防止ELK報錯影響到主程序洪囤。

OK,測試：
我們啟動ELK，在Kibana的Dev Tools上創(chuàng)建一個java的動態(tài)日志模板：

    PUT _template/java-log
    {
        "template" : "java-log-*",
            "order" : 0,
            "settings" : {
        "index":{
            "refresh_interval": "5s"
        }
    },
        "mappings": {
        "_default_": {
            "dynamic_templates": [
            {
                "message_field": {
                "match_mapping_type": "string",
                        "path_match": "message",
                        "mapping": {
                    "norms": false,
                            "type": "text",
                            "analyzer": "ik_max_word",
                            "search_analyzer": "ik_max_word"
                }
            }
            },
            {
                "throwable_fields": {
                "match_mapping_type": "string",
                        "path_match": "throwable",
                        "mapping": {
                    "norms": false,
                            "type": "text",
                            "analyzer": "ik_max_word",
                            "search_analyzer": "ik_max_word"
                }
            }
            },
            {
                "string_fields": {
                "match_mapping_type": "string",
                        "match": "*",
                        "mapping": {
                    "norms": false,
                            "type": "text",
                            "analyzer": "ik_max_word",
                            "search_analyzer": "ik_max_word",
                            "fields": {
                        "keyword": {
                            "type": "keyword"
                        }
                    }
                }
            }
            }
  ],
            "_all": {
                "enabled": false
            },
            "properties": {
                "env": {
                    "type": "keyword"
                },
                "host": {
                    "type": "keyword"
                },
                "ip": {
                    "type": "ip"
                },
                "level": {
                    "type": "keyword"
                },
                "location": {
                    "properties": {
                        "line": {
                            "type": "integer"
                        }
                    }
                },
                "timestamp": {
                    "type": "date"
                }
            }
        }
    }
    }

用的是IK分詞器撕氧，refresh時間定為5s用于生產上調高效率瘤缩，定義了幾個參數作為關鍵搜索。

然后將java-log-*作為主要觀察對象伦泥，回到Discover

回到idea剥啤，在ESLogTest上跑testLog：

        [INFO  18-03-07 16:56:49 我是正常信息 test message info] Caller+0  at com.elk.log.ESLogTest.testLog(ESLogTest.java:14)
[ERROR 18-03-07 16:56:49 我一條異常的信息] Caller+0  at com.elk.log.ESLogTest.testLog(ESLogTest.java:15)
        java.lang.Exception: 項目報錯了，加班吧２桓８印！防楷！
        at com.elk.log.ESLogTest.testLog(ESLogTest.java:15)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.junit.internal.runners.TestMethod.invoke(TestMethod.java:59)
        at org.junit.internal.runners.MethodRoadie.runTestMethod(MethodRoadie.java:98)
        at org.junit.internal.runners.MethodRoadie$2.run(MethodRoadie.java:79)
        at org.junit.internal.runners.MethodRoadie.runBeforesThenTestThenAfters(MethodRoadie.java:87)
        at org.junit.internal.runners.MethodRoadie.runTest(MethodRoadie.java:77)
        at org.junit.internal.runners.MethodRoadie.run(MethodRoadie.java:42)
        at org.junit.internal.runners.JUnit4ClassRunner.invokeTestMethod(JUnit4ClassRunner.java:88)
        at org.junit.internal.runners.JUnit4ClassRunner.runMethods(JUnit4ClassRunner.java:51)
        at org.junit.internal.runners.JUnit4ClassRunner$1.run(JUnit4ClassRunner.java:44)
        at org.junit.internal.runners.ClassRoadie.runUnprotected(ClassRoadie.java:27)
        at org.junit.internal.runners.ClassRoadie.runProtected(ClassRoadie.java:37)
        at org.junit.internal.runners.JUnit4ClassRunner.run(JUnit4ClassRunner.java:42)
        at org.junit.runner.JUnitCore.run(JUnitCore.java:130)
        at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
        at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
        at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
        at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
[DEBUG 18-03-07 16:56:49 debug消息 debug hello hi] Caller+0    at com.elk.log.ESLogTest.testLog(ESLogTest.java:16)
[WARN  18-03-07 16:56:49 警告警告] Caller+0  at com.elk.log.ESLogTest.testLog(ESLogTest.java:17)
[DEBUG 18-03-07 16:56:50 Request and operation succeeded] Caller+0   at io.searchbox.action.AbstractAction.createNewElasticSearchResult(AbstractAction.java:75)
        io.searchbox.core.DocumentResult@4421d75d
                [DEBUG 18-03-07 16:56:51 Request and operation succeeded] Caller+0   at io.searchbox.action.AbstractAction.createNewElasticSearchResult(AbstractAction.java:75)
        io.searchbox.core.DocumentResult@25a5426c
                [DEBUG 18-03-07 16:56:51 Request and operation succeeded] Caller+0   at io.searchbox.action.AbstractAction.createNewElasticSearchResult(AbstractAction.java:75)
        io.searchbox.core.DocumentResult@5fd33fbc
                [DEBUG 18-03-07 16:56:51 Request and operation succeeded] Caller+0   at io.searchbox.action.AbstractAction.createNewElasticSearchResult(AbstractAction.java:75)
        io.searchbox.core.DocumentResult@25a52369

可以說都是我們想要的信息牺丙。

我們再回到Kibana：

image.png

可以看到，測試上的4條日志信息已經可以插到ES上复局。

然后再看看錯誤的那條日志信息：

image.png

報錯的類冲簿，方法，行數亿昏，報錯內容峦剔。。角钩。都一一列出來吝沫。

任務完成，到此递礼，我們可以在不影響其他代碼的情況下惨险，將日志插入到ES里，在以后的編程中可以快速定位問題宰衙。

擴展：
我們看看除了DBAppender實現Appender這條線外平道，還有哪些類走這條線：

image.png

我們看到實現Appender接口的就有兩個類：AppenderBase和UnsynchronizedAppenderBase。
從字面上的意思就是一個同步的供炼，一個異步的，我們用對比工具對比一下窘疮，發(fā)現這兩個類幾乎沒什么區(qū)別袋哼，最大不同就是AppenderBase的doAppend方法多了synchronized的鎖。
像插到數據庫日志的DBAppender闸衫，控制臺打印日志的ConsoleAppender涛贯。。蔚出。就會走異步的UnsynchronizedAppenderBase這條線弟翘。
像SMTPAppender虫腋，JMSAppenderBase，SocketAppenderBase稀余。悦冀。。就會走AppenderBase這條線睛琳。
所以像DBAppender和DBAppenderBase盒蟆，這兩個類可以合成一個整體，直接繼承UnsynchronizedAppenderBase师骗，走異步這條線历等。

-----------------------謝謝觀看，希望本文對你有幫助----------------------------