ELK配合Redis接收NGINX、HTTPD讨韭、SYSLOG脂信、TOMCAT日志

[TOC]

一癣蟋、Library Environment

Hostname IP Software OS Identity Remark
master.app.com 10.66.3.155 Elasticsearch、Logstash狰闪、Kibana疯搅、Redis、Redis-browser埋泵、Java CentOS release 6.6 服務(wù)端 軟件自備
node1.app.com 10.66.3.136 Logstash幔欧、Java、Nginx丽声、Rsyslog CentOS release 6.6 客戶端 軟件自備

開(kāi)機(jī)啟動(dòng)命令（這里作說(shuō)明礁蔗，先不用操作）

# vim /etc/rc.local  //需創(chuàng)建elasticsearch、kibana用戶雁社，并修改相應(yīng)目錄權(quán)限浴井；logstash因?yàn)橐l(fā)郵件，所以用root用戶身份運(yùn)行

su -l -c "su elasticsearch /opt/elasticsearch/bin/elasticsearch >/dev/null 2>&1 &"
su -l -c "su kibana /opt/kibana/bin/kibana >/dev/null 2>&1 &"
su -l -c "nohup /opt/logstash/bin/logstash -f /opt/logstash/config/log_indexer_tomcat_catalina_local_250.conf >/dev/null 2>&1 &"

二歧胁、Software Installation

2.1.Server Software Installation

2.1.1 Install JDK
# tar xf jdk-7u79-linux-x64.tar.gz -C /opt/  //解壓JDK至指定目錄
# ln -sv /opt/jdk1.7.0_79/ /opt/java  //創(chuàng)建鏈接滋饲，保留原來(lái)目錄可以方便一目了然的看到版本號(hào)
# vim /etc/profile.d/java.sh  //創(chuàng)建環(huán)境變量

export JAVA_HOME=/opt/java
export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$PATH:$JAVA_HOME/bin

# . /etc/profile.d/java.sh  //使環(huán)境變量生效
# java -version  // 查看是否生效，生效了會(huì)顯示java版本信息

java version "1.7.0_79" Java(TM) SE Runtime Environment (build 1.7.0_79-b15) Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)

2.1.2 Install Elasticsearch
# groupadd -g 92 elasticsearch
# useradd -g 92 -u 92 elasticsearch
# tar xf elasticsearch-2.3.1.tar.gz -C /opt/  //解壓elasticsearch至指定目錄
# ln -sv /opt/elasticsearch-2.3.1/bin/elasticsearch /usr/bin/  //將elasticsearch可執(zhí)行文件鏈接至/usr/bin目錄喊巍，方便在任何目錄下都可直接執(zhí)行`elasticsearch`
# chown -R elasticsearch.elasticsearch /opt/elasticsearch/
2.1.3 Install Logstash
# tar xf logstash-2.3.1.tar.gz -C /opt/  //解壓logstash至指定目錄
# ln -sv /opt/logstash-2.3.1/bin/logstash /usr/bin/  //將logstash可執(zhí)行文件鏈接至/usr/bin目錄屠缭，方便在任何目錄下都可直接執(zhí)行`logstash`
2.1.4 Install Redis
# tar xf redis-3.0.7.tar.gz  //解壓至當(dāng)前目錄
# cd redis-3.0.7  //切換至redis源碼目錄
# make  //編譯
# yum install tcl  //安裝測(cè)試依賴工具
# make test  //有可能會(huì)失敗，只是看看崭参，不用在意
# make install  //安裝
# mkdir /opt/redis/{db,conf} -pv  //創(chuàng)建redis安裝目錄
# cp redis.conf /opt/redis/conf/  //復(fù)制配置文件至redis安裝目錄
# cd src
# cp redis-benchmark redis-check-aof redis-check-dump redis-cli redis-server mkreleasehdr.sh /opt/redis/  //復(fù)制各文件至redis安裝目錄
# ln -sv /opt/redis/redis-cli /usr/bin/  //將redis-cli可執(zhí)行文件鏈接至/usr/bin目錄呵曹，方便在任何目錄下都可直接執(zhí)行`redis-cli`
# vim /opt/redis/conf/redis.conf  //修改redis.conf 中的 `daemonize`為`yes`,讓server以守護(hù)進(jìn)程在后臺(tái)執(zhí)行，這一步可以不做何暮，因?yàn)楹竺嬉獔?zhí)行的腳本會(huì)自動(dòng)創(chuàng)建這個(gè)文件奄喂，且這個(gè)值會(huì)設(shè)置為`yes`

daemonize yes

make install僅僅在你的系統(tǒng)上安裝了二進(jìn)制文件，不會(huì)替你默認(rèn)配置init腳本和配置文件海洼，為了把它用在生產(chǎn)環(huán)境而安裝它跨新，在源碼目錄的utils目錄下Redis為系統(tǒng)提供了
這樣的一個(gè)腳本install_server.sh

#　./utils/install_server.sh  //執(zhí)行sh格式的安裝腳本

`Welcome to the redis service installer
This script will help you easily set up a running redis server
Please select the redis port for this instance: [6379]
Selecting default: 6379
Please select the redis config file name [/etc/redis/6379.conf] /opt/redis/conf/redis.conf
Please select the redis log file name [/var/log/redis_6379.log]
Selected default - /var/log/redis_6379.log
Please select the data directory for this instance [/var/lib/redis/6379] /opt/redis/db/6379.db
Please select the redis executable path [/usr/bin/redis-server]
Selected config:

Port : 6379
Config file : /opt/redis/conf/redis.conf
Log file : /var/log/redis_6379.log
Data dir : /opt/redis/db/6379.db
Executable : /opt/redis/redis-server
Cli Executable : /usr/bin/redis-cli`

# chkconfig --add redis_6379  //將redis加入系統(tǒng)服務(wù)
# chkconfig redis_6379 on  //加入開(kāi)機(jī)啟動(dòng)
# vim /opt/redis/conf/redis.conf

requirepass Carsing2582#    //設(shè)置密碼

# /etc/init.d/redis_6379 restart
2.1.5 Install Kibana
# groupadd -g 56 kibana
# useradd -g 56 -u 56 kibana
# tar xf kibana-4.5.0-linux-x64.tar.gz -C /opt/  //解壓kibana至指定目錄
# ln -sv /opt/kibana-4.5.0-linux-x64/bin/kibana /usr/bin/  //將kibana可執(zhí)行文件鏈接至/usr/bin目錄，方便在任何目錄下都可直接執(zhí)行`kibana`
# chown -R kibana.kibana /opt/kibana/

2.2 Client Software Installation

2.2.1 Install JDK
# tar xf jdk-7u79-linux-x64.tar.gz -C /opt/  //解壓JDK至指定目錄
# ln -sv /opt/jdk1.7.0_79/ /opt/java  //創(chuàng)建鏈接坏逢，保留原來(lái)目錄可以方便一目了然的看到版本號(hào)
# vim /etc/profile.d/java.sh  //創(chuàng)建環(huán)境變量

export JAVA_HOME=/opt/java
export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$PATH:$JAVA_HOME/bin

# . /etc/profile.d/java.sh  //使環(huán)境變量生效
# java -version  // 查看是否生效域帐，生效了會(huì)顯示java版本信息

java version "1.7.0_79" Java(TM) SE Runtime Environment (build 1.7.0_79-b15) Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)

2.2.2 Install Logstash
# tar xf logstash-2.3.1.tar.gz -C /opt/  //解壓logstash至指定目錄
# ln -sv /opt/logstash-2.3.1/bin/logstash /usr/bin/  //將logstash可執(zhí)行文件鏈接至/usr/bin目錄，方便在任何目錄下都可直接執(zhí)行`logstash`

三是整、Start Service

服務(wù)端

3.1 Start Redis

# /etc/init.d/redis_6379 start
# netstat -tnlp  //查看是否有6379端口

3.2 Start Elasticsearch

elasticsearch只能以普通用戶運(yùn)行

# nohup elasticsearch >nohup &  //啟動(dòng)并放至后臺(tái)運(yùn)行
# vim /etc/rc.local  //設(shè)置開(kāi)機(jī)啟動(dòng)

su -l -c "su elasticsearch /opt/elasticsearch/bin/elasticsearch >/dev/null 2>&1 &"

# netstat -tnlp  //ElasticSearch默認(rèn)的對(duì)外服務(wù)的HTTP端口是9200肖揣，節(jié)點(diǎn)間交互的TCP端口是9300，注意打開(kāi)tcp端口
# exit  //退出當(dāng)前用戶

http://10.66.3.155:9200 //可以看到如下信息

{ "name" : "node0", "cluster_name" : "es_cluster", "version" : { "number" : "2.3.1", "build_hash" : "bd980929010aef404e7cb0843e61d0665269fc39", "build_timestamp" : "2016-04-04T12:25:05Z", "build_snapshot" : false, "lucene_version" : "5.5.0" }, "tagline" : "You Know, for Search" }

# curl -X GET http://10.66.3.155:9200  //獲取網(wǎng)頁(yè)內(nèi)容
# curl -I GET http://10.66.3.155:9200  //獲取網(wǎng)頁(yè)頭部信息,200正常

3.3 Start Kibana

# nohup kibana >nohup &  //啟動(dòng)并放至后臺(tái)運(yùn)行
# vim /etc/rc.local  //設(shè)置開(kāi)機(jī)啟動(dòng)

su -l -c "su kibana /opt/kibana/bin/kibana >/dev/null 2>&1 &"

# netstat -tnlp  //啟用端口為5601

http://10.66.3.155:5601 //可以看到網(wǎng)頁(yè)內(nèi)容
# vim /etc/rc.local //開(kāi)機(jī)自啟動(dòng)

su -l -c "nohup /opt/logstash/bin/logstash -f /opt/logstash/config/log_indexer_tomcat_catalina_local_250.conf >/dev/null 2>&1 &"

四浮入、Monitor Nginx Log

4.1 只監(jiān)控access日志龙优，，從文件獲取將access日志導(dǎo)向服務(wù)端的redis

客戶端

# vim /opt/logstash-2.3.1/conf/log_agent_nginx_access.conf  //定義一個(gè)實(shí)例配置事秀，從access.log獲取日志并存儲(chǔ)至redis

input {
    file {
        type => "nginx access log"
        path => ["/var/log/nginx/access.log"] 
    }
}
output {
    redis {
        host => "10.66.3.155"
        port => "6379"
        data_type => "list"
        key => "nginx_access_136:redis"
    }
}

# /opt/logstash-2.3.1/bin/logstash -f /opt/logstash-2.3.1/conf/log_nginx_access.log  //啟動(dòng)實(shí)例測(cè)試,正常顯示如下

Settings: Default pipeline workers: 8 Pipeline main started

ctrl + c 退出

# nohup /opt/logstash-2.3.1/bin/logstash -f /opt/logstash-2.3.1/conf/log_nginx_access.log >nohup &  //放至后臺(tái)運(yùn)行
 # vim /etc/rc.local  //開(kāi)機(jī)自啟動(dòng)

su -l -c "nohup /opt/logstash/bin/logstash -f /opt/logstash/config/log_nginx_access.log >/dev/null 2>&1 &"

服務(wù)端

# redis-cli  //登錄redis
# 127.0.0.1:6379> exists nginx_access_136:redis

(integer) 1 //如果存在會(huì)顯示這個(gè)

# vim /opt/logstash-2.3.1/config/log_indexer_nginx_access_136.conf  //從redis獲取鍵名為nginx_access_136:redis的日志彤断，將之過(guò)慮并導(dǎo)向elasticsearch

input {
    redis {
        host => "10.66.3.155"
        port => "6379"
        data_type => "list"
        key => "nginx_access_136:redis"
        type => "redis-input"
        }
}
 
filter {
    if [type] =~ "nginx access log" {
        mutate {
            replace => { "type" => "apache_access" }
        }
        grok {
            match => { "message" => "%{COMBINEDAPACHELOG}" }
        }
    }
 
    date {
        match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
    }
}
 
output {
    elasticsearch {
        hosts => "10.66.3.155:9200"
    }
    stdout { codec => rubydebug }
}

# logstash -f /opt/logstash-2.3.1/config/log_indexer_nginx_access_136.conf  //啟動(dòng)一個(gè)實(shí)例野舶，如果正常會(huì)顯示如下信息

{ "message" => "10.66.0.1 - - [15/Apr/2016:12:30:05 +0800] \"POST /weixin/services/SysServiceLog?wsdl HTTP/1.0\" 200 594 \"-\" \"Axis/1.4\"", "@version" => "1", "@timestamp" => "2016-04-15T04:30:05.000Z", "path" => "/opt/nginx/logs/access.log", "host" => "LO-T-DEMO-AP", "type" => "apache_access", "clientip" => "10.66.0.1", "ident" => "-", "auth" => "-", "timestamp" => "15/Apr/2016:12:30:05 +0800", "verb" => "POST", "request" => "/weixin/services/SysServiceLog?wsdl", "httpversion" => "1.0", "response" => "200", "bytes" => "594", "referrer" => "\"-\"", "agent" => "\"Axis/1.4\"" }

ctrl+c取消，可以放到后臺(tái)運(yùn)行

# nohup logstash -f /opt/logstash-2.3.1/config/log_indexer_nginx_access_136.conf >nohup &  //放至后臺(tái)運(yùn)行
# vim /etc/rc.local  //設(shè)置開(kāi)機(jī)啟動(dòng)

su -l -c "nohup /opt/logstash/bin/logstash -f /opt/logstash/config/log_indexer_nginx_access_136.conf >/dev/null 2>&1 &"

4.2 同時(shí)監(jiān)控access瓦糟、error日志筒愚，，從文件獲取將access菩浙、error日志導(dǎo)向服務(wù)端的redis

客戶端

# vim /opt/logstash-2.3.1/conf/log_agent_nginx_all.conf  //定義一個(gè)實(shí)例配置，從access.log句伶、error*.log獲取日志并存儲(chǔ)至redis

input {
    file {
        path => "/opt/nginx/logs/access.log"
        type => "nginx_access"
    }
    file {
        path => "/opt/nginx/logs/erro*.log"
        type => "nginx_error"
    }
}
 
output {
    redis {
        host => "10.66.3.155"
        port => "6379"
        data_type => "list"
        key => "nginx_all_136:redis"
    }
}

# /opt/logstash-2.3.1/bin/logstash -f /opt/logstash-2.3.1/conf/log_nginx_all.log  //啟動(dòng)實(shí)例測(cè)試,正常顯示如下

Settings: Default pipeline workers: 8 Pipeline main started

ctrl + c 退出

# nohup /opt/logstash-2.3.1/bin/logstash -f /opt/logstash-2.3.1/conf/log_nginx_all.log >nohup &  //放至后臺(tái)運(yùn)行
# vim /etc/rc.local  //設(shè)置開(kāi)機(jī)自啟動(dòng)

su -l -c "nohup /opt/logstash/bin/logstash -f /opt/logstash/config/log_nginx_all.log >/dev/null 2>&1 &"

服務(wù)端

# redis-cli  //登錄redis劲蜻，驗(yàn)證鍵名是否存在
# 127.0.0.1:6379> exists nginx_all_136:redis

(integer) 1 //如果存在會(huì)顯示這個(gè)

# vim /opt/logstash-2.3.1/config/log_indexer_nginx_all_136.conf  //從redis獲取鍵名為`nginx_all_136:redis`的日志，將之過(guò)慮并導(dǎo)向elasticsearch  

input {
    redis {
        host => "10.66.3.155"
        port => "6379"
        data_type => "list"
        key => "nginx_all_136:redis"
        type => "redis-input"
        }
}
 
filter {
    if [type] =~ "access" {
        mutate {
            replace => { type => "apache_access" }
        }
        grok {
            match => { "message" => "%{COMBINEDAPACHELOG}" }
        }
        date {
            match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
        }
    }
    else if [type] =~ "error" {
        mutate { replace => { type => "apache_error" } }
    }
    else {
        mutate { replace => { type => "random_logs" } }
    }
}
 
output {
    elasticsearch {
        hosts => "10.66.3.155:9200"
    }
    stdout { codec => rubydebug }
}

# logstash -f /opt/logstash-2.3.1/config/log_indexer_nginx_all_136.conf  //啟動(dòng)一個(gè)實(shí)例考余，如果正常先嬉，會(huì)顯示如下信息

{ "message" => "10.66.0.1 - - [15/Apr/2016:19:15:05 +0800] \"POST /weixin/services/SysServiceLog?wsdl HTTP/1.0\" 200 435 \"-\" \"Axis/1.4\"", "@version" => "1", "@timestamp" => "2016-04-15T11:15:05.000Z", "path" => "/opt/nginx/logs/access.log", "host" => "LO-T-DEMO-AP", "type" => "apache_access", "clientip" => "10.66.0.1", "ident" => "-", "auth" => "-", "timestamp" => "15/Apr/2016:19:15:05 +0800", "verb" => "POST", "request" => "/weixin/services/SysServiceLog?wsdl", "httpversion" => "1.0", "response" => "200", "bytes" => "435", "referrer" => "\"-\"", "agent" => "\"Axis/1.4\"" }

ctrl+c取消，可以放到后臺(tái)運(yùn)行

# nohup logstash -f /opt/logstash-2.3.1/config/log_indexer_nginx_all_136.conf >nohup &  //放至后臺(tái)運(yùn)行
# vim /etc/rc.local  //設(shè)置開(kāi)機(jī)啟動(dòng)

su -l -c "nohup /opt/logstash/bin/logstash -f /opt/logstash/config/log_indexer_nginx_all_136.conf >/dev/null 2>&1 &"

五楚堤、Monitor System Log

5.1Client rsyslog Install (If not installed)

客戶端

# yum install rsyslog

5.2 The configuration of /etc/rsyslog.conf

客戶端

# vim /etc/rsyslog.conf  //在最后一行加上如下所示疫蔓，5000端口為服務(wù)端自定義的，服務(wù)端是多少這里就是多少

*.* @10.66.3.155:5000

# vim /etc/bashrc  //定義日常命令操作也記錄至syslog日志身冬，最后一行加入如下所示

export PROMPT_COMMAND='{ msg=$(history 1 | { read x y; echo $y; });logger "[euid=$(whoami)]":$(who am i):[`pwd`]"$msg"; }'

# service rsyslog restart  //重啟rsyslog服務(wù)

5.3 Start the instance on the server

服務(wù)端

# vim /opt/logstash-2.3.1/config/log_agent_136.conf  //定義logstash實(shí)例監(jiān)聽(tīng)在5000端口衅胀，接收10.66.3.136發(fā)過(guò)來(lái)的日志

input {
    tcp {
        port => 5000
        type => syslog
    }
    udp {
        port => 5000
        type => syslog
    }
}
output {
    redis {
        host => "10.66.3.155"
            port => "6379"
            data_type => "list"
            key => "syslog_136:redis"
        }
}

# logstash -f /opt/logstash-2.3.1/config/log_agent_syslog_136.conf  //啟動(dòng)實(shí)例，如果正常酥筝，會(huì)顯示如下信息

Settings: Default pipeline workers: 6 Pipeline main started

ctrl + c退出滚躯，可以放至后臺(tái)運(yùn)行

# nohup logstash -f /opt/logstash-2.3.1/config/log_agent_syslog_136.conf >nohup &  //放至后臺(tái)運(yùn)行
# vim /etc/rc.local  //設(shè)置開(kāi)機(jī)啟動(dòng)

su -l -c "nohup /opt/logstash/bin/logstash -f /opt/logstash/config/log_agent_syslog_136.conf >/dev/null 2>&1 &"

# redis-cli  //登錄redis，驗(yàn)證是否獲取信息
# 127.0.0.1:6379> exists syslog_136:redis

(integer) 1 //如果存在會(huì)顯示這個(gè)

# vim /opt/logstash-2.3.1/config/log_indexer_syslog_136.conf  //定義實(shí)例從redis中取得鍵名為`syslog_136:redis`的數(shù)據(jù),并將數(shù)據(jù)導(dǎo)向elasticsearch  

input {
    redis {
        host => "10.66.3.155"
        port => "6379"
        data_type => "list"
        key => "syslog_136:redis"
        type => "redis-input"
        }
}
 
filter {
 
    if [type] == "syslog" {
 
        grok {
            match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
 
            add_field => [ "received_at", "%{@timestamp}" ]
            add_field => [ "received_from", "%{host}" ]
        }
 
        syslog_pri { }
 
        date {
            match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
        }
 
    }
 
}
 
output {
    elasticsearch {
        hosts => "10.66.3.155:9200"
    }
    stdout {
        codec => rubydebug
    }
}

# logstash -f /opt/logstash-2.3.1/config/log_indexer_syslog_136.conf  //啟動(dòng)實(shí)例嘿歌，正常顯示如下

`{

             "message" => "<13>Apr 15 19:46:52 LO-T-DEMO-AP root: [euid=root]:root pts/0 2016-04-15 18:51 (10.66.13.36):[/opt/logstash-2.3.1]/opt/logstash-2.3.1/bin/logstash -f /opt/logstash-2.3.1/conf/log_nginx_access.log",
            "@version" => "1",
          "@timestamp" => "2016-04-15T11:46:52.000Z",
                "type" => "syslog",
                "host" => "10.66.3.136",
    "syslog_timestamp" => "Apr 15 19:46:52",
     "syslog_hostname" => "LO-T-DEMO-AP",
      "syslog_program" => "root",
      "syslog_message" => "[euid=root]:root pts/0 2016-04-15 18:51 (10.66.13.36):[/opt/logstash-2.3.1]/opt/logstash-2.3.1/bin/logstash -f /opt/logstash-2.3.1/conf/log_nginx_access.log",
         "received_at" => "2016-04-15T11:46:51.736Z",
       "received_from" => "10.66.3.136",
"syslog_severity_code" => 5,
"syslog_facility_code" => 1,
     "syslog_facility" => "user-level",
     "syslog_severity" => "notice"

}`

ctrl + c退出掸掏，可以放至后臺(tái)運(yùn)行

# nohup logstash -f /opt/logstash-2.3.1/config/log_indexer_syslog_136.conf >nohup &  //放至后臺(tái)運(yùn)行
# vim /etc/rc.local  //設(shè)置開(kāi)機(jī)啟動(dòng)

su -l -c "nohup /opt/logstash/bin/logstash -f /opt/logstash/config/log_indexer_syslog_136.conf >/dev/null 2>&1 &"

六、Monitor Tomcat-catalina Log

6.1 從文件獲取日志

客戶端

# vim /opt/logstash-2.3.1/conf/log_agent_tomcat_catalina_local_250.conf  //定義一個(gè)實(shí)例配置宙帝，從catalina.log獲取日志丧凤，直接在客戶端進(jìn)行日志filter，并存儲(chǔ)至redis步脓，鍵名定義為`tomcat_catalina_local_250:redis`愿待，在filter中用到了replace，即自定義日志類型分類


input {
    file {
        path => "/opt/apache-tomcat-7.0.53/logs/catalina.out"
        type => "tomcat_catalina"
        codec=> multiline {
            pattern => "(^.+[^\[INFO\]]Exception:.+)|(^.+\[ERROR\].+)|(^[a-zA-Z])|(^\s+at .+)|(^\s+... \d+ more)|(^\s*Caused by:.+)"

            #pattern => "(^[a-zA-Z].+)|(^\s+at .+)|(^\s+... \d+ more)|(^\s*Caused by:.+)"
            #匹配這種日志沪编，有的以[開(kāi)頭呼盆，有的直接以數(shù)字開(kāi)頭
            #[04-21 15:42:00,123][DefaultQuartzScheduler_Worker-6][INFO] carsing.crm.customer.service.impl.ServiceNoteWsServiceImpl.queryPeriodFromContract(line:742) CRM<<<<<<<<<Contract:<resultset></resultset>
            #2016-04-21 15:42:15,022 [com.trade.info.impl.InfoPlatformDispatcherImpl:41]-[INFO] ---------線程開(kāi)始提交--------

            #pattern => "(^\s+)|(^=)|(^\d+=\d+)|(^\()|(^[a-zA-Z].+)|(^\s+at .+)|(^\s+... \d+ more)|(^\s*Caused by:.+)"
            #pattern => "(^[^\[])|(^\s+at .+)|(^\s*Caused by:.+)"
            #匹配這種日志，直接以[開(kāi)頭蚁廓，即不是以[開(kāi)頭的都并入下一行
            #[04-21 16:07:55,150][http-bio-8080-exec-837][INFO] carsing.crm.log.InfoInteractionWS.sysCarInfoToWechatService(line:304) >>>>>>>>>sysCarInfoToWechatService:
            #pattern => "(^.+Exception:.+)|(^[a-zA-Z])|(^\s+at .+)|(^\s*Caused by:.+)"

            what=> "previous"    #如果不換行的話放在哪访圃，這里表示放在前面
        }
    }
}

filter {
    if "ERROR" in [message] {    #如果消息里有ERROR字符則將type改為自定義的標(biāo)記
        mutate { replace => { type => "tomcat_catalina_error" } }
    }

    else if "WARN" in [message] {
        mutate { replace => { type => "tomcat_catalina_warn" } }
    }

    else {
        mutate { replace => { type => "tomcat_catalina_info" } }
    }

    grok {
        #match => { "message" => "%{COMBINEDAPACHELOG}" }
        #match => [ "message", "%{TOMCATLOG}", "message", "%{CATALINALOG}" ]
        match => [ "message", "\[%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:%{MINUTE}:(%{SECOND})\]\[(?<thread_name>.+?)\]\[(?<log_level>\w+)\]\s*(?<content>.*)", "message", "%{TIMESTAMP_ISO8601:date} \[(?<thread_name>.+?)\]-\[(?<log_level>\w+)\]\s*(?<content>.*)" ]
        #多種格式匹配，如下
        #[04-21 15:42:00,123][DefaultQuartzScheduler_Worker-6][INFO] carsing.crm.customer.service.impl.ServiceNoteWsServiceImpl.queryPeriodFromContract(line:742) CRM<<<<<<<<<Contract:<resultset></resultset>
        #2016-04-21 15:42:15,022 [com.trade.info.impl.InfoPlatformDispatcherImpl:41]-[INFO] ---------線程開(kāi)始提交--------
        remove_field => ["message"]    #這表示匹配成功后是否刪除原始信息相嵌，這個(gè)看個(gè)人情況腿时，如果為了節(jié)省空間可以考慮刪除
    }
}

output {
    redis {
        host => "10.66.3.155"
        port => "6379"
        data_type => "list"
        key => "tomcat_catalina_local_250:redis"
    }
}

# /opt/logstash-2.3.1/bin/logstash -f /opt/logstash-2.3.1/conf/log_agent_tomcat_catalina_local_250.conf  //啟動(dòng)實(shí)例測(cè)試况脆，正常顯示如下

Settings: Default pipeline workers: 8 Pipeline main started

ctrl + c退出，可以放至后臺(tái)運(yùn)行

# nohup /opt/logstash-2.3.1/bin/logstash -f /opt/logstash-2.3.1/conf/log_agent_tomcat_catalina_local_250.conf >nohup &  //放至后臺(tái)運(yùn)行
# vim /etc/rc.local  //設(shè)置開(kāi)機(jī)啟動(dòng)

su -l -c "nohup /opt/logstash/bin/logstash -f /opt/logstash/config/log_agent_tomcat_catalina_local_250.conf >/dev/null 2>&1 &"

6.2 服務(wù)端從redis中提取相應(yīng)鍵名的數(shù)據(jù)批糟，并錯(cuò)誤日志通過(guò)郵件發(fā)送（msmtp + mutt請(qǐng)另行配置）

服務(wù)端
# vim /opt/logstash-2.3.1/config/log_indexer_tomcat_catalina_local_250.conf //定義實(shí)例格了，從redis中取得鍵名為tomcat_catalina_local_250:redis的數(shù)據(jù)，即取完redis就沒(méi)有數(shù)據(jù)了徽鼎，如果有錯(cuò)誤日志則執(zhí)行mutt命令發(fā)送郵件通知


input {
    redis {
        host => "10.66.3.155"
        port => "6379"
        data_type => "list"
        key => "tomcat_catalina_local_250:redis"
        type => "redis-input"
    }
}

output {
    elasticsearch {
        hosts => "10.66.3.155:9200"
        #user => "root"    #如果安裝了shield并配置了用戶盛末，則加上用戶名及密碼
        #password => "admin1"
        #ssl => true    #如果安裝了shield并在elasticsearch啟用了https，則在這里啟用ssl否淤，并在下行指定證書
        #cacert => "/etc/logstash/ssl/node01.crt"    #指定證書
        index => "tomcat-catalina-local_250_%{+YYYY.MM.dd}"    #索引名稱
    }

    if "ERROR" in [message] {
        exec {
            command => "echo '%{message}' | mutt -s '服務(wù)器%{host} : %{type}日志發(fā)現(xiàn)異常!!!' wangjinhou@carsing.com.cn -c jhw11211@163.com"
        }
    }
    stdout {
        codec => rubydebug
    }
}

# /opt/logstash-2.3.1/bin/logstash -f /opt/logstash-2.3.1/config/log_indexer_tomcat_catalina_local_250.conf  //啟動(dòng)實(shí)例測(cè)試悄但，正常顯示如下，如果有錯(cuò)誤出現(xiàn)還會(huì)發(fā)送郵件

Settings: Default pipeline workers: 8 Pipeline main started { "@timestamp" => "2016-04-17T01:15:00.164Z", "message" => "[04-17 09:15:00,019][DefaultQuartzScheduler_Worker-1][INFO] carsing.crm.customer.service.impl.FollowAssignServiceImpl.automaticAllocation(line:140) automaticAllocation start.....", "@version" => "1", "path" => "/opt/apache-tomcat-7.0.53/logs/catalina.out", "host" => "LO-T-DEMO-AP", "type" => "tomcat_catalina", "tags" => [ [0] "_grokparsefailure" ] }
ctrl + c退出石抡，可以放至后臺(tái)運(yùn)行

# nohup /opt/logstash-2.3.1/bin/logstash -f /opt/logstash-2.3.1/config/log_indexer_tomcat_catalina_local_250.conf >nohup &  //放至后臺(tái)運(yùn)行
# vim /etc/rc.local  //設(shè)置開(kāi)機(jī)啟動(dòng)

su -l -c "nohup /opt/logstash/bin/logstash -f /opt/logstash/config/log_indexer_tomcat_catalina_local_250.conf >/dev/null 2>&1 &"

七檐嚣、kibana Show（可選操作，按個(gè)人需求安裝與否啰扛，安裝方式有變嚎京，請(qǐng)參考官方文檔）

7.1Install Plugin

服務(wù)端
head插件: (以查看集群幾乎所有信息，還能進(jìn)行簡(jiǎn)單的搜索查詢隐解，觀察自動(dòng)恢復(fù)的情況等等鞍帝。)

# /opt/elasticsearch-2.3.1/bin/plugin install mobz/elasticsearch-head

kopf插件:(它提供了一個(gè)簡(jiǎn)單的方法，一個(gè)elasticsearch集群上執(zhí)行常見(jiàn)的任務(wù)厢漩。)

# /opt/elasticsearch-2.3.1/bin/plugin install lmenezes/elasticsearch-kopf/1.6

bigdesk插件: (集群監(jiān)控插件膜眠，通過(guò)該插件可以查看整個(gè)集群的資源消耗情況，cpu溜嗜、內(nèi)存宵膨、http鏈接等等。代碼已許久未更新炸宵，該插件可能已不再支持)

# /opt/elasticsearch-2.3.1/bin/plugin install lukas-vlcek/bigdesk

7.2 Start Elasticsearch

服務(wù)端
上面已經(jīng)啟動(dòng)了辟躏，可以kill掉pid，再重新啟動(dòng)

7.3 Kibana Usage

通過(guò)插件查看集群狀態(tài)
http://10.66.3.155:9200/_plugin/head/

http://10.66.3.155:9200/_plugin/bigdesk/

http://10.66.3.155:9200/_plugin/kopf/

八土全、Redis-browser

該工具用于網(wǎng)頁(yè)在線瀏覽redis中存儲(chǔ)的鍵值對(duì)

8.1 Install Ruby

服務(wù)端

# yum install openssl* openssl-devel zlib-devel gcc gcc-c++ make autoconf readline-devel curl-devel expat-devel gettext-devel

Ruby包淘寶網(wǎng)址：https://ruby.taobao.org/

# wget https://ruby.taobao.org/mirrors/ruby/ruby-2.3.0.tar.gz
# tar xf ruby-2.2.0.tar.gz 
# ./configure --prefix=/opt/ruby 
# make 
# make install

配置gem 鏡像

＃gem sources --remove https://rubygems.org/ 
# gem sources -a https://ruby.taobao.org/ 
# gem sources -l

8.2 Install redis-browser

服務(wù)端

# gem install redis-browser    //如果有錯(cuò)誤百度一下解決

8.3 Start redis-browser

服務(wù)端

# vim /opt/ruby/lib/ruby/gems/2.3.0/gems/redis-browser-0.3.3/config.yml

connections:
  default:
    url: redis://127.0.0.1:6379/0
    auth: password    //如果有密碼的話填入
  production:
    host: mydomain.com
    port: 6666
    db: 1
    auth: password


# redis-browser --config /opt/ruby/lib/ruby/gems/2.3.0/gems/redis-browser-0.3.3/config.yml -B 10.66.3.155    //測(cè)試運(yùn)行

http://10.66.3.155:4567

# vim /etc/rc.local    //加入開(kāi)機(jī)自動(dòng)運(yùn)行

su -l -c "nohup redis-browser --config /opt/ruby/lib/ruby/gems/2.3.0/gems/redis-browser-0.3.3/config.yml -B 10.66.3.155 >/dev/null 2>&1 &"
最后編輯于 ：
?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請(qǐng)聯(lián)系作者
  • 人面猴
    序言：七十年代末捎琐，一起剝皮案震驚了整個(gè)濱河市，隨后出現(xiàn)的幾起案子裹匙，更是在濱河造成了極大的恐慌瑞凑，老刑警劉巖，帶你破解...
    沈念sama閱讀 218,122評(píng)論 6贊 505
  • 死咒
    序言：濱河連續(xù)發(fā)生了三起死亡事件概页，死亡現(xiàn)場(chǎng)離奇詭異籽御，居然都是意外死亡，警方通過(guò)查閱死者的電腦和手機(jī)，發(fā)現(xiàn)死者居然都...
    沈念sama閱讀 93,070評(píng)論 3贊 395
  • 救了他兩次的神仙讓他今天三更去死
    文/潘曉璐 我一進(jìn)店門技掏，熙熙樓的掌柜王于貴愁眉苦臉地迎上來(lái)铃将，“玉大人，你說(shuō)我怎么就攤上這事哑梳【⒀郑” “怎么了？”我有些...
    開(kāi)封第一講書人閱讀 164,491評(píng)論 0贊 354
  • 道士緝兇錄：失蹤的賣姜人
    文/不壞的土叔 我叫張陵鸠真，是天一觀的道長(zhǎng)悯仙。 經(jīng)常有香客問(wèn)我，道長(zhǎng)弧哎，這世上最難降的妖魔是什么雁比？ 我笑而不...
    開(kāi)封第一講書人閱讀 58,636評(píng)論 1贊 293
  • ?港島之戀（遺憾婚禮）
    正文 為了忘掉前任，我火速辦了婚禮撤嫩，結(jié)果婚禮上，老公的妹妹穿的比我還像新娘蠢终。我一直安慰自己序攘，他們只是感情好，可當(dāng)我...
    茶點(diǎn)故事閱讀 67,676評(píng)論 6贊 392
  • 惡毒庶女頂嫁案：這布局不是一般人想出來(lái)的
    文/花漫 我一把揭開(kāi)白布寻拂。 她就那樣靜靜地躺著，像睡著了一般。 火紅的嫁衣襯著肌膚如雪衙四。 梳的紋絲不亂的頭發(fā)上它改，一...
    開(kāi)封第一講書人閱讀 51,541評(píng)論 1贊 305
  • 城市分裂傳說(shuō)
    那天，我揣著相機(jī)與錄音慌核，去河邊找鬼距境。 笑死，一個(gè)胖子當(dāng)著我的面吹牛垮卓，可吹牛的內(nèi)容都是我干的垫桂。 我是一名探鬼主播，決...
    沈念sama閱讀 40,292評(píng)論 3贊 418
  • 雙鴛鴦連環(huán)套：你想象不到人心有多黑
    文/蒼蘭香墨 我猛地睜開(kāi)眼粟按，長(zhǎng)吁一口氣：“原來(lái)是場(chǎng)噩夢(mèng)啊……” “哼诬滩！你這毒婦竟也來(lái)了？” 一聲冷哼從身側(cè)響起灭将，我...
    開(kāi)封第一講書人閱讀 39,211評(píng)論 0贊 276
  • 萬(wàn)榮殺人案實(shí)錄
    序言：老撾萬(wàn)榮一對(duì)情侶失蹤疼鸟，失蹤者是張志新（化名）和其女友劉穎，沒(méi)想到半個(gè)月后庙曙，有當(dāng)?shù)厝嗽跇?shù)林里發(fā)現(xiàn)了一具尸體空镜，經(jīng)...
    沈念sama閱讀 45,655評(píng)論 1贊 314
  • ?護(hù)林員之死
    正文 獨(dú)居荒郊野嶺守林人離奇死亡，尸身上長(zhǎng)有42處帶血的膿包…… 初始之章·張勛 以下內(nèi)容為張勛視角 年9月15日...
    茶點(diǎn)故事閱讀 37,846評(píng)論 3贊 336
  • ?白月光啟示錄
    正文 我和宋清朗相戀三年，在試婚紗的時(shí)候發(fā)現(xiàn)自己被綠了姑裂。 大學(xué)時(shí)的朋友給我發(fā)了我未婚夫和他白月光在一起吃飯的照片馋袜。...
    茶點(diǎn)故事閱讀 39,965評(píng)論 1贊 348
  • 活死人
    序言：一個(gè)原本活蹦亂跳的男人離奇死亡，死狀恐怖舶斧，靈堂內(nèi)的尸體忽然破棺而出欣鳖，到底是詐尸還是另有隱情，我是刑警寧澤茴厉，帶...
    沈念sama閱讀 35,684評(píng)論 5贊 347
  • ?日本核電站爆炸內(nèi)幕
    正文 年R本政府宣布泽台，位于F島的核電站，受9級(jí)特大地震影響矾缓，放射性物質(zhì)發(fā)生泄漏怀酷。R本人自食惡果不足惜，卻給世界環(huán)境...
    茶點(diǎn)故事閱讀 41,295評(píng)論 3贊 329
  • 男人毒藥：我在死后第九天來(lái)索命
    文/蒙蒙 一嗜闻、第九天 我趴在偏房一處隱蔽的房頂上張望蜕依。 院中可真熱鬧，春花似錦琉雳、人聲如沸样眠。這莊子的主人今日做“春日...
    開(kāi)封第一講書人閱讀 31,894評(píng)論 0贊 22
  • 一樁弒父案翠肘，背后竟有這般陰謀
    文/蒼蘭香墨 我抬頭看了看天上的太陽(yáng)檐束。三九已至，卻和暖如春束倍，著一層夾襖步出監(jiān)牢的瞬間被丧，已是汗流浹背。 一陣腳步聲響...
    開(kāi)封第一講書人閱讀 33,012評(píng)論 1贊 269
  • 情欲美人皮
    我被黑心中介騙來(lái)泰國(guó)打工绪妹， 沒(méi)想到剛下飛機(jī)就差點(diǎn)兒被人妖公主榨干…… 1. 我叫王不留甥桂，地道東北人。 一個(gè)月前我還...
    沈念sama閱讀 48,126評(píng)論 3贊 370
  • 代替公主和親
    正文 我出身青樓喂急，卻偏偏與公主長(zhǎng)得像格嘁，于是被迫代替她去往敵國(guó)和親。 傳聞我的和親對(duì)象是個(gè)殘疾皇子廊移，可洞房花燭夜當(dāng)晚...
    茶點(diǎn)故事閱讀 44,914評(píng)論 2贊 355

推薦閱讀更多精彩內(nèi)容