上一篇完整介紹elk等的安裝步驟，下面介紹下它們的配置
我們做日志采集的時(shí)候一般步驟如：
image
日志龐大時(shí),filebeat和logstash或者logstash和es之間可以增加kafka或redis
首先來看下它們各自的日志中文解析：
我的版本5.6.16
filebeat.yml
###################### Filebeat Configuration Example #########################
# This file is an example configuration file highlighting only the most common
# options. The filebeat.full.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html
#=========================== Filebeat prospectors =============================
filebeat.prospectors:
# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.
# 指定要監(jiān)控的日志，可以指定具體得文件或者目錄
- input_type: log #輸入filebeat的類型宙攻，包括log(具體路徑的日志)和stdin(鍵盤輸入)兩種陷嘴。
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /var/log/nginx/* #（這是默認(rèn)的）（自行可以修改）
#- c:\programdata\elasticsearch\logs\*
type: "nginx"
fields:
logtype: nginx
# Exclude lines. A list of regular expressions to match. It drops the lines that are
# matching any regular expression from the list.
#exclude_lines: ["^DBG"] #排除正則匹配的日志
# Include lines. A list of regular expressions to match. It exports the lines that are
# matching any regular expression from the list.
#include_lines: ["^ERR", "^WARN"] #收集正則匹配的日志
# Exclude files. A list of regular expressions to match. Filebeat drops the files that
# are matching any regular expression from the list. By default, no files are dropped.
#exclude_files: [".gz$"] #排除正則匹配的文件
# Optional additional fields. These field can be freely picked
# to add additional information to the crawled log files for filtering
# 向輸出的每一條日志添加額外的信息蠢涝，比如“l(fā)evel:debug”乏苦，方便后續(xù)對日志進(jìn)行分組統(tǒng)計(jì)。
# 默認(rèn)情況下护姆，會(huì)在輸出信息的fields子目錄下以指定的新增fields建立子目錄，例如fields.level
# 這個(gè)得意思就是會(huì)在es中多添加一個(gè)字段掏击，格式為 "filelds":{"level":"debug"}
fields:
level: debug
review: 1
### Multiline options
# Mutiline can be used for log messages spanning multiple lines. This is common
# for Java Stack Traces or C-Line Continuation
# The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
# 適用于日志中每一條日志占據(jù)多行的情況卵皂，比如各種語言的報(bào)錯(cuò)信息調(diào)用棧
# 多行日志開始的那一行匹配的pattern
multiline.pattern: ^\[
# Defines if the pattern set under pattern should be negated or not. Default is false.
# 是否需要對pattern條件轉(zhuǎn)置使用，不翻轉(zhuǎn)設(shè)為true砚亭，反轉(zhuǎn)設(shè)置為false灯变。 【建議設(shè)置為true】
multiline.negate: true
# Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
# that was (not) matched before or after or as long as a pattern is not matched based on negate.
# Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
# 匹配pattern后殴玛，與前面（before）還是后面（after）的內(nèi)容合并為一條日志
multiline.match: after
#================================ General =====================================
# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
name: first
# The tags of the shipper are included in their own field with each
# transaction published.
tags: ["service-X", "web-tier"]
# Optional fields that you can specify to add additional information to the
# output.
fields:
env: staging
#================================ Outputs =====================================
# Configure what outputs to use when sending the data collected by the beat.
# Multiple outputs may be used.
#默認(rèn)的輸出到es，配置es的地址端口和ssl信息
#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
# Array of hosts to connect to.
#hosts: ["localhost:9200"]
# Optional protocol and basic auth credentials.
#protocol: "https"
#username: "elastic"
#password: "changeme"
#默認(rèn)的輸出到logstash添祸，配置logstash的地址端口和ssl信息
#----------------------------- Logstash output --------------------------------
output.logstash:
# The Logstash hosts
hosts: ["localhost:5044"]
# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
# Certificate for SSL client authentication
#ssl.certificate: "/etc/pki/client/cert.pem"
# Client Certificate Key
#ssl.key: "/etc/pki/client/cert.key"
#================================ Logging =====================================
# Sets log level. The default log level is info.
# Available log levels are: critical, error, warning, info, debug
logging.level: debug
# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]
logstash 要自己編寫配置文件可以覆蓋yml
logstash-sample.conf
# stdin{
# add_field => {"key" => "value"} #向事件添加一個(gè)字段
# codec => "plain" #默認(rèn)是line, 可通過這個(gè)參數(shù)設(shè)置編碼方式
# tags => ["std"] #添加標(biāo)記
# type => "std" #添加類型
# id => 1 #添加一個(gè)唯一的ID, 如果沒有指定ID, 那么將生成一個(gè)ID
# enable_metric => true #是否開啟記錄日志, 默認(rèn)true
# }
# file{
# path => ["/var/log/nginx/access.log", "/var/log/nginx/error.log"] #處理的文件的路徑, 可以定義多個(gè)路徑
# exclude => "*.zip" #匹配排除
# sincedb_path => "/data/" #sincedb數(shù)據(jù)文件的路徑, 默認(rèn)<path.data>/plugins/inputs/file
# codec => "plain" #默認(rèn)是plain,可通過這個(gè)參數(shù)設(shè)置編碼方式
# tags => ["nginx"] #添加標(biāo)記
# type => "nginx" #添加類型
# discover_interval => 2 #每隔多久去查一次文件, 默認(rèn)15s
# stat_interval => 1 #每隔多久去查一次文件是否被修改過, 默認(rèn)1s
# start_position => "beginning" #從什么位置開始讀取文件數(shù)據(jù), beginning和end,
#默認(rèn)是結(jié)束位置end
# }
# tcp{
# port => 8888 #端口
# mode => "server" #操作模式, server:監(jiān)聽客戶端連接, client:連接到服務(wù)器
# host => "0.0.0.0" #當(dāng)mode為server, 指定監(jiān)聽地址, 當(dāng)mode為client, 指定連接地址,
# 默認(rèn)0.0.0.0
# ssl_enable => false #是否啟用SSL, 默認(rèn)false
# ssl_cert => "" #SSL證書路徑
# ssl_extra_chain_certs => [] #將額外的X509證書添加到證書鏈中
# ssl_key => "" #SSL密鑰路徑
# ssl_key_passphrase => "nil" #SSL密鑰密碼, 默認(rèn)nil
# ssl_verify => true #核實(shí)與CA的SSL連接的另一端的身份
# tcp_keep_alive => false #TCP是否保持alives
# }
# udp{
# buffer_size => 65536 #從網(wǎng)絡(luò)讀取的最大數(shù)據(jù)包大小, 默認(rèn)65536
# host => 0.0.0.0 #監(jiān)聽地址
# port => 8888 #端口
# queue_size => 2000 #在內(nèi)存中保存未處理的UDP數(shù)據(jù)包的數(shù)量, 默認(rèn)2000
# workers => 2 #處理信息包的數(shù)量, 默認(rèn)2
#}
input{
stdin { }
beats {
port => "5044"
codec => plain
{
charset => "GBK" #處理亂碼
}
}
tcp{
host => "localhost"
mode => "server"
port => 1337
}
http{
host => "0.0.0.0"
port => 80
additional_codecs => {"application/json" => "json"}
codec => "plain"
threads => 4
ssl => false
type => "info"
}
}
filter {
if ([message]== "")
{
drop {}
}
}
output{
if [type] == "error" {
elasticsearch {
hosts => [ "127.0.0.1:9200"]
index => "logstash-error"
}}
if [type] == "info" {
elasticsearch {
hosts => [ "127.0.0.1:9200"]
index => "logstash-info"
}}
if [fields][logtype] == "nginx" {
##按照type類型創(chuàng)建多個(gè)索引
elasticsearch {
hosts => ["127.0.0.1:9200"]
index => "nginx_%{+YYYY.MM.dd}"
}
}
stdout{
codec => rubydebug
}
}
elasticsearch.yml
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
#
# Path to log files:
#
#path.logs: /path/to/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 0.0.0.0
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.zen.ping.unicast.hosts: ["127.0.0.1"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
#discovery.zen.minimum_master_nodes: 3
#
# For more information, consult the zen discovery module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
kibana.yml
# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5600
# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "127.0.0.1"
# Enables you to specify a path to mount Kibana at if you are running behind a proxy. This only affects
# the URLs generated by Kibana, your proxy is expected to remove the basePath value before forwarding requests
# to Kibana. This setting cannot end in a slash.
#server.basePath: ""
# The maximum payload size in bytes for incoming server requests.
#server.maxPayloadBytes: 1048576
# The Kibana server's name. This is used for display purposes.
#server.name: "your-hostname"
# The URL of the Elasticsearch instance to use for all your queries.
elasticsearch.url: "http://127.0.0.1:9200"
# When this setting's value is true Kibana uses the hostname specified in the server.host
# setting. When the value of this setting is false, Kibana uses the hostname of the host
# that connects to this Kibana instance.
#elasticsearch.preserveHost: true
# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn't already exist.
#kibana.index: ".kibana"
# The default application to load.
#kibana.defaultAppId: "discover"
# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
#elasticsearch.username: "user"
#elasticsearch.password: "pass"
# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
#server.ssl.enabled: false
#server.ssl.certificate: /path/to/your/server.crt
#server.ssl.key: /path/to/your/server.key
# Optional settings that provide the paths to the PEM-format SSL certificate and key files.
# These files validate that your Elasticsearch backend uses the same key files.
#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key
# Optional setting that enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
#elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]
# To disregard the validity of SSL certificates, change this setting's value to 'none'.
#elasticsearch.ssl.verificationMode: full
# Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
# the elasticsearch.requestTimeout setting.
#elasticsearch.pingTimeout: 1500
# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
#elasticsearch.requestTimeout: 30000
# List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
# headers, set this value to [] (an empty list).
#elasticsearch.requestHeadersWhitelist: [ authorization ]
# Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
# by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.
#elasticsearch.customHeaders: {}
# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
#elasticsearch.shardTimeout: 0
# Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying.
#elasticsearch.startupTimeout: 5000
# Specifies the path where Kibana creates the process ID file.
#pid.file: /var/run/kibana.pid
# Enables you specify a file where Kibana stores log output.
#logging.dest: stdout
# Set the value of this setting to true to suppress all logging output.
#logging.silent: false
# Set the value of this setting to true to suppress all logging output other than error messages.
#logging.quiet: false
# Set the value of this setting to true to log all events, including system usage information
# and all requests.
#logging.verbose: false
# Set the interval in milliseconds to sample system and process performance
# metrics. Minimum is 100ms. Defaults to 5000.
#ops.interval: 5000
# The default locale. This locale can be used in certain circumstances to substitute any missing
# translations.
#i18n.defaultLocale: "en"
上面只是一些配置文件的梳理滚粟，實(shí)際上我們要啟動(dòng)這些 并不需要所有的配置
elasticsearch.yml
$vim /etc/elasticsearch/elasticsearch.yml
path.data: /data/elasticsearch #日志存儲目錄
path.logs: /data/elasticsearch/log #elasticsearch啟動(dòng)日志路徑
network.host: elk1 #這里是主機(jī)IP,我寫了hosts
node.name: "node-2" #節(jié)點(diǎn)名字,不同節(jié)點(diǎn)名字要改為不一樣
http.port: 9200 #api接口url
node.master: true #主節(jié)點(diǎn)
node.data: true #是否存儲數(shù)據(jù)
#手動(dòng)發(fā)現(xiàn)節(jié)點(diǎn),多個(gè)節(jié)點(diǎn)配置
#discovery.zen.ping.unicast.hosts: [elk1, elk2]
kibana.yml
vim /opt/kibana/config/kibana.yml
server.port: 5601
#server.host: "localhost"
server.host: "0.0.0.0"
elasticsearch.url: "http://elk1:9200"
logstash-sample.conf
input{
beats {
port => "5044"
codec => plain
{
charset => "GBK" #處理亂碼
}
}
filter {
}
output{
if [type] == "error" {
elasticsearch {
hosts => [ "127.0.0.1:9200"]
index => "logstash-error"
}}
if [type] == "info" {
elasticsearch {
hosts => [ "127.0.0.1:9200"]
index => "logstash-info"
}}
stdout{
codec => rubydebug
}
}
filebeat自帶了很多moudle,mysql,apache,nginx等，在filebeat.full.yml中刃泌。
filebeat.yml
filebeat.prospectors:
# 指定要監(jiān)控的日志坦刀，可以指定具體得文件或者目錄
- input_type: log #輸入filebeat的類型，包括log(具體路徑的日志)和stdin(鍵盤輸入)兩種蔬咬。
paths:
- /var/log/* #（這是默認(rèn)的）（自行可以修改）
type: "info"
fields:
logtype: info
multiline.pattern: ^\[
multiline.negate: true
multiline.match: after
output.logstash:
hosts: ["localhost:5044"]
歡迎關(guān)注公眾號：麻雀唯伊 鲤遥， 不定時(shí)更新資源文章，生活優(yōu)惠林艘，或許有你想看的
image
