ELK 流程圖：
filebeat --> kafka --> logstash --> elasticsearch --> kibana
1.部署kafka
安裝JAVA1.8(略過)?
下載 kafka_2.11-0.10.0.0.tgz
下載地址：http://kafka.apache.org/downloads.html
# yum -y install glibc.i686
# tar zxvf kafka_2.11-0.10.0.0.tgz
# mv kafka_2.11-0.10.0.0 kafka
配置zookeeper
# cd kafka
# [root@yqvm-elk kafka]# cat config/zookeeper.properties |grep -v ^# |grep -v ^$
dataDir=/tmp/zookeeper
clientPort=2181
maxClientCnxns=100
tickTime=2000
initLimit=10
啟動(dòng)zookeeper
# nohup ./bin/zookeeper-server-start.sh config/zookeeper.properties &
配置kafka
# cat config/server.properties |grep -v ^# |grep -v ^$
broker.id=0
listeners=PLAINTEXT://192.168.10.12:9092 ? ? ? ? #本機(jī)IP
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=192.168.10.12:2181 ? ? ? ? ?#本機(jī)IP
zookeeper.connection.timeout.ms=6000
啟動(dòng)kafka
# nohup ./bin/kafka-server-start.sh config/server.properties &
2.部署filebeat
下載 filebeat-5.4.0?
下載地址：https://www.elastic.co/downloads
# rpm -ivh filebeat-5.4.0-x86_64.rpm
配置filebeat
# cat /etc/filebeat/filebeat.yml
filebeat.prospectors:
- input_type: log
paths:
- /home/test/tomcat01/logs/catalina.out
document_type: tomcat01
tail_files: true
- input_type: log
paths:
- /home/test/tomcat02/logs/catalina.out
document_type: tomcat02
tail_files: true
output.kafka:
enabled: true
hosts: ["192.168.10.12:9092"]
topic: test-tomcat
啟動(dòng)filebeat
# service filebeat start
測(cè)試filebeat日志是否輸出到kafka
實(shí)時(shí)查看kafka消費(fèi)
# ./bin/kafka-console-consumer.sh --zookeeper 192.168.10.12:2181 --topic test-tomcat
再新開一個(gè)會(huì)話雳殊，寫點(diǎn)日志
# echo "haha" >> /home/test/tomcat01/logs/catalina.out
返回查看kafka消費(fèi)
# ./bin/kafka-console-consumer.sh --zookeeper 192.168.10.12:2181 --topic test-tomcat
{"@timestamp":"2017-07-19T18:11:05.881Z","beat":{"hostname":"localhost.localdomain","name":"localhost.localdomain","version":"5.4.0"},"input_type":"log","message":"haha","offset":55,"source":"/home/test/tomcat01/logs/catalina.out","type":"tomcat01"}
如果出現(xiàn)上面的內(nèi)容橘沥。說明filebeat 輸出日志到 kafka 成功。
