Logstash采集Nginx日志方式

• RPM安裝Logstash采集Nginx日志
• Docker安裝Logstash采集Nginx日志

RPM安裝Logstash采集Nginx日志

• 安裝jdk

  #新建jdk目錄 ~/software/java
  cd ~/software/java
  #下載jdk1.8
  wget http://download.oracle.com/otn-pub/java/jdk/8u181-b13/96a7b8442fe848ef90c96a2fad6ed6d1/jdk-8u181-linux-x64.tar.gz?AuthParam=1534129356_6b3ac55c6a38ba5a54c912855deb6a22
  #解壓
  tar -zxvf jdk-8u181-linux-x64.tar.gz
  #配置JAVA環(huán)境變量
  vi /etc/profile
  #寫入以下內(nèi)容
  #java
  export JAVA_HOME=/root/software/java/jdk1.8.0_181
  export PATH=$JAVA_HOME/bin:$PATH
  export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib
  #加載環(huán)境變量
  source /etc/profile
  

• 驗證jdk

image.png

• 安裝Logstash

  #下載logstash rpm文件【與ES、Kibana版本一致】
  wget https://artifacts.elastic.co/downloads/logstash/logstash-7.8.0.rpm
  #rpm安裝
  rpm -ivh logstash-7.8.0.rpm
  

  etc/logstash/pipelines.yml 為管道配置缚去，添加新的logstash配置文件后要檢查管道中是否關(guān)聯(lián)啟用

• Nginx配置文件

  #user  nobody;
  worker_processes  2;

  #error_log  logs/error.log;
  #error_log  logs/error.log  notice;
  #error_log  logs/error.log  info;

  #pid        logs/nginx.pid;

  events {
      worker_connections  1024;
  }

  http {
      include       mime.types;
      default_type  application/octet-stream;

      log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                  '$status $body_bytes_sent "$http_referer" '
                  '"$http_user_agent" "$http_x_forwarded_for" "$upstream_response_time" "$request_time"';

      #access log json格式配置
      log_format json '{ "@timestamp": "$time_iso8601", '
                     '"time": "$time_iso8601", '
                     '"remote_addr": "$remote_addr", '
                     '"remote_user": "$remote_user", '
                     '"body_bytes_sent": "$body_bytes_sent", '
                     '"request_time": "$request_time", '
                     '"status": "$status", '
                     '"host": "$host", '
                     '"request": "$request", '
                     '"request_method": "$request_method", '
                     '"uri": "$uri", '
                     '"http_referrer": "$http_referer", '
                     '"body_bytes_sent":"$body_bytes_sent", '
                     '"http_x_forwarded_for": "$http_x_forwarded_for", '
                     '"http_user_agent": "$http_user_agent" '
                '}';

      access_log  /var/log/nginx/access.log  main;
      #設(shè)置json格式access log文件路徑
      access_log  /var/log/nginx/access_json.log json;

      sendfile        on;
      #tcp_nopush     on;

      #keepalive_timeout  0;
      keepalive_timeout  65;

      include /etc/nginx/vhost/*.conf;

      #gzip  on;
}

• Logstash配置文件nginx日志采集配置

  cp /etc/logstash/logstash-sample.conf  /etc/logstash/conf.d/nginx-log.conf
  vi nginx-log.conf      
  #寫入以下內(nèi)容
  input {
      file{
          path => "/var/log/nginx/access_json.log"  #需要采集的json格式nginx access日志文件路徑
          codec => json                             #數(shù)據(jù)格式
          #start_position => "beginning"            #默認(rèn)為采集新增內(nèi)容，begining為從文件開頭開始采集
          #type => "nginx_log"                      #類型
      }
  }
  output {
      #es配置
      elasticsearch {
          hosts => "eshost:9200"      #es地址
          index => "logstash-nginx"   #寫入的索引名
          user => elastic             #es用戶
          password => espassword      #es密碼
      }
      #logstash控制臺輸入采集到的數(shù)據(jù)，用于調(diào)試
      #stdout {
      #    codec => rubydebug
      #}
  }
  

• 后臺啟動logstash服務(wù)

  #logstash服務(wù)路徑：/usr/share/logstash/bin/logstash
  #后臺啟動logstash服務(wù)，并將內(nèi)容輸出到/var/log/logstash/output_nginx.log文件下
  nohup /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/nginx-log.conf >> /var/log/logstash/output_nginx.log 2>&1 &
  #查看logstash服務(wù)
  ps -ef | grep logstash
  #關(guān)閉logstash服務(wù)
  kill -9 logstash進(jìn)程號
  

kibana查看配置

• 選擇logstash-nginx索引，創(chuàng)建索引模式

• discover面板查看

  
  
  image.png

Docker安裝Logstash采集Nginx日志

• docker安裝logstash，logstash服務(wù)運(yùn)行在容器內(nèi)，采用file類型只能采集容器內(nèi)存在的文件內(nèi)容鲜锚，無法采集宿主機(jī)文件，需要使用syslog方式采集宿主機(jī)日志

• nginx配置需要開啟syslog推送

  access_log syslog:server=接收主機(jī)ip:514,facility=local7,tag=nginx,severity=info,nohostname main_json;