實(shí)現(xiàn)思路:
1.硬件端日志訪(fǎng)問(wèn)nginx附帶json格式日志
2.后通過(guò)logstash實(shí)時(shí)讀取nginx日志存儲(chǔ)于mysql中
1.nginx安裝和配置
1.1 point-record為自定義nginx服務(wù)名
docker run -p 80:80 --name=point-record -v /opt/point-record/conf.d:/etc/nginx/conf.d -v /opt/point-record/web:/etc/nginx/web -v /opt/nginx-log/point-record/:/var/log/nginx/ -v /etc/timezone:/etc/timezone -v /etc/localtime:/etc/localtime -d nginx
1.2 在/opt/point-record/conf.d創(chuàng)建配置文件point.conf
cd /opt/point-record/conf.d
nano point.conf
point.conf:
log_format json escape=json '{"timestamp":"$time_iso8601",'
'"version":"1",'
'"client":"$remote_addr",'
'"url":"$uri",'
'"status":"$status",'
'"domain":"$host",'
'"host":"$server_addr",'
'"size":$body_bytes_sent,'
'"responsetime":$request_time,'
'"referer":"$http_referer",'
#消息體打印
'"body":$request_body,'
'"ua": "$http_user_agent"'
'}';
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name _;
root /etc/nginx/web;
location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_redirect off;
#必須經(jīng)過(guò)一次轉(zhuǎn)發(fā),否則nginx不會(huì)讀取消息體,打印的request_body為空
proxy_pass $scheme://127.0.0.1:$server_port/success;
access_log /var/log/nginx/access_json.log json;
}
location /success {
return 200;
}
error_page 404 /404.html;
location = /40x.html {
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
}
}
2.logstash安裝與配置
2.1 官網(wǎng)下載 logstash-7.5.1.tar.gz 上傳服務(wù)器并解壓
tar -xzvf logstash-7.5.1.tar.gz -C /usr/local/
cd /usr/local/
#修改文件夾名稱(chēng) ("logstash7" 是為了區(qū)分版本,可以同時(shí)安裝多個(gè)版本)
mv logstash-7.5.1 logstash7
cd logstash7
目錄下文件:
image.png
2.2 logstash原生版本只有input中自帶jdbc插件,這意味著只支持jdbc導(dǎo)入不支持jdbc導(dǎo)出,需要額外安裝網(wǎng)友的自制插件:
2.2.1 logstash配置mysql-connector-java包
MySQL Connector/J是MySQL官方JDBC驅(qū)動(dòng)程序，JDBC（Java Data Base Connectivity,java數(shù)據(jù)庫(kù)連接）是一種用于執(zhí)行SQL語(yǔ)句的Java API，可以為多種關(guān)系數(shù)據(jù)庫(kù)提供統(tǒng)一訪(fǎng)問(wèn)忌傻，它由一組用Java語(yǔ)言編寫(xiě)的類(lèi)和接口組成深纲。
官方下載地址：https://dev.mysql.com/downloads/connector/
下載地址：https://dev.mysql.com/downloads/connector/j/
mkdir -p /usr/local/logstash7/jdbc
cd /usr/local/logstash7/jdbc
# 上傳mysql-connector-java包
rz
tar -xzvf mysql-connector-java-8.0.19.tar.gz
2.2.2 安裝 logstash-output-jdbc插件
更改gem源：
國(guó)外的gem源由于網(wǎng)絡(luò)原因浸卦，從國(guó)內(nèi)訪(fǎng)問(wèn)太慢而且不穩(wěn)定拓型，還經(jīng)常安裝不成功
cd /usr/local/logstash7/
在安裝 目錄下找到Gemfile文件
image.png
編輯該文件將source改為 https://gems.ruby-china.com
image.png
修改完成后運(yùn)行安裝命令
/usr/local/logstash7/bin/logstash-plugin install logstash-output-jdbc
運(yùn)行后輸出Installation successful表示成功
image.png
2.3 配置config文件
在 /usr/local/logstash7/config/目錄下創(chuàng)建nginx.conf文件
input {
file {
path => ["/opt/nginx-log/point-record/access_json.log"]
type => "nginx"
}
}
filter {
mutate {
#先替換掉字符串當(dāng)中的特殊字符在做json轉(zhuǎn)換
gsub =>["message", "\\n", ""]
gsub =>["message", "\\r", ""]
gsub =>["message", "\\t", ""]
gsub =>["message", "[\\]", ""]
}
json {
source => "message"
}
mutate {
gsub =>["timestamp", "T", " "]
gsub =>["timestamp", "\+08:00", ""]
add_field => {"pass_flag"=>"0"}
add_field =>{"appId"=>"%{[body][appID]}"}
add_field =>{"appType"=>"%{[body][appType]}"}
add_field =>{"appVer"=>"%{[body][appVer]}"}
add_field =>{"userID"=>"%{[body][userID]}"}
add_field =>{"sessionID"=>"%{[body][sessionID]}"}
add_field =>{"deviceID"=>"%{[body][deviceID]}"}
add_field =>{"deviceType"=>"%{[body][deviceType]}"}
add_field =>{"OSVer"=>"%{[body][OSVer]}"}
add_field =>{"lang"=>"%{[body][lang]}"}
add_field =>{"deviceInfo_devRes"=>"%{[body][deviceInfo][devRes]}"}
add_field =>{"deviceInfo_auxRes"=>"%{[body][deviceInfo][auxRes]}"}
add_field =>{"sign"=>"%{[body][sign]}"}
add_field =>{"ntime"=>"%{[body][ntime]}"}
add_field =>{"eventId"=>"%{[body][eventId]}"}
add_field =>{"arg1"=>"%{[body][arg1]}"}
add_field =>{"arg2"=>"%{[body][arg2]}"}
add_field =>{"arg3"=>"%{[body][arg3]}"}
add_field =>{"arg4"=>"%{[body][arg4]}"}
add_field =>{"arg5"=>"%{[body][arg5]}"}
add_field =>{"arg6"=>"%{[body][arg6]}"}
add_field =>{"arg7"=>"%{[body][arg7]}"}
add_field =>{"arg8"=>"%{[body][arg8]}"}
add_field =>{"arg9"=>"%{[body][arg9]}"}
add_field =>{"arg10"=>"%{[body][arg10]}"}
}
ruby {
code =>"
salt='XXXXX.com@0723'
appid = event.get('appId')
appType =event.get('appType')
appVer=event.get('appVer')
userID=event.get('userID')
sessionID=event.get('sessionID')
deviceID = event.get('deviceID')
target = '' << appType << appid << appVer << userID << salt << sessionID << deviceID
event.set('md5_value',target)
require 'digest/md5';
md5id = Digest::MD5.hexdigest(target)
event.set('md5id',md5id)
if md5id== event.get('sign')
event.set('pass_flag','1')
end
"
}
mutate {
remove_field =>["body","@timestamp","message"]
}
}
output {
stdout {codec => "rubydebug"}
if [pass_flag] == "1" {
jdbc {
driver_jar_path => "/usr/local/logstash7/jdbc/mysql-connector-java-8.0.19/mysql-connector-java-8.0.19.jar"
driver_class => "com.mysql.jdbc.Driver"
#數(shù)據(jù)庫(kù)連接
connection_string => "jdbc:mysql://rm-bp131frjzi5x1h304.mysql.rds.aliyuncs.com:3306/edu-big-class-test?user=root&password=sairobo@mxxz3rzzz"
#insert語(yǔ)句
statement => [ "insert into point_record (`client_ip`,`svrtime`,`app_type`,`app_id`,`app_ver`,`user_id`,`session_id`,`device_id`,`device_type`,`os_ver`,`lang`,`device_dev_res`,`device_aux_res`,`ntime`,`event_id`,`arg1`,`arg2`,`arg3`,`arg4`,`arg5`,`arg6`,`arg7`,`arg8`,`arg9`,`arg10`) value(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)",
"client","timestamp","appType","appId","appVer","userID","sessionID","deviceID","deviceType","OSVer","lang","deviceInfo_devRes","deviceInfo_auxRes","ntime","eventId","arg1","arg2","arg3","arg4","arg5","arg6","arg7","arg8","arg9","arg10"]
}
}
}
3 啟動(dòng)
前臺(tái)啟動(dòng)命令如下,
/usr/local/logstash7/bin/logstash -f /usr/local/logstash7/config/nginx.conf
結(jié)束: Ctrl+c
后臺(tái)啟動(dòng)
nohup /usr/local/logstash7/bin/logstash -f /usr/local/logstash7/config/nginx.conf >/dev/null &
結(jié)束:
jobs -l
image.png
獲取到pid為28978
kill -9 28978
使用postman模擬測(cè)試
image.png
每次send請(qǐng)求模擬一次硬件端的日志請(qǐng)求
