簡(jiǎn)述
rsyslog 處理多服務(wù)器日志分散不容易管理晦攒，可以將多臺(tái)服務(wù)器日志發(fā)送到指定的日志服務(wù)器上分析气笙。
1. 準(zhǔn)備環(huán)境
2臺(tái)liunx機(jī)器,并且開通514端口
- 3.17.190.209 Server
- 3.21.75.99 Client
tomcat
- apache-tomcat-8.5.60
2.安裝
- 查看rsyslog 版本
-
命令：rsyslogd -vrsyslog-version.png
-
命令：rsyslogd -v
注意版本不同配置格式也不同,但是高版本上面也可以使用低版本的配置
可以使用下面的命令升級(jí)成最新的
cd /etc/yum.repos.d
wget http://rpms.adiscon.com/v8-stable/rsyslog.repo
yum update rsyslog
- 安裝wget
- 下載wget：yum install wget
wget 是liunx下載文件工具
-
安裝jdk
- 下載jdk：wget https://download.oracle.com/otn/java/jdk/8u271-b09/61ae65e088624f5aaa0b1d2d801acb16/jdk-8u271-linux-x64.tar.gz
- 配置jdk環(huán)境
- vi /etc/profile
export JAVA_HOME="/home/user/jdk1.8.0_271"
export CLASSPATH=".:/home/user/jdk1.8.0_271/lib:/home/ec2-user/jdk1.8.0_271/jre/lib"
pathmunge /home/user/jdk1.8.0_271/bin:/home/user/jdk1.8.0_271/jre/bin - source /etc/profile
-
java -version jdk.png
- vi /etc/profile
-
安裝tomcat
- 下載tomcat： wget https://mirrors.bfsu.edu.cn/apache/tomcat/tomcat-8/v8.5.60/bin/apache-tomcat-8.5.60.tar.gz
-
啟動(dòng)tomcat: tomcat.png
3. Client 配置Rsyslog
-
配置rsyslog.conf
- vi /etc/rsyslog.conf
默認(rèn)的路徑：etc/rsyslog.conf
#### MODULES ### module(load="imuxsock" SysSock.Use="off") #imuxsock模塊，支持本地系統(tǒng)日志的模塊 module(load="imjournal" StateFile="imjournal.state") #imjournal模塊醋火，支持對(duì)系統(tǒng)日志的訪問（此模塊與上一模塊默認(rèn)啟用 module(load="imfile" PollingInterval="1") #imfile模塊悠汽，支持對(duì)文件進(jìn)行操作 module(load="imklog") #imklog模塊，支持內(nèi)核日志的模塊 module(load="immark") #immark模塊芥驳，支持日志標(biāo)記 #提供遠(yuǎn)程rsyslog日志的udp協(xié)議的接收支持 module(load="imudp") #imudp模塊柿冲，用于支持udp協(xié)議 input(type="imudp" port="514") #允許通過514端口接收使用udp協(xié)議的遠(yuǎn)程日志 #提供遠(yuǎn)程rsyslog日志的tcp協(xié)議的接收支持 module(load="imtcp") #imtcp模塊，用于支持tcp協(xié)議 input(type="imtcp" port="514") #允許通過514端口接收使用tcp協(xié)議的遠(yuǎn)程日志 #### GLOBAL DIRECTIVES #### global(workDirectory="/var/lib/rsyslog") #工作目錄 module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat") #定義日志格式默認(rèn)模板（可以自行設(shè)定兆旬，參看template部分） include(file="/etc/rsyslog.d/*.conf" mode="optional") #需要引入的自定義配置文件的路徑 #### RULES #### #內(nèi)核消息假抄，默認(rèn)不啟用 #kern.* /dev/console # 記錄所有日志類型的，信息等級(jí)大于等于info級(jí)別的信息到messages文件（mail郵件信息丽猬，authpriv驗(yàn)證信息和corn時(shí)間和任務(wù)信息除外） *.info;mail.none;authpriv.none;cron.none /var/log/messages # authpriv驗(yàn)證相關(guān)的所有信息存放在/var/log/secure authpriv.* /var/log/secure # 郵件的所有信息存在/var/log/maillog宿饱；這里有一個(gè)“-”符號(hào)表示是使用異步的方式記錄 mail.* -/var/log/maillog # 任務(wù)計(jì)劃有關(guān)的信息存放在/var/log/cron cron.* /var/log/cron # 記錄所有的≥emerg級(jí)別信息，發(fā)送給每個(gè)登錄到系統(tǒng)的日志 *.emerg :omusrmsg:* # 記錄uucp脚祟，news.crit等 uucp,news.crit /var/log/spooler # 本地服務(wù)器的啟動(dòng)的所有日志 local7.* /var/log/boot.log #日志發(fā)送的配置谬以，@表示傳輸協(xié)議（@表示udp，@@表示tcp）由桌，后面是ip和端口为黎，格式可配置 #*.* @@3.17.190.209:514 # 定制規(guī)則 ruleset(name="remote"){ action(type="omfwd" #omfwd輸出遠(yuǎn)程模式 target="3.17.190.209" #遠(yuǎn)程服務(wù)器的ip地址 port="514" #端口 protocol="tcp" #使用協(xié)議 queue.type="linkedList" #使用異步處理 queue.spoolDirectory="/var/log/rsyslog" #隊(duì)列目錄 queue.fileName="remoteQueue_3_17_190_209" #隊(duì)列名稱 queue.maxDiskSpace="5g" #隊(duì)列占最大磁盤空間 queue.saveOnShutdown="on" #保存內(nèi)存數(shù)據(jù)如果rsyslog關(guān)閉 action.resumeRetryCount="-1" #無(wú)限重試插入失敗 ) stop } ruleset(name="MyRuleSet") { action(type="omfile" #輸出文件模式 File="/var/log/test.out") #輸出文件的位置 stop } - vi /etc/rsyslog.conf
-
配置tomcat-log.conf
- vi /etc/rsyslog.d/tomcat-log.conf
如果沒有就創(chuàng)建一個(gè)新的文件
#工作目錄 $WorkDirectory /var/log/rsyslog #輸入文件模式 input(type="imfile" File="/var/log/apache-tomcat-8.5.60/logs/catalina.out" #文件的位置 Tag="tomcat_catalina" #文件標(biāo)識(shí)，服務(wù)器接收到消息可以使用這個(gè)tag分類 Severity="info" #日志級(jí)別 Facility="local0" PersistStateInterval="1" #回寫偏移量數(shù)據(jù)到文件間隔時(shí)間(秒) ruleset="remote") #規(guī)則集行您，rsyslog.conf中定義的rule名稱 #測(cè)試從本機(jī)的test.in 寫到本地的test.out文件 input(type="imfile" File="/var/log/test.in" Tag="test1" Severity="info" Facility="local0" ruleset="MyRuleSet") - vi /etc/rsyslog.d/tomcat-log.conf
-
保存文件碍舍，并且檢查配置文件是否正確
-
rsyslogd -N 1檢查文件.png
-
rsyslogd -N 1
-
重啟rsyslog
- systemctl restart rsyslog.service
-
查看啟動(dòng)日志
-
tail -f /var/log/messages
要認(rèn)真查看確保沒有錯(cuò)誤
rsyslog啟動(dòng)日志.png
-
4. Server 配置Rsyslog
- 配置rsyslog.conf
- vi /etc/rsyslog.conf
默認(rèn)的路徑：etc/rsyslog.conf
$FileGroup root #文件所屬組 $FileOwner root #文件的擁有者 $FileCreateMode 0777 #生成文件權(quán)限 $DirCreateMode 0777 #生成文件目錄權(quán)限 $Umask 0022 $PrivDropToUser root #可以刪除日志的用戶 $PrivDropToGroup root #可以刪除日志的用戶組 #### MODULES ### module(load="imuxsock" SysSock.Use="off") #imuxsock模塊，支持本地系統(tǒng)日志的模塊 module(load="imjournal" StateFile="imjournal.state") #imjournal模塊邑雅，支持對(duì)系統(tǒng)日志的訪問（此模塊與上一模塊默認(rèn)啟用 module(load="imfile" PollingInterval="1") #imfile模塊，支持對(duì)文件進(jìn)行操作 module(load="imklog") #imklog模塊妈经，支持內(nèi)核日志的模塊 module(load="immark") #immark模塊淮野，支持日志標(biāo)記 #提供遠(yuǎn)程rsyslog日志的udp協(xié)議的接收支持 module(load="imudp") #imudp模塊，用于支持udp協(xié)議 input(type="imudp" port="514") #允許通過514端口接收使用udp協(xié)議的遠(yuǎn)程日志 #提供遠(yuǎn)程rsyslog日志的tcp協(xié)議的接收支持 module(load="imtcp") #imtcp模塊吹泡，用于支持tcp協(xié)議 input(type="imtcp" port="514") #允許通過514端口接收使用tcp協(xié)議的遠(yuǎn)程日志 #### GLOBAL DIRECTIVES #### global(workDirectory="/var/lib/rsyslog") #工作目錄 module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat") #定義日志格式默認(rèn)模板（可以自行設(shè)定骤星，參看template部分） include(file="/etc/rsyslog.d/*.conf" mode="optional") #需要引入的自定義配置文件的路徑 #### RULES #### #內(nèi)核消息，默認(rèn)不啟用 #kern.* /dev/console # 記錄所有日志類型的爆哑，信息等級(jí)大于等于info級(jí)別的信息到messages文件（mail郵件信息洞难，authpriv驗(yàn)證信息和corn時(shí)間和任務(wù)信息除外） *.info;mail.none;authpriv.none;cron.none /var/log/messages # authpriv驗(yàn)證相關(guān)的所有信息存放在/var/log/secure authpriv.* /var/log/secure # 郵件的所有信息存在/var/log/maillog；這里有一個(gè)“-”符號(hào)表示是使用異步的方式記錄 mail.* -/var/log/maillog # 任務(wù)計(jì)劃有關(guān)的信息存放在/var/log/cron cron.* /var/log/cron # 記錄所有的≥emerg級(jí)別信息揭朝，發(fā)送給每個(gè)登錄到系統(tǒng)的日志 *.emerg :omusrmsg:* # 記錄uucp队贱，news.crit等 uucp,news.crit /var/log/spooler # 本地服務(wù)器的啟動(dòng)的所有日志 local7.* /var/log/boot.log #生成模板 $template SpiceTmpl,"%msg:2:$%\n" $template CatalinaDynaFile,"/var/log/rsyslog/%fromhost-ip%/catalina_%$YEAR%-%$MONTH%-%$DAY%.log" #匹配規(guī)則色冀，文章后面將分享其他配置類型 :fromhost-ip,contains,"3.21.75.99" ?CatalinaDynaFile;SpiceTmpl - vi /etc/rsyslog.conf
- 保存文件，并且重新啟動(dòng)Rsyslog
- systemctl restart rsyslog.conf
- 查看啟動(dòng)日志
- tail -f /var/log/message
- 查看514端口是否開放
如果沒有開放,請(qǐng)關(guān)閉514防火墻
http://tool.chinaz.com/port/scanport.png
5. 測(cè)試寫入local文件
讀取client 文件/var/log/test.in, 寫入client /var/log/test.out 文件
這個(gè)規(guī)則在上面的client 配置里面已經(jīng)配置了輸入文件類型柱嫌，位置和輸出文件類型锋恬，位置。-
查看實(shí)時(shí)日志test.out
-
tail -f /var/log/test.outtest.out.png
-
tail -f /var/log/test.out
客戶端重新開一個(gè)窗口编丘，添加日志到test.in
- echo yanglibo date:
date +%Y-%m-%d:%H:%M:%S>> /var/log/test.in
-
測(cè)試結(jié)果与学，test.out 應(yīng)該同時(shí)接收到文件測(cè)試結(jié)果.png
6. 測(cè)試寫入遠(yuǎn)程server
- 讀取tomat 日志文件 /var/log/apache-tomcat-8.5.60/logs/catalina.out, 寫入server 3.17.190.209 /var/log/rsyslog/3.17.190.209/catalina-2020-12-03.out
這個(gè)輸出路徑是上面server rsyslog.conf 里面 template，filter 已經(jīng)配置好的可以查看上面的配置 - 啟動(dòng)tomcat
- ./var/log/apache-tomcat-8.5.60/bin/startup.sh
- 查看服務(wù)器端生成的文件
- tail -f /var/log/rsyslog/3.17.190.209/catalina-2020-12-03.out
-
結(jié)果start tomcat.png
- echo lbyang date:
date +%Y-%m-%d:%H:%M:%S>> /var/log/apache-tomcat-8.5.60/logs/catalina.out
手動(dòng)寫入文件.png
附錄
1. 登錄服務(wù)器AWS問題
我使用的是AWS使用.pem 證書的方式登錄嘉抓，用戶是來(lái)賓用戶索守。很多權(quán)限都沒有，切換到root 加自定義密碼方式
- sudo passwd root #回車然后設(shè)置密碼
- su root #切換用戶
- vi /etc/ssh/sshd_config #打開文件抑片，修改下面的內(nèi)容為yes
passwordAuthentication yes- systemctl restart sshd.service #重啟服務(wù)
2. 目錄權(quán)限問題
沒有辦法自定義目錄卵佛。比如/home/lb/tomcat/logs/Catalina.out,沒有辦法讀取啟動(dòng)rsyslog報(bào)錯(cuò)，沒有權(quán)限
如果把tomat 安裝到 /var/log/下面就可以讀取3. 參考的網(wǎng)址
- https://blog.csdn.net/weixin_33896726/article/details/92900499?utm_medium=distribute.pc_relevant_t0.none-task-blog-searchFromBaidu-1.control&depth_1-utm_source=distribute.pc_relevant_t0.none-task-blog-searchFromBaidu-1.control
- https://www.cnblogs.com/paul8339/p/12463868.html
- https://blog.csdn.net/weixin_41432553/article/details/93098599
- https://blog.csdn.net/weixin_33704234/article/details/89042131?utm_medium=distribute.pc_relevant.none-task-blog-searchFromBaidu-3.control&depth_1-utm_source=distribute.pc_relevant.none-task-blog-searchFromBaidu-3.control
- https://blog.csdn.net/vantler/article/details/104676128
property ( in English)
Message Properties:
These are extracted by rsyslog parsers from the original message. All message properties start with a letter.
The following message properties exist：
msg
the MSG part of the message (aka “the message” ; ))
rawmsg
the message excactly as it was received from the socket. Should be useful for debugging.
hostname
hostname from the message
source
alias for HOSTNAME
fromhost
hostname of the system the message was received from (in a relay chain, this is the system immediately in front of us and not necessarily the original sender). This is a DNS-resolved name, except if that is not possible or DNS resolution has been disabled.
fromhost-ip
The same as fromhost, but alsways as an IP address. Local inputs (like imklog) use 127.0.0.1 in this property.
syslogtag
TAG from the message
programname
the “static” part of the tag, as defined by BSD syslogd. For example, when TAG is “named[12345]”, programname is “named”.
pri
PRI part of the message - undecoded (single value)
pri-text
the PRI part of the message in a textual form with the numerical PRI appended in brackes (e.g. “l(fā)ocal0.err<133>”)
iut
the monitorware InfoUnitType - used when talking to a MonitorWare backend (also for Adiscon LogAnalyzer)
syslogfacility
the facility from the message - in numerical form
syslogfacility-text
the facility from the message - in text form
syslogseverity
severity from the message - in numerical form
syslogseverity-text
severity from the message - in text form
syslogpriority
an alias for syslogseverity - included for historical reasons (be careful: it still is the severity, not PRI!)
syslogpriority-text
an alias for syslogseverity-text
timegenerated
timestamp when the message was RECEIVED. Always in high resolution
timereported
timestamp from the message. Resolution depends on what was provided in the message (in most cases, only seconds)
timestamp
alias for timereported
protocol-version
The contents of the PROTCOL-VERSION field from IETF draft draft-ietf-syslog-protcol
structured-data
The contents of the STRUCTURED-DATA field from IETF draft draft-ietf-syslog-protocol
app-name
The contents of the APP-NAME field from IETF draft draft-ietf-syslog-protocol
procid
The contents of the PROCID field from IETF draft draft-ietf-syslog-protocol
msgid
The contents of the MSGID field from IETF draft draft-ietf-syslog-protocol
inputname
The name of the input module that generated the message (e.g. “imuxsock”, “imudp”). Note that not all modules necessarily provide this property. If not provided, it is an empty string. Also note that the input module may provide any value of its liking. Most importantly, it is not necessarily the module input name. Internal sources can also provide inputnames. Currently, “rsyslogd” is defined as inputname for messages internally generated by rsyslogd, for example startup and shutdown and error messages. This property is considered useful when trying to filter messages based on where they originated - e.g. locally generated messages (“rsyslogd”, “imuxsock”, “imklog”) should go to a different place than messages generated somewhere.
System Properties:
These properties are provided by the rsyslog core engine. They are not related to the message. All system properties start with a dollar-sign.
For example, timereported contains the timestamp from the message. Depending on how long the message was in the relay chain, this can be quite old. In contrast, $now is the system time when the message is being processed. Depending on your needs, you need one or the other. Usually, the message-based timestamp is the more important one, but that really depdends on the use case.
The following system properties exist:
$bom
The UTF-8 encoded Unicode byte-order mask (BOM). This may be useful in templates for RFC5424 support, when the character set is know to be Unicode.
$now
The current date stamp in the format YYYY-MM-DD
$year
The current year (4-digit)
$month
The current month (2-digit)
$day
The current day of the month (2-digit)
$hour
The current hour in military (24 hour) time (2-digit)
$hhour
The current half hour we are in. From minute 0 to 29, this is always 0 while from 30 to 59 it is always 1.
HHOUR, but values range from 0 to 3 (for the four quater hours that are in each hour)
$minute
The current minute (2-digit)
$myhostname
The name of the current host as it knows itself (probably useful for filtering in a generic way)
