ELK常見(jiàn)異常

1.內(nèi)存不足

[sandwich@centos-elk bin]$ ./logstash -e 'input {input stdin{}} output{stdout{}}'
Using LS_JAVA_HOME defined java: /home/sandwich/app/elk/elasticsearch-7.17.1/jdk.
WARNING: Using LS_JAVA_HOME while Logstash distribution comes with a bundled JDK.
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000d5550000, 715849728, 0) failed; error='Not enough space' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 715849728 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /home/sandwich/app/elk/logstash-7.17.1/bin/hs_err_pid35687.log

這種錯(cuò)誤會(huì)根據(jù)pid打印到錯(cuò)誤日志里面

[sandwich@centos-elk bin]$ ls -lrt | grep pid
-rw-rw-r--. 1 sandwich sandwich 24253 Apr 23 21:43 hs_err_pid34878.log
-rw-rw-r--. 1 sandwich sandwich 24187 Apr 24 02:59 hs_err_pid35512.log
-rw-rw-r--. 1 sandwich sandwich 24038 Apr 24 03:06 hs_err_pid35687.log

加內(nèi)存就完了

2.Pipelines YAML file is empty

[sandwich@centos-elk bin]$ ./logstash
Using LS_JAVA_HOME defined java: /home/sandwich/app/elk/elasticsearch-7.17.1/jdk.
WARNING: Using LS_JAVA_HOME while Logstash distribution comes with a bundled JDK.
Sending Logstash logs to /home/sandwich/app/elk/logstash-7.17.1/logs which is now configured via log4j2.properties
[2022-04-24T03:30:18,875][INFO ][logstash.runner          ] Log4j configuration path used is: /home/sandwich/app/elk/logstash-7.17.1/config/log4j2.properties
[2022-04-24T03:30:18,898][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.17.1", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 17.0.2+8 on 17.0.2+8 +indy +jit [linux-x86_64]"}
[2022-04-24T03:30:18,902][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED]
ERROR: Pipelines YAML file is empty. Location: /home/sandwich/app/elk/logstash-7.17.1/config/pipelines.yml
usage:
  bin/logstash -f CONFIG_PATH [-t] [-r] [] [-w COUNT] [-l LOG]
  bin/logstash --modules MODULE_NAME [-M "MODULE_NAME.var.PLUGIN_TYPE.PLUGIN_NAME.VARIABLE_NAME=VALUE"] [-t] [-w COUNT] [-l LOG]
  bin/logstash -e CONFIG_STR [-t] [--log.level fatal|error|warn|info|debug|trace] [-w COUNT] [-l LOG]
  bin/logstash -i SHELL [--log.level fatal|error|warn|info|debug|trace]
  bin/logstash -V [--log.level fatal|error|warn|info|debug|trace]
  bin/logstash --help
[2022-04-24T03:30:19,537][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
    at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
    at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
    at home.sandwich.app.elk.logstash_minus_7_dot_17_dot_1.lib.bootstrap.environment.<main>(/home/sandwich/app/elk/logstash-7.17.1/lib/bootstrap/environment.rb:94) ~[?:?]

當(dāng)啟動(dòng)的時(shí)候不加任何參數(shù),默認(rèn)就需要讀logstash-7.17.1/config/pipelines.yml指定的config.
這個(gè)時(shí)候pipelines.yml是空的就會(huì)報(bào)錯(cuò)毒嫡。
如果啟動(dòng)的時(shí)候帶了config相關(guān)的command line options參數(shù)或者已經(jīng)指定了module五辽,它會(huì)直接讀傳過(guò)去的參數(shù)好乐,忽略pipelines.yml


它還會(huì)默認(rèn)創(chuàng)建一個(gè)pipeline id為main的pipeline瘾蛋。
用一個(gè)指定的config file(/home/sandwich/app/elk/logstash-7.17.1/config/logstash.conf)來(lái)啟動(dòng)試試看

nohup /home/sandwich/app/elk/logstash-7.17.1/bin/logstash -f /home/sandwich/app/elk/logstash-7.17.1/config/logstash.conf >> /home/sandwich/app/elk/logstash-7.17.1/startup.log 2>&1 &

也是會(huì)忽略pipelines.yml


3.FATAL Error: [config validation of [elasticsearch].password]: expected value of type [string] but got [number]

這里就是往keystore添加密碼的時(shí)候希望是string格式搁拙,但是輸入的是number格式
這就是純數(shù)字密碼在這里導(dǎo)致的問(wèn)題掺炭,建議把密碼改成帶字符的辫诅。

4.重置密碼報(bào)錯(cuò)(./elasticsearch-setup-passwords interactive)

在我們?cè)O(shè)置了密碼之后,想更新密碼報(bào)錯(cuò)如下

[sandwich@centos-elk bin]$ ./elasticsearch-setup-passwords interactive

Failed to authenticate user 'elastic' against http://192.168.32.3:9200/_security/_authenticate?pretty
Possible causes include:
 * The password for the 'elastic' user has already been changed on this cluster
 * Your elasticsearch node is running against a different keystore
   This tool used the keystore at /home/sandwich/app/elk/elasticsearch-7.17.1/config/elasticsearch.keystore


ERROR: Failed to verify bootstrap password

解決辦法:
確認(rèn)是否有.security-7索引


刪除.security-7索引

然后重新執(zhí)行密碼設(shè)置命令涧狮,不用重啟es集群即可生效炕矮。

5.setting [cluster.initial_master_nodes] is not allowed when [discovery.type] is set to [single-node]

這個(gè)是因?yàn)榕渲昧藛喂?jié)點(diǎn)的時(shí)候跟以下配置有沖突了

cluster.initial_master_nodes: ["es1"]

把這個(gè)注釋掉就好了么夫。

6.IOException[keystore password was incorrect]

啟動(dòng)https后添加如下配置

xpack.security.transport.ssl.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.authc.api_key.enabled: true
xpack.security.http.ssl.keystore.path: elastic-certificates.p12
xpack.security.http.ssl.truststore.path: elastic-certificates.p12

重啟后爆出如下詳細(xì)異常:

ElasticsearchSecurityException[failed to load SSL configuration [xpack.security.http.ssl]]; nested: ElasticsearchException[failed to initialize SSL TrustManager]; nested: IOException[keystore password was incorrect]; nested: UnrecoverableKeyException[failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.];
Likely root cause: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
        at java.base/sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2159)
        at java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:221)
        at java.base/java.security.KeyStore.load(KeyStore.java:1473)

再添加證書(shū)keystore加密的密碼就好了

[sandwich@centos-elk config]$ tail -n 2 elasticsearch.yml
xpack.security.http.ssl.keystore.password: bbbbbb
xpack.security.http.ssl.truststore.password: bbbbbb
?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請(qǐng)聯(lián)系作者
  • 序言:七十年代末,一起剝皮案震驚了整個(gè)濱河市肤视,隨后出現(xiàn)的幾起案子档痪,更是在濱河造成了極大的恐慌,老刑警劉巖邢滑,帶你破解...
    沈念sama閱讀 211,042評(píng)論 6 490
  • 序言:濱河連續(xù)發(fā)生了三起死亡事件腐螟,死亡現(xiàn)場(chǎng)離奇詭異,居然都是意外死亡困后,警方通過(guò)查閱死者的電腦和手機(jī)乐纸,發(fā)現(xiàn)死者居然都...
    沈念sama閱讀 89,996評(píng)論 2 384
  • 文/潘曉璐 我一進(jìn)店門(mén),熙熙樓的掌柜王于貴愁眉苦臉地迎上來(lái)摇予,“玉大人汽绢,你說(shuō)我怎么就攤上這事≈貉危” “怎么了庶喜?”我有些...
    開(kāi)封第一講書(shū)人閱讀 156,674評(píng)論 0 345
  • 文/不壞的土叔 我叫張陵,是天一觀的道長(zhǎng)救鲤。 經(jīng)常有香客問(wèn)我久窟,道長(zhǎng),這世上最難降的妖魔是什么本缠? 我笑而不...
    開(kāi)封第一講書(shū)人閱讀 56,340評(píng)論 1 283
  • 正文 為了忘掉前任斥扛,我火速辦了婚禮,結(jié)果婚禮上丹锹,老公的妹妹穿的比我還像新娘稀颁。我一直安慰自己,他們只是感情好楣黍,可當(dāng)我...
    茶點(diǎn)故事閱讀 65,404評(píng)論 5 384
  • 文/花漫 我一把揭開(kāi)白布匾灶。 她就那樣靜靜地躺著,像睡著了一般租漂。 火紅的嫁衣襯著肌膚如雪阶女。 梳的紋絲不亂的頭發(fā)上,一...
    開(kāi)封第一講書(shū)人閱讀 49,749評(píng)論 1 289
  • 那天哩治,我揣著相機(jī)與錄音秃踩,去河邊找鬼。 笑死业筏,一個(gè)胖子當(dāng)著我的面吹牛憔杨,可吹牛的內(nèi)容都是我干的。 我是一名探鬼主播蒜胖,決...
    沈念sama閱讀 38,902評(píng)論 3 405
  • 文/蒼蘭香墨 我猛地睜開(kāi)眼消别,長(zhǎng)吁一口氣:“原來(lái)是場(chǎng)噩夢(mèng)啊……” “哼抛蚤!你這毒婦竟也來(lái)了?” 一聲冷哼從身側(cè)響起妖啥,我...
    開(kāi)封第一講書(shū)人閱讀 37,662評(píng)論 0 266
  • 序言:老撾萬(wàn)榮一對(duì)情侶失蹤霉颠,失蹤者是張志新(化名)和其女友劉穎对碌,沒(méi)想到半個(gè)月后荆虱,有當(dāng)?shù)厝嗽跇?shù)林里發(fā)現(xiàn)了一具尸體,經(jīng)...
    沈念sama閱讀 44,110評(píng)論 1 303
  • 正文 獨(dú)居荒郊野嶺守林人離奇死亡朽们,尸身上長(zhǎng)有42處帶血的膿包…… 初始之章·張勛 以下內(nèi)容為張勛視角 年9月15日...
    茶點(diǎn)故事閱讀 36,451評(píng)論 2 325
  • 正文 我和宋清朗相戀三年怀读,在試婚紗的時(shí)候發(fā)現(xiàn)自己被綠了。 大學(xué)時(shí)的朋友給我發(fā)了我未婚夫和他白月光在一起吃飯的照片骑脱。...
    茶點(diǎn)故事閱讀 38,577評(píng)論 1 340
  • 序言:一個(gè)原本活蹦亂跳的男人離奇死亡菜枷,死狀恐怖,靈堂內(nèi)的尸體忽然破棺而出叁丧,到底是詐尸還是另有隱情啤誊,我是刑警寧澤,帶...
    沈念sama閱讀 34,258評(píng)論 4 328
  • 正文 年R本政府宣布拥娄,位于F島的核電站蚊锹,受9級(jí)特大地震影響,放射性物質(zhì)發(fā)生泄漏稚瘾。R本人自食惡果不足惜牡昆,卻給世界環(huán)境...
    茶點(diǎn)故事閱讀 39,848評(píng)論 3 312
  • 文/蒙蒙 一、第九天 我趴在偏房一處隱蔽的房頂上張望族壳。 院中可真熱鬧这橙,春花似錦陡厘、人聲如沸。這莊子的主人今日做“春日...
    開(kāi)封第一講書(shū)人閱讀 30,726評(píng)論 0 21
  • 文/蒼蘭香墨 我抬頭看了看天上的太陽(yáng)免糕。三九已至赢乓,卻和暖如春,著一層夾襖步出監(jiān)牢的瞬間说墨,已是汗流浹背骏全。 一陣腳步聲響...
    開(kāi)封第一講書(shū)人閱讀 31,952評(píng)論 1 264
  • 我被黑心中介騙來(lái)泰國(guó)打工, 沒(méi)想到剛下飛機(jī)就差點(diǎn)兒被人妖公主榨干…… 1. 我叫王不留尼斧,地道東北人姜贡。 一個(gè)月前我還...
    沈念sama閱讀 46,271評(píng)論 2 360
  • 正文 我出身青樓,卻偏偏與公主長(zhǎng)得像棺棵,于是被迫代替她去往敵國(guó)和親楼咳。 傳聞我的和親對(duì)象是個(gè)殘疾皇子熄捍,可洞房花燭夜當(dāng)晚...
    茶點(diǎn)故事閱讀 43,452評(píng)論 2 348

推薦閱讀更多精彩內(nèi)容