排查logstash2.4升級(jí)到5.0版本后kafka不兼容問(wèn)題
參考文檔：
/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.0.5/CHANGELOG.md
/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.0.5/DEVELOPER.md
/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.0.5/README.md.md
/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.0.5/lib/logstash/inputs/kafka.rb
/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-kafka-5.0.4/lib/logstash/outputs/kafka.rb
緣由：
之前對(duì)ELKB環(huán)境從2.4版本升級(jí)到最新的5.0穩(wěn)定版本，主要升級(jí)步驟可以參考http://jerrymin.blog.51cto.com/3002256/1870205还栓，后來(lái)發(fā)現(xiàn)kafka集群運(yùn)行報(bào)錯(cuò)簿煌，現(xiàn)在把排查過(guò)程記錄如下违诗，僅供參考
之前環(huán)境：
logstash2.4
logstash-input-kafka-2.0.9
logstash-output-kafka-2.0.5
kafka_2.10-0.8.2.2.tgz
升級(jí)后環(huán)境：
logstash5.0
logstash-input-kafka-2.0.9
logstash-output-kafka-2.0.5
報(bào)錯(cuò)信息：
[2016-11-16T14:35:44,739][ERROR][logstash.inputs.kafka????]?Unknown?setting'zk_connect'forkafka
[2016-11-16T14:35:44,741][ERROR][logstash.inputs.kafka????]?Unknown?setting'topic_id'forkafka
[2016-11-16T14:35:44,741][ERROR][logstash.inputs.kafka????]?Unknown?setting'reset_beginning'forkafka
實(shí)施步驟：
1榆纽，根據(jù)錯(cuò)誤查看程序哪里報(bào)錯(cuò)
grep"Unknown?setting"/usr/share/logstash/-R
/usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb:??????????self.logger.error("Unknown?setting?'#{name}'?for?#{@plugin_name}")
2吗铐，查看程序相關(guān)代碼，發(fā)現(xiàn)需要查看plugins的config定義文件等
def?validate_check_invalid_parameter_names(params)
invalid_params?=?params.keys
#?Filter?out?parameters?that?match?regexp?keys.
#?These?are?defined?in?plugins?like?this:
#???config?/foo.*/?=>?...
@config.each_keydo|config_key|
ifconfig_key.is_a?(Regexp)
invalid_params.reject!?{?|k|?k?=~?config_key?}
elsif?config_key.is_a?(String)
invalid_params.reject!?{?|k|?k?==?config_key?}
end
end
ifinvalid_params.size?>?0
invalid_params.eachdo|name|
self.logger.error("Unknown?setting?'#{name}'?for?#{@plugin_name}")
end
returnfalse
end#?if?invalid_params.size?>?0
returntrue
end#?def?validate_check_invalid_parameter_names
3急鳄，進(jìn)入插件總目錄查看具體信息
cd ?/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.0.5
發(fā)現(xiàn)重點(diǎn)查看如下文件
grepconfig?./*?-R?|awk'{print?$1}'|uniq
./CHANGELOG.md:
./DEVELOPER.md:See
./lib/logstash/inputs/kafka.rb:#
./lib/logstash/inputs/kafka.rb:
./README.md:-
Binary
1）首先看CHANGELOG.md重绷，就有發(fā)現(xiàn)logstash-input-3.0.0.beta1開(kāi)始就不在向后兼容沸停，且剔除了jruby-kafka，注意這里有個(gè)坑2）會(huì)講到昭卓，4.0.0版本說(shuō)開(kāi)始支持kafka 0.9愤钾，5.0.0又說(shuō)開(kāi)始
支持0.10切不向后兼容，這破壞性更新也是夠了候醒∧馨洌看來(lái)問(wèn)題找到了我的kafka版本是kafka_2.10-0.8.2.2.tgz，kafka版本不兼容導(dǎo)致的火焰。
CHANGELOG.md部分文檔如下：
##?5.0.4
-?Update?to?Kafka?version?0.10.0.1forbug?fixes
##?5.0.0
-?SupportforKafka?0.10whichis?not?backward?compatible?with?0.9?broker.
##?4.0.0
-?Republish?all?the?gems?under?jruby.
-?Update?the?plugin?to?the?version?2.0?of?the?plugin?api,?this?change?is?requiredforLogstash?5.0?compatibility.?See?https://github.com/elastic/logstash/issues/5141
-?SupportforKafka?0.9forLS?5.x
##?3.0.0.beta1
-?Refactor?to?use?new?Java?based?consumer,?bypassing?jruby-kafka
-?Breaking:?Change?configuration?to?match?Kafka's?configuration.?This?version?is?not?backward?compatible
2）之前我看DEVELOPER.md文檔時(shí),看配置語(yǔ)法都正確劲装，還以為是卻少依賴(lài)關(guān)系jruby-kafka library呢，這個(gè)再logstash2.x是在用的(另外對(duì)比logstash5.x發(fā)現(xiàn)5版本少了不少插件昌简。另外
kafka版本寫(xiě)的是0.8.1.1占业，看來(lái)這個(gè)DEVELOPER.md沒(méi)有及時(shí)更新（與后面kafka.rb文件不一致），誰(shuí)要是看到了麻煩及時(shí)更新啊纯赎，雖是小問(wèn)題但是也可能誤導(dǎo)我等屁民谦疾。當(dāng)然也有可能是我沒(méi)
有全面看文檔導(dǎo)致的。
DEVELOPER.md文檔結(jié)尾如下：
Dependencies
====================
*?Apache?Kafka?version?0.8.1.1
*?jruby-kafka?library
3）開(kāi)始看README.md文檔犬金，特意看了下kafka的兼容性 看來(lái)logstas-input-kafka5.0.5和logstash-output-kafka5.0.4只能用kafka0.10了念恍。如果你想用Kafka0.9還想用Logstash5.0,你的
logstash-input-kafka和logstash-output-kafka只能降級(jí)版本到4.0.0了，這里都說(shuō)他是中間過(guò)渡版本了晚顷，所以還是隨大流吧峰伙。
##?Kafka?Compatibility
Here's?a?table?that?describes?the?compatibility?matrixforKafka?Broker?support.?Please?remember?that?it?is?good?advice?to?upgrade?brokers?before?consumers/producers
since?brokers?target?backwards?compatibility.?The?0.9?broker?will?work?with?both?the?0.8?consumer?and?0.9?consumer?APIs?but?not?the?other?way?around.
|?Kafka?Broker?Version?|?Logstash?Version?|?Input?Plugin?|?Output?Plugin?|?Why??|
|:---------------:|:------------------:|:--------------:|:---------------:|:------|
|?0.8???????????|?2.0?-?2.x???|?<?3.0.0?|?<3.0.0?|?Legacy,?0.8?is?still?popular?|
|?0.9???????????|?2.0?-?2.3.x???|???3.0.0?|?3.0.0??|?Intermediate?release?before?0.10?that?works?with?old?Ruby?Event?API?`[]`??|
|?0.9??????????|?2.4,?5.0???????????|???4.0.0?|?4.0.0??|?Intermediate?release?before?0.10?with?new?get/setAPI?|
|?0.10?????????|?2.4,?5.0???????????|???5.0.0?|?5.0.0??|?Track?latest?Kafka?release.?Not?compatible?with?0.9?broker?|
4）現(xiàn)在看來(lái)只能升級(jí)kafka版本了。最后我看了下jar-dependencies發(fā)現(xiàn)了kafka-clients-0.10.0.1.jar
ls/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.0.5/vendor/jar-dependencies/runtime-jars/
kafka-clients-0.10.0.1.jar??log4j-1.2.17.jar??lz4-1.3.0.jar??slf4j-api-1.7.21.jar??slf4j-log4j12-1.7.21.jar??snappy-java-1.1.2.6.jar
5）還有一個(gè)文件沒(méi)有看该默，懷著好奇心我看了一眼瞳氓，發(fā)現(xiàn)之前都白費(fèi)力氣了，這里才是最有參考價(jià)值的的主參考文檔啊栓袖，是捷徑啊匣摘，隱藏的夠深的，差點(diǎn)錯(cuò)過(guò)了裹刮，汗音榜！
/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.0.5/lib/logstash/inputs/kafka.rb
kafka.rb部分文檔如下：
#?This?input?will?read?events?from?a?Kafka?topic.?It?uses?the?the?newly?designed
#?0.10?version?of?consumer?API?provided?by?Kafka?to?read?messages?from?the?broker.
#
#?Here's?a?compatibility?matrix?that?shows?the?Kafka?client?versions?that?are?compatible?with?each?combination
#?of?Logstash?and?the?Kafka?input?plugin:
#
#?[options="header"]
#?|==========================================================
#?|Kafka?Client?Version?|Logstash?Version?|Plugin?Version?|Security?Features?|Why?
#?|0.8???????|2.0.0?-?2.x.x???|<3.0.0?|?|Legacy,?0.8?is?still?popular
#?|0.9???????|2.0.0?-?2.3.x???|?3.x.x?|Basic?Auth,?SSL?|Works?with?the?old?Ruby?Event?API?(`event['product']['price']?=?10`)
#?|0.9???????|2.4.0?-?5.0.x???|?4.x.x?|Basic?Auth,?SSL?|Works?with?the?new?getter/setter?APIs?(`event.set('[product][price]',?10)`)
#?|0.10??????|2.4.0?-?5.0.x???|?5.x.x?|Basic?Auth,?SSL?|Not?compatible?with?the?0.9?broker
#?|==========================================================
#
#?NOTE:?We?recommended?that?you?use?matching?Kafka?client?and?broker?versions.?During?upgrades,?you?should
#?upgrade?brokers?before?clients?because?brokers?target?backwards?compatibility.?For?example,?the?0.9?broker
#?is?compatible?with?both?the?0.8?consumer?and?0.9?consumer?APIs,?but?not?the?other?way?around.
6）升級(jí)kafka_2.10-0.8.2.2.tgz為kafka_2.11-0.10.0.1.tgz （我看kafka-clients-0.10.0.1.jar，所以沒(méi)有用最新的kafka_2.11-0.10.1.0.tgz）
大概步驟
關(guān)閉老kafka
/usr/local/kafka/bin/kafka-server-stop.sh /usr/local/kafka/config/server.properties
備份老配置文件
server.properties和zookeeper.properties
刪除kafka
rm -rf /usr/local/kafka/
rm -rf /data/kafkalogs/*
安裝配置新kafka
wget http://mirrors.hust.edu.cn/apache/kafka/0.10.0.1/kafka_2.11-0.10.0.1.tgz
tar zxvf kafka_2.11-0.10.0.1.tgz -C /usr/local/
ln -s /usr/local/kafka_2.11-0.10.0.1 /usr/local/kafka
diff了下server.properties和zookeeper.properties變動(dòng)不大可以直接使用
啟動(dòng)新kafka
/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties &
7)注意幾個(gè)關(guān)鍵配置需要修改
config :bootstrap_servers, :validate => :string, :default => "localhost:9092"
config :group_id, :validate => :string, :default => "logstash"
config :topics, :validate => :array, :default => ["logstash"]
config :consumer_threads, :validate => :number, :default => 1
除了上面的幾個(gè)關(guān)鍵配置外捧弃，kafka的topic分片信息需要重新create一份赠叼，否則KafkaMonitor監(jiān)控不出Active Topic Consumer圖形，但實(shí)際是在工作中。
