在使用Spark-Streaming消費Kafka數(shù)據(jù)的時候，使用如下命令提交到yarn

Caused by: java.lang.IllegalArgumentException: Could not find a 'KafkaClient' entry in the JAAS configuration. System property 'java.security.auth.login.config' is not set
    at org.apache.kafka.common.security.JaasContext.defaultContext(JaasContext.java:131)
    at org.apache.kafka.common.security.JaasContext.load(JaasContext.java:96)
    at org.apache.kafka.common.security.JaasContext.load(JaasContext.java:78)
    at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:104)
    at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:61)
    at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:86)
    at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:710)
    ... 17 more

如出現(xiàn)如上異常
配置代碼如下

def main(args: Array[String]) {
    System.setProperty("java.security.krb5.conf", "/etc/krb5.conf")
    System.setProperty("java.security.auth.login.config", "/tmp/kafka_jaas.conf")

/tmp/kafka_jaas.conf

KafkaClient{
  com.sun.security.auth.module.Krb5LoginModule required
  doNotPrompt=true
  useTicketCache=true
  principal="admin/admin@DEMO.com"
  useKeyTab=true
  serviceName="kafka"
  keyTab="/etc/security/keytabs/admin.keytab"
  client=true;
};

提交命令

spark-submit --master yarn \
    --conf spark.yarn.tokens.hbase.enabled=true \
    --deploy-mode client \
    --class com.starsriver.platform.kafka.SparkStreamingKafka \
    --executor-memory 1G \
    --num-executors 3 \
    --executor-cores 2 \
    --keytab /etc/security/keytabs/admin.keytab \
    --principal admin/admin@dounine.com \
    target/demo-1.0.0-SNAPSHOT-jar-with-dependencies.jar > out.log

問題解決

只需要在提交的時候再添加一項配置即可

--conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=/tmp/kafka_jaas.conf"  \

最終正確提交命令如下

spark-submit --master yarn \
    --conf spark.yarn.tokens.hbase.enabled=true \
    --deploy-mode client \
    --class com.starsriver.platform.kafka.SparkStreamingKafka \
    --conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=/tmp/kafka_jaas.conf"  \
    --executor-memory 1G \
    --num-executors 3 \
    --executor-cores 2 \
    --keytab /etc/security/keytabs/admin.keytab \
    --principal admin/admin@dounine.com \
    target/demo-1.0.0-SNAPSHOT-jar-with-dependencies.jar > out.log

描述

經(jīng)過測試以下配置缺一不可

System.setProperty("java.security.auth.login.config", "/tmp/kafka_jaas.conf")

--conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=/tmp/kafka_jaas.conf"

如果大家也遇到上面問題的異常兩句都添加上即可