SASL/SCRAM+ACL實(shí)現(xiàn)動(dòng)態(tài)創(chuàng)建用戶及權(quán)限控制
??研究了一段時(shí)間Kafka的權(quán)限控制. 之前一直看SASL_PLAINTEXT, 結(jié)果發(fā)現(xiàn)這玩意不能動(dòng)態(tài)創(chuàng)建用戶. 今天沒(méi)死心與查了查,發(fā)現(xiàn)這個(gè)人也問(wèn)了這個(gè)問(wèn)題
https://stackoverflow.com/questions/54147460/kafka-adding-sasl-users-dynamically-without-cluster-restart
于是研究了下SCRAM. 寫了這邊完整的文檔
本篇文檔中使用的是自己部署的zookeeper, zookeeper無(wú)需做任何特殊配置
使用SASL / SCRAM進(jìn)行身份驗(yàn)證
請(qǐng)先在不配置任何身份驗(yàn)證的情況下啟動(dòng)Kafka
1. 創(chuàng)建SCRAM Credentials
Kafka中的SCRAM實(shí)現(xiàn)使用Zookeeper作為憑證(credential)存儲(chǔ)杠河。 可以使用kafka-configs.sh在Zookeeper中創(chuàng)建憑據(jù)崖面。 對(duì)于啟用的每個(gè)SCRAM機(jī)制宙攻，必須通過(guò)添加具有機(jī)制名稱的配置來(lái)創(chuàng)建憑證米丘。 必須在啟動(dòng)Kafka broker之前創(chuàng)建代理間通信的憑據(jù)遇伞。 可以動(dòng)態(tài)創(chuàng)建和更新客戶端憑證异雁，并使用更新的憑證來(lái)驗(yàn)證新連接另假。
創(chuàng)建broker建通信用戶(或稱超級(jí)用戶)
bin/kafka-configs.sh --zookeeper 192.168.2.229:2182 --alter --add-config 'SCRAM-SHA-256=[password=admin-secret],SCRAM-SHA-512=[password=admin-secret]' --entity-type users --entity-name admin
創(chuàng)建客戶端用戶fanboshi
bin/kafka-configs.sh --zookeeper 192.168.2.229:2182 --alter --add-config 'SCRAM-SHA-256=[iterations=8192,password=fanboshi],SCRAM-SHA-512=[password=fanboshi]' --entity-type users --entity-name fanboshi
查看SCRAM證書
[root@node002229 kafka]# bin/kafka-configs.sh --zookeeper localhost:2182 --describe --entity-type users --entity-name fanboshi
Configs for user-principal 'fanboshi' are SCRAM-SHA-512=salt=MWwwdWJqcjBncmUwdzY1Mzdoa2NwNXppd3A=,stored_key=mGCJy5k3LrE2gs6Dp4ALRhgy37l1WYPUIdoOncCF+B3Ti3wL2sQNmzg8oEz3tUs9DFsclFCygjbysb0S0BU9bA==,server_key=iTyX0U0Jt02dkddUm6QrVwNf3lJk72dBNs9EDHTqe8kLlNGIp9ypzRkcgkc+WVMd1bkAF3cg8vk9Q1LrJ/2i/A==,iterations=4096,SCRAM-SHA-256=salt=ZDg5MHVlYW40dW9jbXJ6MndvZDVlazd3ag==,stored_key=cgX1ldpXnDL1+TlLHJ3IHn7tAQS/7pQ7BVZUtECpQ3A=,server_key=i7Mcnb5sPUqfIFs6qKWWHZ2ortoKiRc7oabHOV5dawI=,iterations=8192
刪除SCRAM證書
這里只演示,不操作
[root@node002229 kafka]# bin/kafka-configs.sh --zookeeper localhost:2182 --alter --delete-config 'SCRAM-SHA-512' --entity-type users --entity-name fanboshi
2. 配置Kafka Brokers
- 在每個(gè)Kafka broker的config目錄中添加一個(gè)類似下面的JAAS文件础淤，我們稱之為kafka_server_jaas.conf：
[root@node002229 config]# cat kafka_server_jaas.conf
KafkaServer {
org.apache.kafka.common.security.scram.ScramLoginModule required
username="admin"
password="admin-secret";
};
注意不要少寫了分號(hào)
- 將JAAS配置文件位置作為JVM參數(shù)傳遞給每個(gè)Kafka broker：
修改 /usr/local/kafka/bin/kafka-server-start.sh
將exec $base_dir/kafka-run-class.sh $EXTRA_ARGS kafka.Kafka "$@"注釋, 增加下面的內(nèi)容
#exec $base_dir/kafka-run-class.sh $EXTRA_ARGS kafka.Kafka "$@"
exec $base_dir/kafka-run-class.sh $EXTRA_ARGS -Djava.security.auth.login.config=$base_dir/../config/kafka_server_jaas.conf kafka.Kafka "$@"
或者不修改kafka-server-start.sh腳本, 而是將下面的內(nèi)容添加到~/.bashrc
export KAFKA_PLAIN_PARAMS="-Djava.security.auth.login.config=/usr/local/kafka/config/kafka_server_jaas.conf"
export KAFKA_OPTS="$KAFKA_PLAIN_PARAMS $KAFKA_OPTS"
- 如此處所述莹捡，在server.properties中配置SASL端口和SASL機(jī)制鬼吵。 例如：
# 認(rèn)證配置
listeners=SASL_PLAINTEXT://node002229:9092
security.inter.broker.protocol=SASL_PLAINTEXT
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-256
sasl.enabled.mechanisms=SCRAM-SHA-256
# ACL配置
allow.everyone.if.no.acl.found=false
super.users=User:admin
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
再官方文檔中寫的是
listeners=SASL_SSL://host.name:port
security.inter.broker.protocol=SASL_SSL
這里其實(shí)沒(méi)必要寫成SASL_SSL , 我們可以根據(jù)自己的需求選擇SSL或PLAINTEXT, 我這里選擇PLAINTEXT不加密明文傳輸, 省事, 性能也相對(duì)好一些
- 重啟ZK/Kafka
重啟ZK / Kafka服務(wù). 所有broker在連接之前都會(huì)引用'kafka_server_jaas.conf'.
Zookeeper所有節(jié)點(diǎn)
[root@node002229 zookeeper]# zkServer.sh stop /usr/local/zookeeper/bin/../conf/zoo.cfg
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper/bin/../conf/zoo1.cfg
Stopping zookeeper ... STOPPED
[root@node002229 zookeeper]# zkServer.sh start /usr/local/zookeeper/bin/../conf/zoo.cfg
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper/bin/../conf/zoo1.cfg
Kafka所有Broker
cd /usr/local/kafka/;bin/kafka-server-stop.sh
cd /usr/local/kafka/;bin/kafka-server-start.sh -daemon config/server.properties
客戶端配置
先使用kafka-console-producer 和 kafka-console-consumer 測(cè)試一下
kafka-console-producer
- 創(chuàng)建 config/client-sasl.properties 文件
[root@node002229 kafka]# vim config/client-sasl.properties
security.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-256
- 創(chuàng)建config/kafka_client_jaas_admin.conf文件
[root@node002229 kafka]# vim config/kafka_client_jaas_admin.conf
KafkaClient {
org.apache.kafka.common.security.scram.ScramLoginModule required
username="admin"
password="admin-secret";
};
- 修改kafka-console-producer.sh腳本
這里我復(fù)制一份,再改
cp bin/kafka-console-producer.sh bin/kafka-console-producer-admin.sh
vim bin/kafka-console-producer-admin.sh
#exec $(dirname $0)/kafka-run-class.sh kafka.tools.ConsoleProducer "$@"
exec $(dirname $0)/kafka-run-class.sh -Djava.security.auth.login.config=$(dirname $0)/../config/kafka_client_jaas_admin.conf kafka.tools.ConsoleProducer "$@"
- 創(chuàng)建測(cè)試topic
bin/kafka-topics.sh --create --zookeeper localhost:2182 --partitions 1 --replication-factor 1 --topic test
- 測(cè)試生產(chǎn)消息
bin/kafka-console-producer-admin.sh --broker-list 192.168.2.229:9092 --topic test --producer.config config/client-sasl.properties
>1
>
可以看到admin用戶無(wú)需配置ACL就可以生成消息
- 測(cè)試fanboshi用戶
如法炮制, 我們創(chuàng)建一個(gè)bin/kafka-console-producer-fanboshi.sh文件, 只是修改其中的kafka_client_jaas_admin.conf 為 kafka_client_jaas_fanboshi.conf
vim config/kafka_client_jaas_fanboshi.conf
KafkaClient {
org.apache.kafka.common.security.scram.ScramLoginModule required
username="fanboshi"
password="fanboshi";
};
cp bin/kafka-console-producer-admin.sh bin/kafka-console-producer-fanboshi.sh
vi bin/kafka-console-producer-fanboshi.sh
exec $(dirname $0)/kafka-run-class.sh -Djava.security.auth.login.config=$(dirname $0)/../config/kafka_client_jaas_fanboshi.conf kafka.tools.ConsoleProducer "$@"
生產(chǎn)消息
[root@node002229 kafka]# bin/kafka-console-producer-fanboshi.sh --broker-list 192.168.2.229:9092 --topic test --producer.config config/client-sasl.properties
>1
[2019-01-26 18:07:50,099] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 1 : {test=TOPIC_AUTHORIZATION_FAILED} (org.apache.kafka.clients.NetworkClient)
[2019-01-26 18:07:50,100] ERROR Error when sending message to topic test with key: null, value: 1 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [test]
可以看到報(bào)錯(cuò)了, 因?yàn)閒anboshi用戶還沒(méi)有權(quán)限
kafka-console-consumer
- 創(chuàng)建 config/consumer-fanboshi.properties 文件
[root@node002229 kafka]# vim config/consumer-fanboshi.properties
security.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-256
group.id=fanboshi-group
- 創(chuàng)建 bin/kafka-console-consumer-fanboshi.sh 文件
cp bin/kafka-console-consumer.sh bin/kafka-console-consumer-fanboshi.sh
vim bin/kafka-console-consumer-fanboshi.sh
#exec $(dirname $0)/kafka-run-class.sh kafka.tools.ConsoleConsumer "$@"
exec $(dirname $0)/kafka-run-class.sh -Djava.security.auth.login.config=$(dirname $0)/../config/kafka_client_jaas_fanboshi.conf kafka.tools.ConsoleConsumer "$@"
- 測(cè)試消費(fèi)者
bin/kafka-console-consumer-fanboshi.sh --bootstrap-server 192.168.2.229:9092 --topic test --consumer.config config/consumer-fanboshi.properties --from-beginning
其實(shí)也會(huì)報(bào)錯(cuò)的, 報(bào)錯(cuò)內(nèi)容就不貼了
ACL配置
授予fanboshi用戶對(duì)test topic 寫權(quán)限, 只允許 192.168.2.* 網(wǎng)段
bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=localhost:2182 --add --allow-principal User:fanboshi --operation Write --topic test --allow-host 192.168.2.*
授予fanboshi用戶對(duì)test topic 讀權(quán)限, 只允許 192.168.2.* 網(wǎng)段
bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=localhost:2182 --add --allow-principal User:fanboshi --operation Read --topic test --allow-host 192.168.2.*
授予fanboshi用戶, fanboshi-group 消費(fèi)者組 對(duì)test topic 讀權(quán)限, 只允許 192.168.2.* 網(wǎng)段
bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=localhost:2182 --add --allow-principal User:fanboshi --operation Read --group fanboshi-group --allow-host 192.168.2.*
查看acl配置
[root@node002229 kafka]# bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=localhost:2182 --list
Current ACLs for resource `Group:LITERAL:fanboshi-group`:
User:fanboshi has Allow permission for operations: Read from hosts: *
Current ACLs for resource `Topic:LITERAL:test`:
User:fanboshi has Allow permission for operations: Write from hosts: *
User:fanboshi has Allow permission for operations: Read from hosts: *
刪除配置
bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=localhost:2182 --remove --allow-principal User:bob --operation Read --topic example10 --allow-host 192.168.1.158
再次測(cè)試
生產(chǎn)者
[root@node002229 kafka]# bin/kafka-console-producer-fanboshi.sh --broker-list 192.168.2.229:9092 --topic test --producer.config config/client-sasl.properties
>1
[2019-01-26 18:07:50,099] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 1 : {test=TOPIC_AUTHORIZATION_FAILED} (org.apache.kafka.clients.NetworkClient)
[2019-01-26 18:07:50,100] ERROR Error when sending message to topic test with key: null, value: 1 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [test]
>1
消費(fèi)者
[root@node002229 kafka]# bin/kafka-console-consumer-fanboshi.sh --bootstrap-server 192.168.2.229:9092 --topic test --consumer.config config/consumer-fanboshi.properties --from-beginning
1
1
都沒(méi)問(wèn)題了
如何查看我們創(chuàng)建了哪些"用戶"
好像只能去zookeeper看?
zkCli.sh -server node002229:2182
ls /config/users
[admin, alice, fanboshi]
嘗試刪除alice
[root@node002229 kafka]# bin/kafka-configs.sh --zookeeper localhost:2182 --describe --entity-type users --entity-name alice
Configs for user-principal 'alice' are SCRAM-SHA-512=salt=MWt1OHRhZnd3cWZvZ2I4bXcwdTM0czIyaTQ=,stored_key=JYeud1Cx5Z2+FaJgJsZGbMcIi63B9XtA9Wyc+KEm2gXK8+2IxxAVvi1CfSjlkqeupfeIMFJ7/EUkOw+zqvYz6w==,server_key=O4NIgjleroia7puK01/ZZoagFeoxh+zHzckGXXooBsWTdx/7Shb0pMHniMu4IY2jb5orWB2t9K8MZkxCliJDsg==,iterations=4096,SCRAM-SHA-256=salt=MTJ3bXRod3EyN3FtZWdsNHk0NXoyeWdlNjE=,stored_key=chQX35reoBYtfg/U5HBtkzvBAk+gSCgskNzUiScOrUE=,server_key=rRTbUzAehwVMUDTMuoOMumGEuvc7wDecKcqK6yYlbWY=,iterations=8192
[root@node002229 kafka]# bin/kafka-configs.sh --zookeeper localhost:2182 --alter --delete-config 'SCRAM-SHA-512' --entity-type users --entity-name alice
Completed Updating config for entity: user-principal 'alice'.
[root@node002229 kafka]# bin/kafka-configs.sh --zookeeper localhost:2182 --describe --entity-type users --entity-name alice
Configs for user-principal 'alice' are SCRAM-SHA-256=salt=MTJ3bXRod3EyN3FtZWdsNHk0NXoyeWdlNjE=,stored_key=chQX35reoBYtfg/U5HBtkzvBAk+gSCgskNzUiScOrUE=,server_key=rRTbUzAehwVMUDTMuoOMumGEuvc7wDecKcqK6yYlbWY=,iterations=8192
[root@node002229 kafka]# bin/kafka-configs.sh --zookeeper localhost:2182 --alter --delete-config 'SCRAM-SHA-256' --entity-type users --entity-name alice
Completed Updating config for entity: user-principal 'alice'.
[root@node002229 kafka]# bin/kafka-configs.sh --zookeeper localhost:2182 --describe --entity-type users --entity-name alice
Configs for user-principal 'alice' are
去ZK查看
[zk: node002229:2182(CONNECTED) 0] ls /config/users
[admin, alice, fanboshi]
還是有, 難道只能在ZK手動(dòng)刪除?
參考文獻(xiàn)
http://kafka.apache.org/documentation/#security_sasl_scram
https://sharebigdata.wordpress.com/category/kafka/kafka-sasl-scram-with-w-o-ssl/
https://developer.ibm.com/opentech/2017/05/31/kafka-acls-in-practice/
https://developer.ibm.com/tutorials/kafka-authn-authz/
Kafka并不難學(xué)! 入門、進(jìn)階篮赢、商業(yè)實(shí)戰(zhàn)
