x-pack搭建破解請(qǐng)參考之前的安裝文檔ELK+filebeat+x-pack平臺(tái)搭建

1. 一些命令

• 查詢所有用戶

[elk@elk ~]$  curl -XGET -u elastic 'localhost:9200/_xpack/security/user?pretty'
配置http ssl認(rèn)證后需要使用證書
[elk@elk ~]$  curl -XGET -u elastic 'https://localhost3:9200/_xpack/security/user?pretty' --key client.key --cert client.cer --cacert client-ca.cer -k -v

• 查詢所有roles

[elk@elk ~]$  curl -XGET -u elastic 'localhost:9200/_xpack/security/role?pretty'

• 創(chuàng)建用戶[chengen]

[elk@elk ~]$ curl -X POST -u elastic "localhost:9200/_xpack/security/user/chengen" -H 'Content-Type: application/json' -d'
{
  "password" : "your passwd",
  "roles" : [ "admin"],
  "full_name" : "chengen",
  "email" : "xxxxxx@qq.com",
  "metadata" : {
    "intelligence" : 7
  }
}
'

畫外音：加上“-u elastic”是因?yàn)橹挥衑lastic用戶有管理用戶權(quán)限目锭，另外，請(qǐng)求參數(shù)后面可以帶上?pretty，這樣返回的格式會(huì)好看一點(diǎn)兒

• 修改密碼

curl -X POST -u elastic "localhost:9200/_xpack/security/user/test/_password" -H 'Content-Type: application/json' -d'
{
  "password" : "your passwd"
}
'

修改username為test的密碼容达，注意test是username不是fullname

• 禁用/啟動(dòng)/刪除用戶

curl -X PUT -u elastic "localhost:9200/_xpack/security/user/test/_disable"  
curl -X PUT -u elastic "localhost:9200/_xpack/security/user/test/_enable"
curl -X DELETE -u elastic "localhost:9200/_xpack/security/user/test"

創(chuàng)建beats_admin1的roles案疲，該用戶組對(duì)filebeat有all權(quán)限擦囊，對(duì).kibana有manage旅东，read谣光，index權(quán)限

curl -X POST -u elastic 'localhost:9200/_xpack/security/role/beats_admin1' -H 'Content-Type: application/json' -d '{
  "indices" : [
    {
      "names" : [ "filebeat*" ],
      "privileges" : [ "all" ]
    },
    {
      "names" : [ ".kibana*" ],
      "privileges" : [ "manage", "read", "index" ]
    }
  ]
}'

安全API:https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api.html

2. ElasticSearchHead

當(dāng)你再次打開瀏覽器ElasticSearchHead插件的時(shí)候，會(huì)提示你輸入密碼

image.png

3. kibana再次打開kibana的時(shí)候會(huì)提示你輸入密碼

image.png

image.png

image.png

啟用了X-Pack安全性之后霜大，如果你加載一個(gè)Kibana指示板构哺，該指示板訪問(wèn)你沒(méi)有權(quán)限查看的索引中的數(shù)據(jù)，那么你將得到一個(gè)索引不存在的錯(cuò)誤战坤。X-Pack安全性目前還沒(méi)有提供一種方法來(lái)控制哪些用戶可以加載哪些儀表板曙强。

4.基于角色的權(quán)限控制

該功能的入口在 Management -> Users/Roles。Users 可以方便的管理用戶并且對(duì)其賦予角色途茫，角色和權(quán)限掛鉤碟嘴。Roles 可以方便的管理角色，對(duì)其進(jìn)行賦權(quán)囊卜。Role 是 Permission 的集合娜扇，Permission 是 Privilege 的集合，下面來(lái)說(shuō)說(shuō)權(quán)限：

集群權(quán)限(Cluster Privilege)栅组；
Run As Privileges：可以使得新建角色擁有所選用戶集的權(quán)限雀瓢；
索引權(quán)限(Index Privilege)：
Indices：指定在哪些索引上賦權(quán)；
Privileges：指定賦予哪些權(quán)限玉掸；
Granted Documents Query(可選)：指定在哪些 Query 上賦權(quán)刃麸；
Granted Fields(可選)：指定在哪些 fields 上賦權(quán)；

只將單個(gè)索引的權(quán)限放開給某些用戶

創(chuàng)建role

image.png

授予role的elasticsearch權(quán)限

image.png

image.png

5.x-pack權(quán)限介紹

內(nèi)置賬號(hào)

Security-Roles權(quán)限

集群權(quán)限和索引權(quán)限請(qǐng)參考

https://www.elastic.co/guide/en/elastic-stack-overview/6.7/security-privileges.html#privileges-list-cluster

6. x-pack功能開關(guān)

修改配置后需要重啟三個(gè)服務(wù)

![image.png](https://upload-images.jianshu.io/upload_images/13799508-78a2203d17a6c8af.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240)

 nohup ./bin/logstash -f config/logstash.conf >/dev/null 2>&1 &
./bin/elasticsearch -d
./bin/kibana --verbose > kibana.log 2>&1 &
./filebeat -c filebeat_secure.yml -e -v

7. 對(duì)es集群洲守，kibana疑务，logsatsh沾凄，filebeat配置SSL，TLS和HTTPS安全

注意：

1.創(chuàng)建es證書證書一定要加--dns 和--ip否則后期通信會(huì)報(bào)錯(cuò)知允，如7.5記錄的問(wèn)題撒蟀。
2.如果是集群請(qǐng)將es1中生成的證書elastic-stack-ca.p12拷貝到es2，es3并重新依次生成帶有dns和ip的elastic-certificates.p12證書
3.如果集群安裝x-pack是一定要配置證書加密的温鸽，否則集群無(wú)法正常啟動(dòng)
4.可以參照我下面的步驟進(jìn)行安全證書操作保屯，也可以參考

https://www.elastic.co/cn/blog/configuring-ssl-tls-and-https-to-secure-elasticsearch-kibana-beats-and-logstash#run-filebeat

配置證書，因?yàn)榕渲猛瓴虐l(fā)現(xiàn)了這篇文檔涤垫，還沒(méi)來(lái)得及嘗試配椭，感覺(jué)文章寫的配置證書方法比較簡(jiǎn)單，有興趣同學(xué)可以嘗試一下雹姊。

7.1 es集群安全配置

• elasticsearch-certuti創(chuàng)建證書

bin/elasticsearch-certutil ca    //創(chuàng)建了elastic-stack-ca.p12
bin/elasticsearch-certutil cert --ca config/certs/elastic-stack-ca.p12 --dns 192.168.100.203 --ip 192.168.100.203     //創(chuàng)建了elastic-certificates.p12

config目錄下創(chuàng)建certs文件夾股缸，放入證書elastic-stack-ca.p12和elastic-certificates.p12

• 傳輸通信配置(第一步--集群節(jié)點(diǎn)間的認(rèn)證)

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12

• 內(nèi)置用戶定義密碼(第二步，需要在配置HTTP ssl之前配置吱雏，因?yàn)樵O(shè)置密碼的命令通過(guò)不安全的http與集群通信)

bin/elasticsearch-setup-passwords interactive

• HTTP SSL加密配置(第三步--配置es集群的https)

對(duì)于http通信,Elasticsearch節(jié)點(diǎn)僅用作服務(wù)器敦姻，在elasticsearch.yml文件中指定如下

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.http.ssl.truststore.path: certs/elastic-certificates.p12
xpack.security.http.ssl.client_authentication: optional

綜上，在elasticsearch.yml配置中定義以下內(nèi)容如下：

xpack.security.enabled: true

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.http.ssl.truststore.path: certs/elastic-certificates.p12
xpack.security.http.ssl.client_authentication: optional

更改完畢后需要重啟es集群此時(shí)訪問(wèn)集群需要https認(rèn)證歧杏。此時(shí)elasticsearch head需要配置https://192.168.100.203:9200/

• 報(bào)錯(cuò)：
  如果在elasticsearch.yml中配置了 xpack.security.authc.realms.pki1.type: pki
  啟動(dòng)kibana后镰惦，會(huì)出現(xiàn)登錄kibana地址時(shí)只能使用內(nèi)置賬戶登錄，如elastic犬绒，kibana等旺入，后創(chuàng)建的自定義賬戶test等不可以登錄。
  解決：取消 xpack.security.authc.realms.pki1.type: pki 就恢復(fù)正常

參考: 在Elasticsearch中啟用安全性凯力，TLS / SSL和PKI身份驗(yàn)證的分步指南

7.2 kibana.yml安全配置

• 配置kibana到es的證書通信(生成elastic-stack-ca.pem)

cd /elk/elasticsearch-6.7.0
openssl pkcs12 -in config/certs/elastic-stack-ca.p12 -clcerts -nokeys -chain -out elastic-stack-ca.pem
config目錄下創(chuàng)建certs并將elastic-stack-ca.pem放入

• 配置kibana https訪問(wèn)
  創(chuàng)建kibana ssl證書（生成kibana.key和kibana.crt茵瘾，在certs中新建ssll放入）

openssl req -subj '/CN=192.168.100.203/' -x509 -days $((100 * 365)) -batch -nodes -newkey rsa:2048 -keyout kibana.key -out kibana.crt

綜上，在kibana.yml中配置文件如下:

#注意es地址為https
elasticsearch.hosts: ["https://192.168.100.203:9200"]

xpack.security.enabled: true
elasticsearch.username: "kibana"
elasticsearch.password: "your passwd"

#配置kibana https登錄
server.ssl.enabled: true
server.ssl.certificate: /elk/kibana-6.7.0/config/certs/ssl/kibana.crt
server.ssl.key: /elk/kibana-6.7.0/config/certs/ssl/kibana.key

#kibana與es信任通信
elasticsearch.ssl.certificateAuthorities: config/certs/old/elastic-stack-ca.pem
elasticsearch.ssl.verificationMode: certificate

參考：
ELK的安全加固
kiban與logstash with x-pack
https://www.elastic.co/guide/en/kibana/6.3/configuring-tls.html

7.3 logstash安全配置

• 配置logstash與es集群通信
  將7.2中生成的elastic-stack-ca.pem放入到/elk/logstash-6.7.0/config/certs中
  在logstash中配置

xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: elastic
xpack.monitoring.elasticsearch.password: your passwd
xpack.monitoring.elasticsearch.hosts: ["https://192.168.100.203:9200","https://192.168.100.202:9200","https://192.168.100.201:9200"]

xpack.monitoring.elasticsearch.ssl.certificate_authority: "/elk/logstash-6.7.0/config/certs/elastic-stack-ca.pem"

xpack.monitoring.elasticsearch.ssl.verification_mode: certificate
xpack.monitoring.elasticsearch.sniffing: false
xpack.monitoring.collection.interval: 60s
xpack.monitoring.collection.pipeline.details.enabled: true

• 配置logstash與filebeat之間的通信
  生成zip文件咐鹤，包含instance.crt拗秘，instance.key

bin/elasticsearch-certutil cert --pem -ca config/certs/elastic-stack-ca.p12 --dns 192.168.100.203 --ip 192.168.100.203    ##生成zip文件，包含instance.crt祈惶，instance.key

轉(zhuǎn)換成filebeat可用格式,生成instance.pkcs8.key

openssl pkcs8 -in instance.key -topk8 -nocrypt -out instance.pkcs8.key

logstash-sample.conf配置

input {
  beats {
    port => 5044
    ssl => true
    ssl_key => "/elk/logstash-6.7.0/config/certs/instance/instance.pkcs8.key"
    ssl_certificate => "/elk/logstash-6.7.0/config/certs/instance/instance.crt"
  }
}

output {
  elasticsearch {
    user => "elastic"
    password => "your passwd"
    ssl => true
    ssl_certificate_verification => true
    cacert => "/elk/logstash-6.7.0/config/certs/elastic-stack-ca.pem"
    hosts => ["https://192.168.100.203:9200/","https://192.168.100.202:9200/","https://192.168.100.201:9200/"]
    index => "testlog"

  }
}

output將數(shù)據(jù)加密傳輸?shù)絜s集群雕旨，input對(duì)filebeat傳輸?shù)臄?shù)據(jù)進(jìn)行加密
如果logstash可以正常啟動(dòng)，說(shuō)明配置正確

7.4 filebeat安全配置

將logsatsh中的ca證書：elastic-stack-ca.pem傳輸?shù)絙eat所在的機(jī)器上捧请，如：/opt/filebeat/certs/
修改filebeat.yml文件的output

output.logstash:
  # The Logstash hosts
    hosts: ["192.168.100.203:5044"]
    ssl.certificate_authorities: ["/opt/filebeat/certs/elastic-stack-ca.pem"]

重啟filebeat凡涩，如果數(shù)據(jù)可以在kibana上正常顯示，便是配置正常

7.5 問(wèn)題處理

1 “Host name '192.168.100.203' does not match the certificate subject provided by the pee”
報(bào)錯(cuò)信息

[2019-06-24T13:57:08,032][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"Host name '192.168.100.203' does not match the certificate subject provided by the peer (CN=instance)"}

解決：如果創(chuàng)建證書不添加dns和ip疹蛉，es和logstash通過(guò)TLS通信時(shí)會(huì)此問(wèn)題
PS 證書問(wèn)題讓我很抓狂活箕，配置es集群以及l(fā)ogstash，kibana氧吐，filebest加密用了兩天@@@讹蘑，配置如下：

bin/elasticsearch-certutil cert --ca config/certs/elastic-stack-ca.p12 --dns 192.168.100.203 --ip 192.168.100.203

2 也是證書問(wèn)題末盔，和1類似
因?yàn)閑lk搭建在內(nèi)網(wǎng)需要將外網(wǎng)（IP:1.2.3.4）的日志發(fā)送到elk，使用了frp內(nèi)網(wǎng)穿透進(jìn)行轉(zhuǎn)發(fā)座慰，外網(wǎng)機(jī)器在通過(guò)外網(wǎng)IP+端口訪問(wèn)時(shí)報(bào)錯(cuò)陨舱，如下：

2019-07-01T16:29:17.253+0800  ERROR pipeline/output.go:100  Failed to connect to backoff(async(tcp://1.2.3.4:5044)): x509: certificate is valid for 192.168.100.203, not 118.25.198.208
2019-07-01T16:29:17.253+0800  INFO  pipeline/output.go:93 Attempting to reconnect to backoff(async(tcp://1.2.3.4:5044)) with 1 reconnect attempt(s)

需要在7.3 logstash安全配置，配置logstash與filebeat之間的通信生成zip文件時(shí)版仔，dns和IP加如外網(wǎng)IP

bin/elasticsearch-certutil cert --pem -ca config/certs/elastic-stack-ca.p12 --dns 192.168.100.203,1.2.3.4 --ip 192.168.100.203,1.2.3.4    ##生成zip文件游盲，包含instance.crt，instance.key

接下來(lái)的步驟一致蛮粮，只需要更改logstash中的證書instance.pkcs8.key和instance.crt益缎，代碼如下，filebeat的ca證書不需要更改然想，數(shù)據(jù)就可以正常傳輸過(guò)來(lái)

input {
  beats {
    port => 5044
    ssl => true
    ssl_key => "/elk/logstash-6.7.0/config/certs/instance/instance.pkcs8.key"
    ssl_certificate => "/elk/logstash-6.7.0/config/certs/instance/instance.crt"
  }
}

參考：https://www.elastic.co/guide/en/elasticsearch/reference/6.7/configuring-tls.html#node-certificates
https://discuss.elastic.co/t/certificates-and-keys-for-kibana-and-logstash-with-x-pack/150390
關(guān)注證書的制作方法
關(guān)注證書的制作方法

3 “Could not index event to Elasticsearch”
報(bào)錯(cuò)信息

][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>404, :action=>["index", {:_id=>nil, :_index=>"logstash-2017.07.05", :_type=>"syslog", :_routing=>nil}, 2017-07-05T03:50:10.577Z 10.91.142.103 <179>Jul 5 09:20:10 10.91.126.1 TMNX: 45006214 Base PORT-MINOR-etherAlarmSet-2017 [Port 5/2/4]: Alarm Remote Fault Set], :response=>{"index"=>{"_index"=>"logstash-2017.07.05", "_type"=>"syslog", "_id"=>nil, "status"=>404, "error"=>{"type"=>"index_not_found_exception", "reason"=>"no such index and [action.auto_create_index] ([.security,.monitoring*,.watches,.triggered_watches,.watcher-history*,.ml*]) doesn't match", "index_uuid"=>"na", "index"=>"logstash-2017.07.05"}}}}

解決：
去掉elasticsearch.yml中的限制即可

action.auto_create_index: ".security*,.monitoring*,.watches,.triggered_watches,.watcher-history*,.ml*"

參考：https://discuss.elastic.co/t/index-404-error-in-logstash/91830

4. es搭建集群時(shí)莺奔，需要集群正常，否則會(huì)報(bào)錯(cuò)如下

 [wp-node1] not enough master nodes discovered during pinging (found [[Candidate{node={wp-node1}{MxvGfK7lR6iIwww_frh_BOzcg}{yRKFsh2PT3aEzjI6IwwwG7n1Q}{192.168.100.203}{192.168.100.203:9300}{ml.machine_memory=16603000832, xpack.installed=true, ml.max_open_jobs=20, 

 ][INFO ][o.e.d.z.ZenDiscovery     ] [wp-node3] failed to send join request to master [{wp-node1}{MxvGfK7lR6iI_frh_BOzcg}{7Bc5VHF9QzCs-G2ERoAYaQ}{192.168.100.203}{192.168.100.203:9300}{ml.machine_memory=16603000832, 

[wp-node2] failed to send join request to master [{wp-node1}{MxvGfK7lR6iI_frh_BOzcg}{M_VIMWHFSPmZT9n2D2_fZQ}{192.168.100.203}{192.168.100.203:9300}{ml.machine_memory=16603000832, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}], reason [RemoteTransportException[[wp-node1][192.168.100.203:9300][internal:discovery/zen/join]]; nested: IllegalArgumentException[can't add node {wp-node2}{MxvGfK7lR6iI_frh_BOzcg}{gO2EuagySwq9d5AfDTK6qg}{192.168.1.202}{192.168.1.202:9300}{ml.machine_memory=12411887616, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}, found existing node } with the same id but is a different node instance]; ]

5 此時(shí)curl訪問(wèn)es集群需要使用證書訪問(wèn)如

curl https://192.168.100.203:9200/_xpack/security/_authenticate?pretty \
--key client.key --cert client.cer --cacert client-ca.cer -k -v

訪問(wèn)logstash

curl -v --cacert ca.crt https://192.168.100.203:5044

6.logtstash日志報(bào)錯(cuò)如下：

[2019-07-03T18:18:29,035][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [https://elastic:xxxxxx@192.168.100.203:9200/][Manticore::ConnectTimeout] Read timed out", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>2}
[2019-07-03T18:18:29,035][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [https://elastic:xxxxxx@192.168.100.203:9200/][Manticore::ConnectTimeout] Read timed out", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>2}
[2019-07-03T18:18:29,515][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"https://elastic:xxxxxx@192.168.100.203:9200/"}

原因：配置是對(duì)的变泄，因?yàn)橛行?shù)據(jù)是可以發(fā)到elasticsearch上的令哟，如果發(fā)的數(shù)據(jù)過(guò)多的話會(huì)發(fā)現(xiàn)這個(gè)問(wèn)題

另：參考

http://www.51niux.com/?id=210
https://www.elastic.co/guide/en/beats/filebeat/current/configuring-ssl-logstash.html