環(huán)境
- 系統(tǒng)版本 Linux version 2.6.32-696.3.1.el6.x86_64 (mockbuild@c1bl.rdu2.centos.org) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-18) (GCC) )
- java版本 openjdk version "1.8.0_131"
安裝版本
- ElasticSearch 5.4.1
- Kibana 5.4.1
- Logstash 5.4.1
部署
0x01 確認(rèn)JDK版本及安裝
- 依賴java的最小版本為1.8
yum install java-1.8.0-openjdk
java version
0x02 ElasticSearch安裝
- 下載es奶赠，官網(wǎng)下載路徑：https://www.elastic.co/downloads/elasticsearch
- 解壓
tar xf elasticsearch-5.4.1.tar.gz - 用戶/目錄/權(quán)限設(shè)置
- es運行不能使用root用戶，需要創(chuàng)建新用戶
useradd es
- 新建目錄药有，用于es的數(shù)據(jù)和日志存儲
mkdir -p /data/Logs
mkdir -p /data/Data
- 設(shè)置目錄所有者為es车柠，否則無法訪問（出現(xiàn)Permission denied的錯誤，記得設(shè)置目錄權(quán)限）
chown -R es:es /data/Logs
chown -R es:es /data/Data
chown -R es:es elasticsearch-5.4.1 #注意塑猖，需要設(shè)置權(quán)限
- 修改配置文件config/elasticsearch.yml
- 集群名: cluster.name, 注意: 兩臺機器配置一致
cluster.name: inner_es_cluster
- 節(jié)點名: node.name, 注意: 兩臺機器配置不同, 一臺為01, 另一臺為02
#第一臺機器
node.name: inner_es_node_01
#第二臺機器
node.name: inner_es_node_02
- 數(shù)據(jù)路徑: path.data, 為新建立的目錄
path.data: /data/Data/
- 日志路徑: path.logs
path.logs: /data/Logs
- LockMemory:
bootstrap.mlockall: true
- 本機ip: network.host, 注意兩臺機器配置不同, 分貝配置為對應(yīng)機器的內(nèi)網(wǎng)ip
#第一臺機器
network.host: 10.10.44.202
#第二臺機器
network.host: 10.10.44.203
- 綁定端口竹祷，注意默認(rèn)http.port=9200绵载，transport.tcp.port=9300
#配置Http訪問的端口
http.port: 9200
#配置節(jié)點之間交互的端口
transport.tcp.port: 9300
- Discovery配置: 注意這里是兩臺機器內(nèi)網(wǎng)ip+9300端口,注意這里minimum_master_nodes=2, 見最后一點防腦裂說明
discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.unicast.hosts: ["10.10.44.202:9300", "10.10.44.203:9300"]
- 設(shè)置es占用內(nèi)存
- ES-5.4.1版本申尼，在config/jvm.options中設(shè)置為4G(根據(jù)自身機器配置, 配置的內(nèi)存最大不超過機器物理內(nèi)存的75%. 兩個變量值相等, 以獲取最大的性能). 當(dāng)然, 實際使用中4g可能遠(yuǎn)遠(yuǎn)不夠, 這個值僅是個示例
-Xms4g
-Xmx4g
- 啟用memlock, 提升性能。修改centos配置: /etc/security/limits.conf
加入, 注意, 示例中用戶為es
es soft memlock unlimited
es hard memlock unlimited
設(shè)置完成谭贪，重啟 /etc/init.d/sshd restart
如果結(jié)果是unlimited, 則無需任何處理, 直接進(jìn)入下一步
ulimit -n
unlimited
- ES運行錯誤
- 以用戶es的身份運行
su es
./bin/elasticsearch
#或者
runuser -l es -c "/data/tools/elasticsearch/bin/elasticsearch"
- 問題：max file descriptors
ERROR: [1] bootstrap checks failed
[1]: max file descriptors [65535] for elasticsearch process is too low, increase to at least [65536]
解決：編輯/etc/security/limits.conf, 加入
es soft nofile 204800
es hard nofile 204800
- 問題：max threads descriptors
ERROR: [1] bootstrap checks failed
[1]: max threads descriptors [1024] for elasticsearch threads is too low, increase to at least [2048]
解決：編輯/etc/security/limits.conf, 加入
es soft nproc 2048
es hard nproc 4096
- 問題：system call filters failed to install
ERROR: [1] bootstrap checks failed
[1]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
解決：編輯config/elasticsearch.yml, 加入
bootstrap.system_call_filter: false
- 問題：max virtual memory areas vm.max_map_count
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
解決：編輯/etc/sysctl.conf蜡励，加入
vm.max_map_count=262144
然后執(zhí)行sysctl -p
或者臨時設(shè)置令花，執(zhí)行sysctl -w vm.max_map_count=262144
- 問題：ES_HEAP_SIZE
Error: encountered environment variables that are no longer supported
Use jvm.options or ES_JAVA_OPTS to configure the JVM
ES_HEAP_SIZE=16g: set -Xms16g and -Xmx16g in jvm.options or add "-Xms16g -Xmx16g" to ES_JAVA_OPTS
解決：ElasticSearch-5.4.1版本的內(nèi)存使用阻桅，在config/jvm.options中設(shè)置，如果設(shè)置了
export ES_HEAP_SIZE=4g
export ES_JAVA_OPTS="-Xms4g -Xmx4g -Xss128m"
需要將環(huán)境變量制空
unset ES_HEAP_SIZE
unset ES_JAVA_OPTS
- 啟動測試
- 以用戶es的身份運行
./bin/elasticsearch
#或者
runuser -l es -c "/data/tools/elasticsearch/bin/elasticsearch -d" #-d代表以damon后臺運行
可以看到啟動日志
[2016-06-30 17:20:26,677][WARN ][bootstrap ] unable to install syscall filter: seccomp unavailable: requires kernel 3.5+ with CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER compiled in
[2016-06-30 17:20:27,390][INFO ][node ] [inner_es_node_01] version[2.3.3], pid[6415], build[218bdf1/2016-05-17T15:40:04Z]
[2016-06-30 17:20:27,390][INFO ][node ] [inner_es_node_01] initializing ...
[2016-06-30 17:20:27,948][INFO ][plugins ] [inner_es_node_01] modules [lang-groovy, reindex, lang-expression], plugins [], sites []
[2016-06-30 17:20:27,974][INFO ][env ] [inner_es_node_01] using [1] data paths, mounts [[/data (/dev/xvdb1)]], net usable_space [67.4gb], net total_space [98.4gb], spins? [no], types [ext3]
[2016-06-30 17:20:27,974][INFO ][env ] [inner_es_node_01] heap size [990.7mb], compressed ordinary object pointers [true]
[2016-06-30 17:20:29,926][INFO ][node ] [inner_es_node_01] initialized
[2016-06-30 17:20:29,926][INFO ][node ] [inner_es_node_01] starting ...
[2016-06-30 17:20:30,083][INFO ][transport ] [inner_es_node_01] publish_address {10.0.0.1:9300}, bound_addresses {10.0.0.1:9300}
[2016-06-30 17:20:30,088][INFO ][discovery ] [inner_es_node_01] inner_es_cluster/odmTjZRHRVaa8Zn4vTPcxA
[2016-06-30 17:21:00,091][WARN ][discovery ] [inner_es_node_01] waited for 30s and no initial state was set by the discovery
[2016-06-30 17:21:00,099][INFO ][http ] [inner_es_node_01] publish_address {10.0.0.1:9200}, bound_addresses {10.0.0.1:9200}
[2016-06-30 17:21:00,099][INFO ][node ] [inner_es_node_01] started
等待約一分鐘后, 看到如下日志代表啟動成功
[2016-06-30 17:21:00,099][INFO ][node ] [inner_es_node_01] started
確認(rèn)集群是否啟動成功
curl http://10.10.44.202:9200/
{
"name" : "inner_es_node_01",
"cluster_name" : "inner_es_cluster",
"version" : {
"number" : "2.3.3",
"build_hash" : "218bdf10790eef486ff2c41a3df5cfa32dadcfde",
"build_timestamp" : "2016-05-17T15:40:04Z",
"build_snapshot" : false,
"lucene_version" : "5.5.0"
},
"tagline" : "You Know, for Search"
}
確認(rèn)健康狀態(tài)兼都，如果為red則需要等待一會嫂沉，或者查看具體原因
curl 10.10.44.202:9200/_cluster/health?pretty
{
"cluster_name" : "juzi-log",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 11,
"active_shards" : 22,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
| 顏色 | 意義 |
|---|---|
| green | 所有主要分片和從分片都可用 |
| yellow | 所有主要分片可用，但不是所有從分片都可用 |
| red | 不是所有的主要分片都可用 |
-
ElasticSearch插件的安裝
此過程可以稍后再安裝
- elasticsearch-sql-5.4.1.0
./bin/elasticsearch-plugin install https://github.com/NLPchina/elasticsearch-sql/releases/download/5.4.1.0/elasticsearch-sql-5.4.1.0.zip
一定要根據(jù)es的版本扮碧，確定安裝對應(yīng)插件版本
通過遠(yuǎn)程下載再安裝往往特別慢趟章，甚至安裝失敗，可以下載后本地安裝
./bin/elasticsearch-plugin install file:///path/to/elasticsearch-sql-5.4.1.0.zip
- 安裝完成重新啟動es
es-5.x版本跟1.x/2.x有很大不一樣
On elasticsearch 1.x / 2.x, visit the elasticsearch-sql web front-end:
http://localhost:9200/_plugin/sql/
On elasticsearch 5.x 需要安裝site-server才可以訪問查看頁面
Then start the web front-end like this:
cd site-server
npm install express --save
node node-server.js
5.x中啟動node-server.js后臺慎王，訪問 http://yourHost:8080即可出現(xiàn)通過SQL語句查詢頁面蚓土，端口可在site_configuration.json配置
需要注意的是node-server.js只是提供頁面展示，具體查詢調(diào)用的接口是POST http://localhost:9200/_sql/赖淤，需要在nginx層配置當(dāng)為GET請求時訪問http://yourHost:8080蜀漆，POST請求訪問：http://localhost:9200
參考
ElasticSearch集群部署文檔
elasticsearch-sql
elasticsearch-sql-wiki
ES插件安裝舉例
Elasticsearch: 權(quán)威指南 (有關(guān)于es健康狀態(tài)查詢說明)
Elasticsearch權(quán)威指南(中文版)配置說明
#cluster.name: elasticsearch 配置集群名稱
#node.name: "Franz Kafka" 配置節(jié)點名稱
#node.master: true 配置當(dāng)前節(jié)點是否具有可選為master節(jié)點的資格、
#node.data: true 配置當(dāng)前節(jié)點是否允許存儲數(shù)據(jù)
#node.max_local_storage_nodes: 1 配置每個幾點可以啟動的Elasticsearch最大實例個數(shù)
#index.number_of_shards: 5 配置索引的默認(rèn)分片數(shù)
#index.number_of_replicas: 1 配置索引的默認(rèn)副本數(shù)
#path.conf: /path/to/conf 配置文件存放目錄
#path.data: /path/to/data 配置該節(jié)點索引數(shù)據(jù)的存放目錄咱旱，多個用都好分割
#path.logs: /path/to/logs 配置日志的存放目錄
#path.plugins: /path/to/plugins 配置插件安裝目錄
#network.host: 192.168.0.1 配置節(jié)點綁定的IP和與其他幾點交互的IP
#transport.tcp.port: 9300 配置節(jié)點之間交互的端口
#http.port: 9200 配置Http訪問的端口
#http.max_content_length: 100mb 配置可允許的文本最大長度
#minimum_master_nodes: 2 決定了選主需要的最少節(jié)點數(shù), N/2+1, 兩個節(jié)點即2
0x03 Kibana安裝
- 下載kibana5.4.1版本确丢，下載地址：https://www.elastic.co/downloads/kibana
- 解壓: tar xf kibana-5.4.1
- 修改配置conf/kibana.conf
- 綁定IP+port
server.port: 5601
server.host: "10.10.44.202"
- 連接ElasticSearch地址
elasticsearch.url: "http://10.10.44.202:9200"
- 運行
[root@bogon kibana-5.4.1]# bin/kibana
log [07:08:09.275] [info][status][plugin:kibana@5.4.1] Status changed from uninitialized to green - Ready
log [07:08:09.414] [info][status][plugin:elasticsearch@5.4.1] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [07:08:09.470] [info][status][plugin:console@5.4.1] Status changed from uninitialized to green - Ready
log [07:08:09.515] [info][status][plugin:elasticsearch@5.4.1] Status changed from yellow to green - Kibana index ready
log [07:08:09.516] [info][status][plugin:metrics@5.4.1] Status changed from uninitialized to green - Ready
log [07:08:09.716] [info][status][plugin:timelion@5.4.1] Status changed from uninitialized to green - Ready
log [07:08:09.722] [info][listening] Server running at http://192.168.2.102:5601
log [07:08:09.723] [info][status][ui settings] Status changed from uninitialized to green - Ready
正常運行打印日志，如果啟動失敗會有紅色日志打油孪蕖：Status changed from uninitialized to red
- 訪問
http://localhost:5601
