7.12.0 es+kibana+logstash

1. 安裝java環(huán)境【說是Elasticsearch 7.0 開始，內(nèi)置了 Java 環(huán)境檀咙，反正也不麻煩，在裝下吧】

1.1 下載相應(yīng)版本的壓縮包

下載壓縮包可以通過官網(wǎng)下載
下載前需先輸入“uname -r”查看自身Linux的版本嗦枢。

[root@localhost local]# uname -r
3.10.0-957.12.2.el7.x86_64

下載jdk

[root@localhost download]# wget https://download.oracle.com/otn-pub/java/jdk/16.0.1+9/7147401fd7354114ac51ef3e1328291f/jdk-16.0.1_linux-x64_bin.tar.gz

java官網(wǎng)應(yīng)該現(xiàn)在已經(jīng)做了處理攀芯，下載需要攜帶參數(shù)，完整參數(shù)格式如下： https://download.oracle.com/otn-pub/java/jdk/16.0.2+7/d4a915d82b4c4fbb9bde534da945d746/jdk-16.0.2_linux-x64_bin.tar.gz?AuthParam=1630152782_c972da8e1ac38fe0fe40fcdc1fe9575c
因此建議直接去官網(wǎng)找到對(duì)應(yīng)需要的版本下載地址在復(fù)制下來到 linux 去 wget

1.2 解壓縮并移至相應(yīng)目錄

[root@localhost download]# tar -zxvf jdk-16.0.1_linux-x64_bin.tar.gz
...
[root@localhost download]# mv jdk-16.0.1 java
[root@localhost download]# mv java /usr/local/
[root@localhost local]# ll /usr/local/
total 0
drwxr-xr-x.  2 root root   6 Apr 11  2018 bin
drwxr-xr-x.  2 root root   6 Apr 11  2018 etc
drwxr-xr-x.  2 root root   6 Apr 11  2018 games
drwxr-xr-x. 10 root root 272 May  6 15:05 go
drwxr-xr-x.  2 root root   6 Apr 11  2018 include
drwxr-xr-x.  9 root root 107 Jun 17 03:00 java
drwxr-xr-x.  2 root root   6 Apr 11  2018 lib
drwxr-xr-x.  2 root root   6 Apr 11  2018 lib64
drwxr-xr-x.  2 root root   6 Apr 11  2018 libexec
drwxr-xr-x.  4 root root  50 Jun  7 06:49 protoc
drwxr-xr-x.  2 root root   6 Apr 11  2018 sbin
drwxr-xr-x.  5 root root  49 Jun  1  2019 share
drwxr-xr-x.  2 root root   6 Apr 11  2018 src

1.3 配置環(huán)境變量
通過vim /etc/profile命令文虏，打開etc目錄下的環(huán)境變量配置文件profile侣诺，最后面追加如下內(nèi)容

export JAVA_HOME=/usr/local/java
export PATH=$JAVA_HOME/bin:$PATH

通過source /etc/profile命令更新配置文件，并驗(yàn)證java版本

[root@localhost local]# source /etc/profile
[root@localhost local]# java -version
java version "16.0.1" 2021-04-20
Java(TM) SE Runtime Environment (build 16.0.1+9-24)
Java HotSpot(TM) 64-Bit Server VM (build 16.0.1+9-24, mixed mode, sharing)

2. 安裝elasticsearch

2.1 下載

[root@localhost local]# cd /data/download/
[root@localhost download]# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.12.0-linux-x86_64.tar.gz
--2021-06-17 06:34:04--  https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.10.2-linux-x86_64.tar.gz
Resolving artifacts.elastic.co (artifacts.elastic.co)... 34.120.127.130, 2600:1901:0:1d7::
Connecting to artifacts.elastic.co (artifacts.elastic.co)|34.120.127.130|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 318808405 (304M) [application/x-gzip]
Saving to: ‘elasticsearch-7.10.2-linux-x86_64.tar.gz’

100%[=====================================================================================================================================================================================================================================>] 318,808,405 27.9MB/s   in 20s

2021-06-17 06:34:25 (15.1 MB/s) - ‘elasticsearch-7.10.2-linux-x86_64.tar.gz’ saved [318808405/318808405]

[root@localhost download]# ll
total 613084
-rw-r--r--. 1 root root 318808405 Jan 14 13:18 elasticsearch-7.12.0-linux-x86_64.tar.gz
-rw-r--r--. 1 root root 129044044 May  6 18:04 go1.16.4.linux-amd64.tar.gz
-rw-r--r--. 1 root root 178276087 Mar 31 17:39 jdk-16.0.1_linux-x64_bin.tar.gz
-rw-r--r--. 1 root root   1660909 Jun  2 21:06 protoc-3.17.2-linux-x86_64.zip

2.2 解壓并移動(dòng)到指定目錄

[root@localhost download]# tar -zxvf elasticsearch-7.12.0-linux-x86_64.tar.gz
[root@localhost download]# mv elasticsearch-7.12.0 /usr/local/elasticsearch
[root@localhost download]# cd /usr/local/elasticsearch/
[root@localhost elasticsearch]# ll
total 560
drwxr-xr-x.  2 root root   4096 Mar 18 14:21 bin
drwxr-xr-x.  3 root root    169 Jun 17 20:57 config
drwxr-xr-x.  9 root root    107 Mar 18 14:21 jdk
drwxr-xr-x.  3 root root   4096 Mar 18 14:21 lib
-rw-r--r--.  1 root root   3860 Mar 18 14:15 LICENSE.txt
drwxr-xr-x.  2 root root      6 Mar 18 14:19 logs
drwxr-xr-x. 60 root root   4096 Mar 18 14:22 modules
-rw-r--r--.  1 root root 545323 Mar 18 14:19 NOTICE.txt
drwxr-xr-x.  2 root root      6 Mar 18 14:19 plugins
-rw-r--r--.  1 root root   7263 Mar 18 14:14 README.asciidoc

Elasticsearch 目錄氧秘、配置文件說明年鸳、注意點(diǎn)，這里大概解答下各個(gè)目錄丸相、配置文件的作用:

目錄	配置文件	描述
bin		放置腳本文件搔确，如啟動(dòng)腳本 elasticsearch, 插件安裝腳本等。
config	elasticserch.yml	elasticsearch 配置文件灭忠，如集群配置膳算、jvm 配置等。
jdk		java 運(yùn)行環(huán)境
data	path.data	數(shù)據(jù)持久化文件
lib		依賴的相關(guān)類庫
logs	path.log	日志文件
modules		包含的所有 ES 模塊
plugins		包含的所有已安裝的插件

注意點(diǎn):

有些童鞋的機(jī)器內(nèi)存可能不夠弛作，就需要修改 JVM 參數(shù)涕蜂，配置文件路徑為 config/jvm.options，ES V7.1 版本默認(rèn)為 1g, 老版本為2g, 你可以自行修改映琳。
Xmx 和Xms 數(shù)值請(qǐng)?jiān)O(shè)置相同机隙；
Xmx 不要超過機(jī)器內(nèi)存的 50%；
內(nèi)存總量不要超過 30GB, 參見官方文檔 https://www.elastic.co/cn/blog/a-heap-of-trouble萨西；

2.3 創(chuàng)建用戶并修改文件所屬用戶用戶組

es用戶密碼設(shè)置為 xxxxxx

[root@localhost elasticsearch]# cd ..
[root@localhost local]# useradd es
[root@localhost local]# passwd es
Changing password for user es.
New password:
BAD PASSWORD: The password contains the user name in some form
Retype new password:
passwd: all authentication tokens updated successfully.
[root@localhost local]# chown -R es:es elasticsearch/
[root@localhost local]# ll
total 0
total 4
drwxr-xr-x.  2 root root    89 Jun 17 16:13 bin
drwxr-xr-x.  9 es   es     155 Mar 18 14:21 elasticsearch
drwxr-xr-x.  2 root root     6 Apr 11  2018 etc
drwxr-xr-x.  6 root root    56 Jun 17 15:54 freetype
drwxr-xr-x.  2 root root     6 Apr 11  2018 games
drwxr-xr-x. 10 root root   272 May  6 23:05 go
drwxr-xr-x.  3 root root   122 Jun 17 15:53 include
drwxr-xr-x.  9 root root   107 Jun 17 11:00 java
drwxr-xr-x.  3 root root  4096 Jun 17 15:53 lib
drwxr-xr-x.  2 root root     6 Apr 11  2018 lib64
drwxr-xr-x.  2 root root     6 Apr 11  2018 libexec
drwxr-xr-x.  3 root root    18 Jun 17 15:53 man
drwxr-xr-x. 14 root mysql  216 Jun 17 16:05 mysql
drwxr-xr-x. 11 root root   151 Jun 17 16:15 nginx
drwxr-xr-x. 10 root root   102 Jun 17 16:13 php
drwxr-xr-x.  4 root root    50 Jun  7 14:49 protoc
drwxr-xr-x.  2 root root     6 Apr 11  2018 sbin
drwxr-xr-x.  8 root root    89 Jun 17 15:53 share
drwxr-xr-x.  2 root root     6 Apr 11  2018 src

2.4 啟動(dòng)es

配置外網(wǎng)訪問

[es@localhost elasticsearch]$ vim config/elasticsearch.yml

修改配置如下：

......
node.name: node-1
......
network.host: 192.168.33.13
......
http.port: 9200
......
cluster.initial_master_nodes: ["node-1"]
......

測(cè)試啟動(dòng)

[root@localhost elasticsearch]# su es
[es@localhost elasticsearch]$ bin/elasticsearch

網(wǎng)頁頁面訪問 http://localhost:9200/?pretty 返回如下則啟動(dòng)成功：

{
  "name" : "node-1",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "9DHKf7XhTXOKZh1EQ5tbUw",
  "version" : {
    "number" : "7.12.0",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "78722783c38caa25a70982b5b042074cde5d3b3a",
    "build_date" : "2021-03-18T06:17:15.410153305Z",
    "build_snapshot" : false,
    "lucene_version" : "8.8.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"

3. 安裝kibana

3.1下載

[root@localhost download]# wget https://artifacts.elastic.co/downloads/kibana/kibana-7.12.0-linux-x86_64.tar.gz
--2021-06-17 21:21:10--  https://artifacts.elastic.co/downloads/kibana/kibana-7.12.0-linux-x86_64.tar.gz
Resolving artifacts.elastic.co (artifacts.elastic.co)... 34.120.127.130, 2600:1901:0:1d7::
Connecting to artifacts.elastic.co (artifacts.elastic.co)|34.120.127.130|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 281022115 (268M) [application/x-gzip]
Saving to: ‘kibana-7.12.0-linux-x86_64.tar.gz’

100%[=====================================================================================================================================================================================================================================>] 281,022,115 16.5MB/s   in 23s

2021-06-17 21:21:34 (11.6 MB/s) - ‘kibana-7.12.0-linux-x86_64.tar.gz’ saved [281022115/281022115]

3.2 解壓并移動(dòng)到指定目錄

[root@localhost download]# tar -zxvf kibana-7.12.0-linux-x86_64.tar.gz
[root@localhost download]# mv kibana-7.12.0-linux-x86_64 /usr/local/kibana
[root@localhost download]# cd /usr/local/kibana/
[root@localhost kibana]# ll
total 1448
drwxr-xr-x   2 root root      94 Mar 18 13:56 bin
drwxr-xr-x   2 root root      44 Mar 18 13:55 config
drwxr-xr-x   2 root root       6 Mar 18 13:55 data
-rw-r--r--   1 root root    3860 Mar 18 13:55 LICENSE.txt
drwxr-xr-x   6 root root     108 Mar 18 13:55 node
drwxr-xr-x 831 root root   24576 Mar 18 13:55 node_modules
-rw-r--r--   1 root root 1428396 Mar 18 13:55 NOTICE.txt
-rw-r--r--   1 root root     740 Mar 18 13:55 package.json
drwxr-xr-x   2 root root       6 Mar 18 13:55 plugins
-rw-r--r--   1 root root    3968 Mar 18 13:55 README.txt
drwxr-xr-x  12 root root     187 Mar 18 13:55 src
drwxr-xr-x   3 root root      79 Mar 18 13:55 x-pack

3.3 配置Kibana有鹿，更多配置： kibana配置文件

vim config/kibana.yml ，配置elasticsearch地址和kibana地址信息

......
server.host: "192.168.33.13"
......
elasticsearch.hosts: ["http://192.168.33.13:9200"]
......
i18n.locale: "zh-CN"

3.4 啟動(dòng)

不能使用root啟動(dòng)

[root@localhost kibana]# bin/kibana
Kibana should not be run as root.  Use --allow-root to continue.

修改kibana的用戶權(quán)限為es谎脯，接著在切到es用戶啟動(dòng)

[root@localhost kibana]# ll
total 1448
drwxr-xr-x   2 root root      94 Mar 18 13:56 bin
drwxr-xr-x   2 root root      44 Jun 17 21:27 config
drwxr-xr-x   2 root root       6 Mar 18 13:55 data
-rw-r--r--   1 root root    3860 Mar 18 13:55 LICENSE.txt
drwxr-xr-x   6 root root     108 Mar 18 13:55 node
drwxr-xr-x 831 root root   24576 Mar 18 13:55 node_modules
-rw-r--r--   1 root root 1428396 Mar 18 13:55 NOTICE.txt
-rw-r--r--   1 root root     740 Mar 18 13:55 package.json
drwxr-xr-x   2 root root       6 Mar 18 13:55 plugins
-rw-r--r--   1 root root    3968 Mar 18 13:55 README.txt
drwxr-xr-x  12 root root     187 Mar 18 13:55 src
drwxr-xr-x   3 root root      79 Mar 18 13:55 x-pack
[root@localhost kibana]# chown -R es:es /usr/local/kibana/
[root@localhost kibana]# ll
total 1448
drwxr-xr-x   2 es es      94 Mar 18 13:56 bin
drwxr-xr-x   2 es es      44 Jun 17 21:27 config
drwxr-xr-x   2 es es       6 Mar 18 13:55 data
-rw-r--r--   1 es es    3860 Mar 18 13:55 LICENSE.txt
drwxr-xr-x   6 es es     108 Mar 18 13:55 node
drwxr-xr-x 831 es es   24576 Mar 18 13:55 node_modules
-rw-r--r--   1 es es 1428396 Mar 18 13:55 NOTICE.txt
-rw-r--r--   1 es es     740 Mar 18 13:55 package.json
drwxr-xr-x   2 es es       6 Mar 18 13:55 plugins
-rw-r--r--   1 es es    3968 Mar 18 13:55 README.txt
drwxr-xr-x  12 es es     187 Mar 18 13:55 src
drwxr-xr-x   3 es es      79 Mar 18 13:55 x-pack

4. 安裝logstash

4.1 下載

[root@localhost download]# rm -f logstash-7.12.0-linux-x86_64.tar.gz
[root@localhost download]# wget https://artifacts.elastic.co/downloads/logstash/logstash-7.12.0-linux-x86_64.tar.gz
--2021-06-17 21:49:25--  https://artifacts.elastic.co/downloads/logstash/logstash-7.12.0-linux-x86_64.tar.gz
Resolving artifacts.elastic.co (artifacts.elastic.co)... 34.120.127.130, 2600:1901:0:1d7::
Connecting to artifacts.elastic.co (artifacts.elastic.co)|34.120.127.130|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 368429061 (351M) [application/x-gzip]
Saving to: ‘logstash-7.12.0-linux-x86_64.tar.gz’

100%[=====================================================================================================================================================================================================================================>] 368,429,061 18.9MB/s   in 45s

2021-06-17 21:50:11 (7.87 MB/s) - ‘logstash-7.12.0-linux-x86_64.tar.gz’ saved [368429061/368429061]

4.2 解壓并移動(dòng)到指定目錄

[root@localhost local]# tar -zxvf logstash-7.12.0-linux-x86_64.tar.gz
[root@localhost download]# mv logstash-7.12.0 /usr/local/logstash
[root@localhost download]# cd /usr/local/
[root@localhost local]# ll
total 4
drwxr-xr-x.  2 root root    89 Jun 17 16:13 bin
drwxr-xr-x. 10 es   es     167 Jun 17 21:00 elasticsearch
drwxr-xr-x.  2 root root     6 Apr 11  2018 etc
drwxr-xr-x.  6 root root    56 Jun 17 15:54 freetype
drwxr-xr-x.  2 root root     6 Apr 11  2018 games
drwxr-xr-x. 10 root root   272 May  6 23:05 go
drwxr-xr-x.  3 root root   122 Jun 17 15:53 include
drwxr-xr-x.  9 root root   107 Jun 17 11:00 java
drwxr-xr-x  10 es   es     210 Jun 17 21:22 kibana
drwxr-xr-x.  3 root root  4096 Jun 17 15:53 lib
drwxr-xr-x.  2 root root     6 Apr 11  2018 lib64
drwxr-xr-x.  2 root root     6 Apr 11  2018 libexec
drwxr-xr-x  13 root root   266 Jun 17 21:50 logstash
drwxr-xr-x.  3 root root    18 Jun 17 15:53 man
drwxr-xr-x. 14 root mysql  216 Jun 17 16:05 mysql
drwxr-xr-x. 11 root root   151 Jun 17 16:15 nginx
drwxr-xr-x. 10 root root   102 Jun 17 16:13 php
drwxr-xr-x.  4 root root    50 Jun  7 14:49 protoc
drwxr-xr-x.  2 root root     6 Apr 11  2018 sbin
drwxr-xr-x.  8 root root    89 Jun 17 15:53 share
drwxr-xr-x.  2 root root     6 Apr 11  2018 src

使用logstash定時(shí)讀取log文件葱跋，并插入mysql數(shù)據(jù)庫中，output使用logstash-output-jdbc插件源梭。該插件不是默認(rèn)安裝的年局，需要使用命令：bin/logstash-plugin install logstash-output-jdbc去官方拉取际看。

[root@localhost logstash]# ./bin/logstash-plugin install logstash-output-jdbc
Using bundled JDK: /usr/local/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Validating logstash-output-jdbc
Installing logstash-output-jdbc
Installation successful

需要準(zhǔn)備一個(gè)mysql-connector-java-x.x.x.jar對(duì)應(yīng)版本的jar咸产，這里使用的是mysql-connector-java-5.1.6.jar矢否。

這個(gè)可以去官網(wǎng)下載：https://dev.mysql.com/downloads/connector/j/5.1.html

我本地的mysql是5.7版本，下載的是mysql-connector-java-5.1.49

[root@localhost logstash]# cd /data/download/
[root@localhost download]# wget https://downloads.mysql.com/archives/get/p/3/file/mysql-connector-java-5.1.49.tar.gz
[root@localhost download]# tar -zxvf mysql-connector-java-5.1.49.tar.gz
[root@localhost download]# cd mysql-connector-java-5.1.49/
[root@localhost download]# cp mysql-connector-java-5.1.49.jar /usr/local/logstash/logstash-core/lib/jars/
[root@localhost download]# cd /usr/local/logstash/

4.3 配置并啟動(dòng)

# 創(chuàng)建自定義的配置文件目錄
[root@localhost logstash]# mkdir sync-diy
# 創(chuàng)建用于記錄增量復(fù)制位置的文件
[root@localhost logstash]# touch sync-diy/account_track_record
# 創(chuàng)建logstash啟動(dòng)的配置文件
[root@localhost logstash]# touch sync-diy/logstash-mysql-es.conf

sync-diy/logstash-mysql-es.conf文件的內(nèi)容如下

input {
  stdin{}
  jdbc {
    # mysql相關(guān)jdbc配置
    jdbc_connection_string => "jdbc:mysql://localhost:3306/db_name?characterEncoding=utf8&serverTimezone=Asia/Shanghai"
    jdbc_user => "db_user"
    jdbc_password => "db_password"

    # jdbc連接mysql驅(qū)動(dòng)的文件目錄
    jdbc_driver_library => "/usr/local/logstash/logstash-core/lib/jars/mysql-connector-java-5.1.49.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    #數(shù)據(jù)庫重連嘗試次數(shù)
    connection_retry_attempts=>"3"
    #判斷數(shù)據(jù)庫連接是否可用脑溢，默認(rèn)false不開啟
    jdbc_validate_connection=>"true"
    #數(shù)據(jù)庫連接可用校驗(yàn)超時(shí)時(shí)間僵朗，默認(rèn)3600S
    jdbc_validation_timeout=>"3600"
    #開啟分頁查詢（默認(rèn)false不開啟）
    jdbc_paging_enabled => true
    jdbc_page_size => "5000"
    jdbc_default_timezone =>"Asia/Shanghai"

    # mysqlSQL語句
    statement => "SELECT user_id, reg_time, 
CASE WHEN JSON_VALID(ext) THEN IFNULL(JSON_UNQUOTE(json_extract(ext, '$.imei')), '') ELSE '' END AS imei,
CASE WHEN JSON_VALID(ext) THEN IFNULL(JSON_UNQUOTE(json_extract(ext, '$.oaid')), '') ELSE '' END AS oaid,
CASE WHEN JSON_VALID(ext) THEN IFNULL(JSON_UNQUOTE(json_extract(ext, '$.idfa')), '') ELSE '' END AS idfa,
CASE WHEN JSON_VALID(ext) THEN IFNULL(JSON_UNQUOTE(json_extract(ext, '$.idfv')), '') ELSE '' END AS idfv
FROM account WHERE reg_time >= :sql_last_value order by reg_time asc"

    # 也可以使用文件
    # statement_filepath => "./config/jdbc.sql"

    #Valuecanbeanyof:fatal,error,warn,info,debug，默認(rèn)info屑彻；
    sql_log_level=>warn

    # 這里類似crontab,可以定制定時(shí)操作验庙，比如每分鐘執(zhí)行一次同步(分 時(shí) 天 月 年)
    schedule => "* * * * *"
    #type => "jdbc"

    # 是否記錄上次執(zhí)行結(jié)果, 如果為真,將會(huì)把上次執(zhí)行到的 tracking_column 字段的值記錄下來,保存到 last_run_metadata_path 指定的文件中
    record_last_run => true

    # 是否需要記錄某個(gè)column 的值,如果record_last_run為真,可以自定義我們需要 track 的 column 名稱，此時(shí)該參數(shù)就要為 true. 否則默認(rèn) track 的是 timestamp 的值.
    use_column_value => true

    # 如果 use_column_value 為真,需配置此參數(shù). track 的數(shù)據(jù)庫 column 名,該 column 必須是遞增的. 一般是mysql主鍵
    tracking_column => "reg_time"

    #Valuecanbeanyof:numeric,timestamp社牲，Defaultvalueis"numeric"
    tracking_column_type => "timestamp"

    #record_last_run上次數(shù)據(jù)存放位置
    last_run_metadata_path => "/usr/local/logstash/sync-diy/account_track_record"

    # 是否清除 last_run_metadata_path 的記錄,如果為真那么每次都相當(dāng)于從頭開始查詢所有的數(shù)據(jù)庫記錄
    clean_run => false

    #是否將 字段(column) 名稱轉(zhuǎn)小寫
    lowercase_column_names => false
  }
}

output {
  elasticsearch {
    hosts => "192.168.33.13:9200"
    index => "sdk_account_device_info"
    # 對(duì)應(yīng)表的主鍵
    document_id => "%{user_id}"
    template_overwrite => true
  }
  
  # 這里輸出調(diào)試粪薛，正式運(yùn)行時(shí)可以注釋掉
  stdout {
      codec => json_lines
  }
}

啟動(dòng)

[root@localhost logstash]# ./bin/logstash -f ./sync-diy/logstash-mysql-es.conf

多表同步

規(guī)則如下:
一個(gè)表,一個(gè)配置
多個(gè)表,多個(gè)配置
需要同步多少表,就需要加多少配置

當(dāng)然配置的內(nèi)容都差不多,改的地方是查詢的表名,和es的索引以及類型的名稱
進(jìn)入logstash/config目錄,修改配置文件vim pipelines.yml編輯文件
直接到最后,添加配置

- pipeline.id: table1
  path.config: "/usr/local/software/my/logstash-6.7.2/config/user.conf"
- pipeline.id: table2
  path.config: "/usr/local/software/my/logstash-6.7.2/config/goods.conf"

啟動(dòng)方式稍有改變,進(jìn)入bin目錄

./logstash

5. php操作es

5.1 安裝composer

https://www.runoob.com/w3cnote/composer-install-and-usage.html
Elasticsearch-php
tip：
Elasticsearch-php 的安裝需要滿足以下 4 個(gè)需求

PHP 7.0.0 或更高版本
Composer
ext-curl：PHP 的 Libcurl 擴(kuò)展
原生 JSON 擴(kuò)展 (ext-json) 1.3.7或更高版本

[root@localhost local]# mkdir elasticsearch-php
[root@localhost local]# cd elasticsearch-php/

vim composer.json寫入如下內(nèi)容

{
    "require": {
        "elasticsearch/elasticsearch": "~6.0"
    }
}

執(zhí)行命令安裝composer擴(kuò)展

[root@localhost elasticsearch-php]# composer install --no-dev

vim search.php寫入如下內(nèi)容測(cè)試

<?php
require 'vendor/autoload.php';

$hosts = [
    '192.168.33.13:9200' // ip和端口
];

$client = Elasticsearch\ClientBuilder::create()
                                ->setHosts($hosts)
                                ->build();

$params = [
    'index' => 'sdk_account_device_info',
    'type' => '_doc',
    "size" => 10,  //代表每次分頁查詢的條數(shù)
    "from" => 10,  //代表每次分頁查詢起點(diǎn)
    //'id' => 'hy1b9aa0a216050816343786939d36680aa687' // http://192.168.247.140:9200/test/test/5
    'body' => [
        'query' => [
            'match' => [
                'imei' => 'b59c16aef5acb153'
            ]
        ]
    ]
];


$response = $client->search($params);
var_dump($response);

執(zhí)行查看效果：

[root@localhost elasticsearch-php]# php search.php
array(4) {
  ["took"]=>
  int(2)
  ["timed_out"]=>
  bool(false)
  ["_shards"]=>
  array(4) {
    ["total"]=>
    int(1)
    ["successful"]=>
    int(1)
    ["skipped"]=>
    int(0)
    ["failed"]=>
    int(0)
  }
  ["hits"]=>
  array(3) {
    ["total"]=>
    array(2) {
      ["value"]=>
      int(90)
      ["relation"]=>
      string(2) "eq"
    }
    ["max_score"]=>
    float(2.5499628)
    ["hits"]=>
    array(10) {
      [0]=>
      array(5) {
        ["_index"]=>
        string(23) "sdk_account_device_info"
        ["_type"]=>
        string(4) "_doc"
        ["_id"]=>
        string(38) "397716164989830981c75643ff4b4edb...."
        ["_score"]=>
        float(2.5499628)
        ["_source"]=>
        array(9) {
          ["@timestamp"]=>
          string(24) "2021-06-18T10:33:03.783Z"
          ["user_id"]=>
          string(38) "397716164989830981c75643ff4b4edb...."
          ["reg_time"]=>
          string(24) "2021-03-23T11:29:43.000Z"
          ["idfv"]=>
          string(0) ""
          ["@version"]=>
          string(1) "1"
          ["oaid"]=>
          string(64) "F362B61F3575BECB9E2...."
          ["idfa"]=>
          string(0) ""
          ["imei"]=>
          string(16) "b59c16aef5acb153"
        }
      }
      [1]=>
      array(5) {
        ["_index"]=>
        string(23) "sdk_account_device_info"
        ["_type"]=>
......

安全相關(guān)

Elastic 安全是非常重要的。沒有這個(gè)我們的數(shù)據(jù)可以被任何的人進(jìn)行訪問搏恤，串改违寿，刪除。Elastic Stack 的安全是由 x-pack 所提供的熟空。在 Elastic Stack 7.0 版本之前藤巢，這個(gè)是商用的版本，需要進(jìn)行安裝息罗，并購買掂咒。從Elastic Stack 7.0之后，x-pack 都已經(jīng)在發(fā)布版中迈喉，所以不需要進(jìn)行安裝绍刮。我們只需要進(jìn)行配置就可以了。
vim config/elasticsearch.yml 追加如下內(nèi)容

http.cors.enabled: true
http.cors.allow-origin: "*"
http.cors.allow-headers: Authorization
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true

修改密碼

[es@localhost elasticsearch]# elasticsearch-setup-passwords interactive
Initiating the setup of passwords for reserved users elastic,apm_system,kibana,logstash_system,beats_system,remote_monitoring_user.
You will be prompted to enter passwords as the process progresses.
Please confirm that you would like to continue [y/N]y

Enter password for [elastic]:
Reenter password for [elastic]:
Enter password for [apm_system]:
Reenter password for [apm_system]:
Enter password for [kibana]:
Reenter password for [kibana]:
Enter password for [logstash_system]:
Reenter password for [logstash_system]:
Enter password for [beats_system]:
Reenter password for [beats_system]:
Enter password for [remote_monitoring_user]:
Reenter password for [remote_monitoring_user]:
Changed password for user [apm_system]
Changed password for user [kibana]
Changed password for user [logstash_system]
Changed password for user [beats_system]
Changed password for user [remote_monitoring_user]
Changed password for user [elastic]

由于主要是es和kibana 所以我設(shè)置了es密碼為 es123pw; kibana密碼為 kibana123pw,其中es的用戶名默認(rèn)是 elastic挨摸，kibana的用戶名默認(rèn)是 kibana_system孩革，其他的系統(tǒng)暫時(shí)不知道作用，因此我都是給了123456的密碼油坝。

修改kibana配置如下：

#elasticsearch.username: "kibana_system"
elasticsearch.username: "kibana_system"
#elasticsearch.password: "pass"
elasticsearch.password: "kibana123pw"

保證操作正常嫉戚，重啟下es和kibana，再次訪問http://192.168.33.13:9200/?pretty和http://192.168.33.13:5601將會(huì)要求輸入密碼澈圈。

最后編輯于：2021.08.28 20:14:21

?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請(qǐng)聯(lián)系作者

人面猴
序言：七十年代末彬檀，一起剝皮案震驚了整個(gè)濱河市，隨后出現(xiàn)的幾起案子瞬女，更是在濱河造成了極大的恐慌窍帝，老刑警劉巖，帶你破解...
沈念sama閱讀 217,826評(píng)論 6贊 506
死咒
序言：濱河連續(xù)發(fā)生了三起死亡事件诽偷，死亡現(xiàn)場(chǎng)離奇詭異坤学，居然都是意外死亡疯坤，警方通過查閱死者的電腦和手機(jī)，發(fā)現(xiàn)死者居然都...
沈念sama閱讀 92,968評(píng)論 3贊 395
救了他兩次的神仙讓他今天三更去死
文/潘曉璐我一進(jìn)店門深浮，熙熙樓的掌柜王于貴愁眉苦臉地迎上來压怠，“玉大人，你說我怎么就攤上這事。” “怎么了途事？”我有些...
開封第一講書人閱讀 164,234評(píng)論 0贊 354
道士緝兇錄：失蹤的賣姜人
文/不壞的土叔我叫張陵，是天一觀的道長雨让。經(jīng)常有香客問我，道長忿等，這世上最難降的妖魔是什么栖忠？我笑而不...
開封第一講書人閱讀 58,562評(píng)論 1贊 293
?港島之戀（遺憾婚禮）
正文為了忘掉前任，我火速辦了婚禮贸街，結(jié)果婚禮上庵寞，老公的妹妹穿的比我還像新娘。我一直安慰自己匾浪，他們只是感情好皇帮，可當(dāng)我...
茶點(diǎn)故事閱讀 67,611評(píng)論 6贊 392
惡毒庶女頂嫁案：這布局不是一般人想出來的
文/花漫我一把揭開白布。她就那樣靜靜地躺著蛋辈，像睡著了一般属拾。火紅的嫁衣襯著肌膚如雪。梳的紋絲不亂的頭發(fā)上冷溶，一...
開封第一講書人閱讀 51,482評(píng)論 1贊 302
城市分裂傳說
那天渐白，我揣著相機(jī)與錄音，去河邊找鬼逞频。笑死纯衍，一個(gè)胖子當(dāng)著我的面吹牛，可吹牛的內(nèi)容都是我干的苗胀。我是一名探鬼主播襟诸，決...
沈念sama閱讀 40,271評(píng)論 3贊 418
雙鴛鴦連環(huán)套：你想象不到人心有多黑
文/蒼蘭香墨我猛地睜開眼，長吁一口氣：“原來是場(chǎng)噩夢(mèng)啊……” “哼基协！你這毒婦竟也來了歌亲？” 一聲冷哼從身側(cè)響起，我...
開封第一講書人閱讀 39,166評(píng)論 0贊 276
萬榮殺人案實(shí)錄
序言：老撾萬榮一對(duì)情侶失蹤澜驮，失蹤者是張志新（化名）和其女友劉穎陷揪，沒想到半個(gè)月后，有當(dāng)?shù)厝嗽跇淞掷锇l(fā)現(xiàn)了一具尸體，經(jīng)...
沈念sama閱讀 45,608評(píng)論 1贊 314
?護(hù)林員之死
正文獨(dú)居荒郊野嶺守林人離奇死亡悍缠，尸身上長有42處帶血的膿包…… 初始之章·張勛以下內(nèi)容為張勛視角年9月15日...
茶點(diǎn)故事閱讀 37,814評(píng)論 3贊 336
?白月光啟示錄
正文我和宋清朗相戀三年卦绣，在試婚紗的時(shí)候發(fā)現(xiàn)自己被綠了。大學(xué)時(shí)的朋友給我發(fā)了我未婚夫和他白月光在一起吃飯的照片飞蚓。...
茶點(diǎn)故事閱讀 39,926評(píng)論 1贊 348
活死人
序言：一個(gè)原本活蹦亂跳的男人離奇死亡滤港，死狀恐怖，靈堂內(nèi)的尸體忽然破棺而出玷坠，到底是詐尸還是另有隱情蜗搔，我是刑警寧澤，帶...
沈念sama閱讀 35,644評(píng)論 5贊 346
?日本核電站爆炸內(nèi)幕
正文年R本政府宣布八堡，位于F島的核電站，受9級(jí)特大地震影響聘芜，放射性物質(zhì)發(fā)生泄漏兄渺。R本人自食惡果不足惜，卻給世界環(huán)境...
茶點(diǎn)故事閱讀 41,249評(píng)論 3贊 329
男人毒藥：我在死后第九天來索命
文/蒙蒙一汰现、第九天我趴在偏房一處隱蔽的房頂上張望挂谍。院中可真熱鬧，春花似錦瞎饲、人聲如沸口叙。這莊子的主人今日做“春日...
開封第一講書人閱讀 31,866評(píng)論 0贊 22
一樁弒父案嗅战，背后竟有這般陰謀
文/蒼蘭香墨我抬頭看了看天上的太陽妄田。三九已至，卻和暖如春驮捍，著一層夾襖步出監(jiān)牢的瞬間疟呐，已是汗流浹背。一陣腳步聲響...
開封第一講書人閱讀 32,991評(píng)論 1贊 269
情欲美人皮
我被黑心中介騙來泰國打工东且，沒想到剛下飛機(jī)就差點(diǎn)兒被人妖公主榨干…… 1. 我叫王不留启具，地道東北人。一個(gè)月前我還...
沈念sama閱讀 48,063評(píng)論 3贊 370
代替公主和親
正文我出身青樓珊泳，卻偏偏與公主長得像鲁冯，于是被迫代替她去往敵國和親。傳聞我的和親對(duì)象是個(gè)殘疾皇子色查，可洞房花燭夜當(dāng)晚...
茶點(diǎn)故事閱讀 44,871評(píng)論 2贊 354

7.12.0 es+kibana+logstash

7.12.0 es+kibana+logstash

1. 安裝java環(huán)境【說是Elasticsearch 7.0 開始，內(nèi)置了 Java 環(huán)境檀咙，反正也不麻煩，在裝下吧】

1.1 下載相應(yīng)版本的壓縮包

1.2 解壓縮并移至相應(yīng)目錄

2. 安裝elasticsearch

2.1 下載

2.2 解壓并移動(dòng)到指定目錄

2.3 創(chuàng)建用戶并修改文件所屬用戶用戶組

2.4 啟動(dòng)es

配置外網(wǎng)訪問

測(cè)試啟動(dòng)

3. 安裝kibana

3.1下載

3.2 解壓并移動(dòng)到指定目錄

3.3 配置Kibana有鹿，更多配置： kibana配置文件

3.4 啟動(dòng)

4. 安裝logstash

4.1 下載

4.2 解壓并移動(dòng)到指定目錄

4.3 配置并啟動(dòng)

多表同步

5. php操作es

5.1 安裝composer

5.2 建一個(gè)目錄作為下載目錄

安全相關(guān)

推薦閱讀更多精彩內(nèi)容