ES 學習記錄
環(huán)境
- Centos7 X64
- ES V7.3.0
- Logstash-7.3.0
注意:
- 請不要使用root用戶啟動面殖,ES 為了安全嘱吗,禁止使用root賬戶啟動
- 內(nèi)存最好大一點,即便是虛擬機也最好大于3G苍鲜, 不然可能會啟動失敗
- ES入門教程 http://www.ruanyifeng.com/blog/2017/08/elasticsearch.html
1. ES 下載安裝
官方安裝 https://www.elastic.co/cn/downloads/elasticsearch
- 下載自己的版本
- 解壓
- 運行 Run bin/elasticsearch (or bin\elasticsearch.bat on Windows)
- 測試是否安裝成功命令胸囱。 curl http://localhost:9200/ ( 如果是windows系統(tǒng), 直接在瀏覽器訪問http://localhost:9200/ )
中文分詞插件
安裝命令
sudo ./bin/elasticsearch-plugin install
https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v7.3.0/elasticsearch-analysis-ik-7.3.0.zip
ES 中 index type document 概念
對于熟悉mysql的:
- index 可以理解為 數(shù)據(jù)庫
- type 理解為 表 [現(xiàn)版本的ES中, 每一個index,只能有一個type; 有廢棄的跡象]
- document 理解為 表中一條記錄
2 命令查詢
2.1查看所有index
curl是一種命令行工具洽瞬,作用是發(fā)出網(wǎng)絡(luò)請求,然后得到和提取數(shù)據(jù)操漠,顯示在"標準輸出"(stdout)上面收津。 可以理解為命令行版的postman
http://www.ruanyifeng.com/blog/2011/09/curl.html
curl -X GET 'http://localhost:9200/_cat/indices?v'
2.2新建index
curl -X PUT 'localhost:9200/weather'
2.3刪除
curl -X DELETE 'localhost:9200/weather'
2.4 POST 增加 Document; localhost:9200/{index}/{type}
curl -H "Content-Type: application/json" -X POST 'localhost:9200/weather/20180101' -d '
{
"date": "老六",
"title": "golang",
"desc": "go開發(fā)工程師"
}'
2.5 PUT 方式 增加document
localhost:9200/{index}/{type}/{id}
curl -H "Content-Type: application/json" -X PUT 'localhost:9200/weather/20180101/1' -d '
{
"date": "張三1",
"title": "工程師1",
"desc": "晴天1"
}'
2.6 查看記錄
查詢index = weather, type = 20180101, id = 1 的記錄
curl 'localhost:9200/weather/20180101/1'
刪除
curl -XDELETE 'localhost:9200/weather/20180101/1'
更新[存在更新,不存在新建]
curl -H "Content-Type: application/json" -XPUT 'localhost:9200/weather/20180101/1' -d '
{
"date":"lisi",
"title":"title2",
"desc":"dadfsaf"
}'
返回所有記錄
curl 'localhost:9200/weather/20180101/_search'
# 20180101 為type,現(xiàn)規(guī)定一個index只能有一個type, 所以不帶type, 也可以查詢所有記錄
curl 'localhost:9200/weather/_search'
# 返回結(jié)果
# 參數(shù)官方解釋 https://www.elastic.co/guide/cn/elasticsearch/guide/current/empty-search.html
# 可以留意下took浊伙, 表示查詢花費的時間
# score 標識匹配程度撞秋, 返回記錄中,每一條都有一個 _score 嚣鄙,記錄以此降序
{
"took":4,
"timed_out":false,
"_shards":{
"total":1,
"successful":1,
"skipped":0,
"failed":0
},
"hits":{
"total":{
"value":3,
"relation":"eq"
},
"max_score":1,
"hits":[
{
"_index":"weather",
"_type":"20180101",
"_id":"IbOjrmwBucBWZ7yGLubW",
"_score":1,
"_source":{
"date":"張三",
"title":"工程師",
"desc":"晴天"
}
},
{
"_index":"weather",
"_type":"20180101",
"_id":"JLOtrmwBucBWZ7yGrubi",
"_score":1,
"_source":{
"date":"張三",
"title":"工程師",
"desc":"晴天"
}
},
{
"_index":"weather",
"_type":"20180101",
"_id":"1",
"_score":1,
"_source":{
"date":"lisi",
"title":"title2",
"desc":"dadfsaf"
}
}
]
}
}
3 復(fù)雜搜索
3.1 查詢帶有 “晴” 的記錄吻贿, match 表示desc字段包含 “晴”
curl -H 'Content-Type: application/Json' 'localhost:9200/weather/20180101/_search' -d '
{
"query": {"match": {"desc":"晴"}}
}'
3.2 數(shù)據(jù)量太大,只想要前幾條數(shù)據(jù)? size:大小
curl -H 'Content-Type: application/Json' 'localhost:9200/weather/20180101/_search' -d '
{
"query": {"match": {"desc":"晴"}},
"size" : 1
}'
3.3 不想要前十條數(shù)據(jù)拗慨? from: 偏移廓八,默認0; 本例表示從位置1開始奉芦,查詢一條記錄
curl -H 'Content-Type: application/Json' 'localhost:9200/weather/20180101/_search' -d '
{
"query": {"match": {"desc":"晴"}},
"from": 1,
"size" : 1
}'
3.4 邏輯OR運算
工 和 師 之間多了一個空格赵抢,標識邏輯OR查詢
curl -H 'Content-Type: application/Json' 'localhost:9200/weather/20180101/_search' -d '
{
"query": {"match": {"title": "工 師"}}
}'
# 下面這個是不可以的, 多個field的查詢 需要用bool查詢,往下看
curl -H 'Content-Type: application/Json' 'localhost:9200/weather/20180101/_search' -d '
{
"query": {"match": {"desc": "工程師", "title":"golang" }}
}'
3.5 邏輯AND運算声功, must, should, must_not, filter
must :
curl -H 'Content-Type: application/Json' 'localhost:9200/weather/20180101/_search' -d '
{
"query":{
"bool": {
"must": [
{"match":{"desc":"js"}},
{"match":{"desc":"工程師"}}
]
}
}
}'
# 下面語句是 must: desc字段包含 "工程師", must_not: title字段不包含"golang"的 docucument
# must_not 語句不會影響評分烦却; 它的作用只是將不相關(guān)的文檔排除。
# 所有 must 語句必須匹配先巴,所有 must_not 語句都必須不匹配
curl -H 'Content-Type: application/Json' 'localhost:9200/weather/20180101/_search' -d '
{
"query":{
"bool": {
"must": {"match": {"desc":"工程師"}},
"must_not":{"match": {"title":"golang"}}
}
}
}'
# 結(jié)果
{
"took":2,
"timed_out":false,
"_shards":{
"total":1,
"successful":1,
"skipped":0,
"failed":0
},
"hits":{
"total":{
"value":2,
"relation":"eq"
},
"max_score":1.7263287,
"hits":[
{
"_index":"weather",
"_type":"20180101",
"_id":"3ffesWwBCUKd2dtSgQyB",
"_score":1.7263287,
"_source":{
"date":"王五",
"title":"JavaScript",
"desc":"js開發(fā)工程師"
}
},
{
"_index":"weather",
"_type":"20180101",
"_id":"3PfesWwBCUKd2dtSIAzZ",
"_score":1.5912248,
"_source":{
"date":"李四",
"title":"java",
"desc":"扎瓦開發(fā)工程師"
}
}
]
}
}
# should 一個文檔不必包含 扎瓦 或 js 這兩個詞項其爵,但如果一旦包含,我們就認為它們 更相關(guān) :
curl -H 'Content-Type: application/Json' 'localhost:9200/weather/20180101/_search' -d '
{
"query":{
"bool": {
"must": {"match": {"desc":"工程師"}},
"must_not":{"match": {"title":"golang"}},
"should": [
{"match": {"desc":"扎瓦"}},
{"match": {"desc":"js"}}
]
}
}
}'
#攜帶should 查詢的結(jié)果伸蚯,雖然返回的記錄相同摩渺,但是他們的得分更高
{
"took":21,
"timed_out":false,
"_shards":{
"total":1,
"successful":1,
"skipped":0,
"failed":0
},
"hits":{
"total":{
"value":2,
"relation":"eq"
},
"max_score":3.9487753,
"hits":[
{
"_index":"weather",
"_type":"20180101",
"_id":"3PfesWwBCUKd2dtSIAzZ",
"_score":3.9487753,
"_source":{
"date":"李四",
"title":"java",
"desc":"扎瓦開發(fā)工程師"
}
},
{
"_index":"weather",
"_type":"20180101",
"_id":"3ffesWwBCUKd2dtSgQyB",
"_score":3.0051887,
"_source":{
"date":"王五",
"title":"JavaScript",
"desc":"js開發(fā)工程師"
}
}
]
}
}
match 標識分詞匹配,term標識完全匹配剂邮, range 標識范圍
filtered 已經(jīng)被 bool query 替代
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-filtered-query.html
curl -H 'Content-Type: application/Json' 'localhost:9200/weather/20180101/_search' -d '
{
"query": {
"bool": {
"must": {
"match":{
"desc": "工程師"
}
},
"filter": {
"term": {
"title":"golang"
}
}
}
}
}
'
curl -H 'Content-Type: application/Json' 'localhost:9200/weather/20180101/_search' -d '
{
"query": {
"term": {
"title":"golang"
}
}
}'
curl -H 'Content-Type: application/Json' 'localhost:9200/weather/20180101/_search' -d '
{
"query": {
"bool": {
"filter": {
"match_phrase": {
"desc": {"query": "工程師"}
}
}
}
}
}
'
用到這里 已經(jīng)使用過兩個關(guān)鍵字 _cat, _search
4 ES 同步 Mysql
4.1 logstash 安裝
官方安裝文檔 https://www.elastic.co/cn/downloads/logstash
- 下載
- 解壓
- 配置一個簡單的文件
- 運行
4.2 配置
# jdbc.conf
# 輸入部分
input {
stdin {}
jdbc {
# mysql數(shù)據(jù)庫驅(qū)動
jdbc_driver_library => "http://home/ccnn/logstash-7.3.0/config/mysql-connector-java-5.1.47.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
# mysql數(shù)據(jù)庫鏈接摇幻,數(shù)據(jù)庫名
jdbc_connection_string => "jdbc:mysql://localhost:3306/test"
# mysql數(shù)據(jù)庫用戶名,密碼
jdbc_user => "root"
jdbc_password => "12345678"
# 設(shè)置監(jiān)聽間隔 各字段含義(分、時绰姻、天枉侧、月、年)狂芋,全部為*默認含義為每分鐘更新一次
schedule => "* * * * *"
# 分頁
jdbc_paging_enabled => "true"
# 分頁大小
jdbc_page_size => "50000"
# sql語句執(zhí)行文件榨馁,也可直接使用 statement => 'select * from t_employee'
statement_filepath => "jdbc_driver_library => "/home/ccnn/logstash-7.3.0/config/t_depart.sql"
# elasticsearch索引類型名
type => "t_employee"
}
}
# 過濾部分(不是必須項)
filter {
json {
source => "message"
remove_field => ["message"]
}
}
# 輸出部分
output {
elasticsearch {
# elasticsearch索引名
index => "octopus"
# 使用input中的type作為elasticsearch索引下的類型名
document_type => "%{type}" # <- use the type from each input
# elasticsearch的ip和端口號
hosts => "localhost:9200"
# 同步mysql中數(shù)據(jù)id作為elasticsearch中文檔id
document_id => "%{id}"
}
stdout {
codec => json_lines
}
}
# 注: 使用時請去掉此文件中的注釋,不然會報錯
# jdbc.sql
select * from t_employee
4.3 啟動服務(wù)帜矾, 指定配置文件
cd logstash-7.3.0
# 檢查配置文件語法是否正確
bin/logstash -f config/jdbc.conf --config.test_and_exit
# 啟動
bin/logstash -f config/jdbc.conf --config.reload.automatic
注冊為linux服務(wù)
- 略
4.4 配置多個同步表怎么辦翼虫?
# 查看index 是否添加進去
curl localhost:9200/_cat/indices
# 查看 index_employee里面有沒有數(shù)據(jù)?
curl localhost:9200/index_employee/_search
4.5 多表增量更新
# sql 中增加 :sql_last_value 過濾黍特, 要求表中存在更新日期字段
# t_depart.sql
SELECT * FROM T_depart where updated_at > :sql_last_value
# t_employee.sql
select * from t_employee where updated_at > :sql_last_value
# jdbc.conf
input {
jdbc {
jdbc_connection_string => "jdbc:mysql://192.168.1.150:3306/test"
jdbc_user => "root"
jdbc_password => "root"
jdbc_driver_library => "/home/ccnn/logstash-7.3.0/config/mysql-connector-java-5.1.47.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_paging_enabled => "true"
jdbc_page_size => "50000"
statement_filepath => "/home/ccnn/logstash-7.3.0/config/t_depart.sql"
schedule => "* * * * *"
type => "depart"
}
jdbc {
jdbc_connection_string => "jdbc:mysql://192.168.1.150:3306/test"
jdbc_user => "root"
jdbc_password => "root"
jdbc_driver_library => "/home/ccnn/logstash-7.3.0/config/mysql-connector-java-5.1.47.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_paging_enabled => "true"
jdbc_page_size => "50000"
parameters => {"number" => "200"}
statement_filepath => "/home/ccnn/logstash-7.3.0/config/t_employee.sql"
schedule => "* * * * *"
type => "employee"
}
}
filter {
json {
source => "message"
remove_field => ["message"]
}
}
output {
if[type] == "depart" {
elasticsearch {
hosts => ["localhost:9200"]
index => "index_depart"
document_id => "%{id}"
}
}
if[type] == "employee" {
elasticsearch {
hosts => ["localhost:9200"]
index => "index_employee"
document_id => "%{id}"
}
}
stdout {
codec => json_lines
}
}
5. go包 olivere/elastic 基本操作
選擇了第三方的工具包 https://gopkg.in/olivere/elastic.v7
go get gopkg.in/olivere/elastic.v7
import "gopkg.in/olivere/elastic.v7"
package main
import (
"context"
"encoding/json"
"log"
"gopkg.in/olivere/elastic.v7"
)
var client *elastic.Client
var host = "http://localhost:9200"
func main() {
Init()
log.Println("test go ", client)
e1 := Employee{1, "周瑞發(fā)", "password", 30}
//創(chuàng)建 其中 index type id 和 ES 中概念相同
put1, err1 := client.Index().Index("go-test").Type("emp").Id("2").BodyJson(e1).Do(context.Background())
if err1 != nil {
log.Println("init fiald", err1.Error())
}
log.Println("添加成功: ", put1.Id, put1.Index, put1.Type)
//修改
res0, err0 := client.Update().Index("go-test").Type("emp").Id("2").Doc(map[string]interface{}{"Name": "劉德華劉德華"}).Do(context.Background())
if err0 != nil {
log.Println("修改失敗 ", err0.Error())
}
log.Println("更新 ", res0.Result)
//查詢id
res2, err2 := client.Get().Index("go-test").Type("emp").Id("2").Do(context.Background())
if err2 != nil {
log.Println("查詢", err2.Error())
} else {
log.Println("查詢結(jié)果: ", res2.Id, res2.Index, res2.Type, res2.Fields)
if res2.Found {
source, _ := res2.Source.MarshalJSON()
log.Println(source)
log.Println(string(source[:]))
}
}
//根據(jù)文檔名字查詢
//搜索
res5, err5 := client.Search("go-test").Type("emp").Do(context.Background())
if err5 != nil {
log.Println(err5)
}
log.Println("TotalHits ", res5.TotalHits())
// match query
matchQuery := elastic.NewMatchQuery("Password", "password")
rsp, err := client.Search("go-test").Type("emp").Query(matchQuery).Do(context.Background())
if err == nil {
log.Println("match query : ", rsp.Hits.Hits)
if err == nil {
for _, hit := range rsp.Hits.Hits {
var cmp Employee
err := json.Unmarshal(hit.Source, &cmp)
if err != nil {
log.Panicln(err)
}
log.Println(cmp)
}
}
}
//刪除
res1, err1 := client.Delete().Index("go-test").Type("emp").Id("1").Do(context.Background())
if err1 != nil {
log.Println("刪除失敗 ", err1.Error())
}
log.Println("刪除: ", res1.Result)
}
func Init() {
log.Println("init start ")
var err0 error
// 初始化一個操作對象
client, err0 = elastic.NewClient(elastic.SetURL(host))
if err0 != nil {
log.Println(err0)
}
//是否連接成功
info, code, err1 := client.Ping(host).Do(context.Background())
if err1 != nil {
log.Println(err1.Error())
return
}
log.Println(info, code)
//獲取ES的信息
esversion, err2 := client.ElasticsearchVersion(host)
if err2 != nil {
log.Println(err2.Error())
return
}
log.Println("ES VERSION : ", esversion)
}
type Employee struct {
Id int
Name string
Password string
Age int
}