快速開(kāi)始
本指南幫助初學(xué)者學(xué)習(xí)如何:
在測(cè)試環(huán)境中安裝并運(yùn)行Elasticsearch
向Elasticsearch添加數(shù)據(jù)
搜索和排序數(shù)據(jù)
在搜索期間從非結(jié)構(gòu)化內(nèi)容中提取字段
步驟1 執(zhí)行 Elasticsearch 命令
建立Elasticsearch最簡(jiǎn)單的方法是在彈性云上使用Elasticsearch Service創(chuàng)建托管部署袭灯。如果您喜歡管理自己的測(cè)試環(huán)境祷舀，可以使用Docker安裝并運(yùn)行Elasticsearch
Install and run Elasticsearch
1. Install and start [Docker Desktop](https://www.docker.com/products/docker-desktop).
2. Run:
docker network create elastic
docker pull docker.elastic.co/elasticsearch/elasticsearch:7.13.2
docker run --name es01-test --net elastic -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node"
docker.elastic.co/elasticsearch/elasticsearch:7.13.2
Install and run Kibana
To analyze, visualize, and manage Elasticsearch data using an intuitive UI, install Kibana.
1. In a new terminal session, run:
docker pull docker.elastic.co/kibana/kibana:7.13.2
docker run --name kib01-test --net elastic -p 5601:5601 -e "ELASTICSEARCH_HOSTS=http://es01-test:9200" docker.elastic.co/kibana/kibana:7.13.2
2. To access Kibana, go to [http://localhost:5601](http://localhost:5601/)
步驟2 向Elasticsearch發(fā)送請(qǐng)求
您使用REST api向Elasticsearch發(fā)送數(shù)據(jù)和其他請(qǐng)求嗡官。這允許您使用任何發(fā)送HTTP請(qǐng)求的客戶端(比如curl)與Elasticsearch進(jìn)行交互巫玻。您還可以使用Kibana的控制臺(tái)向Elasticsearch發(fā)送請(qǐng)求
Use curl
To submit an example API request, run the following curl command in a new terminal session.
curl -X GET http://localhost:9200/
Use Kibana
- Open Kibana’s main menu and go to Dev Tools > Console.
- Run the following example API request in the console:
GET /
步驟3 添加數(shù)據(jù)
您將數(shù)據(jù)作為稱為文檔的JSON對(duì)象添加到Elasticsearch中献宫。Elasticsearch將這些文檔存儲(chǔ)在可搜索的索引中忽孽。
對(duì)于時(shí)間序列數(shù)據(jù)(如日志和度量)硝皂，通常將文檔添加到由多個(gè)自動(dòng)生成的支持索引組成的數(shù)據(jù)流中顷窒。
數(shù)據(jù)流需要一個(gè)與其名稱匹配的索引模板蛙吏。Elasticsearch使用這個(gè)模板來(lái)配置流的后臺(tái)索引源哩。發(fā)送到數(shù)據(jù)流的文檔必須有@timestamp字段。
添加一個(gè)單獨(dú)的文檔
提交以下索引請(qǐng)求鸦做，將單個(gè)日志條目添加到logs-my_app-default數(shù)據(jù)流中励烦。因?yàn)閘ogs-my_app-default不存在，請(qǐng)求自動(dòng)使用內(nèi)置的logs--索引模板創(chuàng)建它
POST logs-my_app-default/_doc
{
"@timestamp": "2099-05-06T16:21:15.000Z",
"event": {
"original": "192.0.2.42 - - [06/May/2099:16:21:15 +0000] \"GET /images/bg.jpg HTTP/1.0\" 200 24736"
}
}
響應(yīng)包含了Elasticsearch為文檔生成的元數(shù)據(jù):
- 包含文檔的backing _index泼诱。Elasticsearch自動(dòng)生成支持索引的名稱坛掠。
- 索引內(nèi)文檔的唯一_id。
{
"_index": ".ds-logs-my_app-default-2099-05-06-000001",
"_type": "_doc",
"_id": "gl5MJXMBMk1dGnErnBW8",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"_seq_no": 0,
"_primary_term": 1
}
添加多個(gè)文件
使用_bulk端點(diǎn)在一個(gè)請(qǐng)求中添加多個(gè)文檔治筒。批量數(shù)據(jù)必須是新行分隔的JSON (NDJSON)屉栓。每一行必須以換行符(\n)結(jié)尾，包括最后一行耸袜。
PUT logs-my_app-default/_bulk
{ "create": { } }
{ "@timestamp": "2099-05-07T16:24:32.000Z", "event": { "original": "192.0.2.242 - - [07/May/2020:16:24:32 -0500] \"GET /images/hm_nbg.jpg HTTP/1.0\" 304 0" } }
{ "create": { } }
{ "@timestamp": "2099-05-08T16:25:42.000Z", "event": { "original": "192.0.2.255 - - [08/May/2099:16:25:42 +0000] \"GET /favicon.ico HTTP/1.0\" 200 3638" } }
步驟4 搜索數(shù)據(jù)
建立了索引的文檔可用于近乎實(shí)時(shí)的搜索友多。下面的搜索將匹配logs-my_app-default中的所有日志條目，并按@timestamp按降序?qū)λ鼈冞M(jìn)行排序句灌。
GET logs-my_app-default/_search
{
"query": {
"match_all": { }
},
"sort": [
{
"@timestamp": "desc"
}
]
}
默認(rèn)情況下夷陋，響應(yīng)的命中部分最多包括匹配搜索的前10個(gè)文檔。每次命中的_source都包含在索引期間提交的原始JSON對(duì)象胰锌。
{
"took": 2,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 3,
"relation": "eq"
},
"max_score": null,
"hits": [
{
"_index": ".ds-logs-my_app-default-2099-05-06-000001",
"_type": "_doc",
"_id": "PdjWongB9KPnaVm2IyaL",
"_score": null,
"_source": {
"@timestamp": "2099-05-08T16:25:42.000Z",
"event": {
"original": "192.0.2.255 - - [08/May/2099:16:25:42 +0000] \"GET /favicon.ico HTTP/1.0\" 200 3638"
}
},
"sort": [
4081940742000
]
},
...
]
}
}
獲得特定字段
對(duì)于大型文檔骗绕，解析整個(gè)_source非常笨拙。要從響應(yīng)中排除它资昧，請(qǐng)將_source參數(shù)設(shè)置為false酬土。相反，使用fields參數(shù)來(lái)檢索所需的字段格带。
GET logs-my_app-default/_search
{
"query": {
"match_all": { }
},
"fields": [
"@timestamp"
],
"_source": false,
"sort": [
{
"@timestamp": "desc"
}
]
}
響應(yīng)以平面數(shù)組的形式包含每個(gè)命中的字段值撤缴。
{
...
"hits": {
...
"hits": [
{
"_index": ".ds-logs-my_app-default-2099-05-06-000001",
"_type": "_doc",
"_id": "PdjWongB9KPnaVm2IyaL",
"_score": null,
"fields": {
"@timestamp": [
"2099-05-08T16:25:42.000Z"
]
},
"sort": [
4081940742000
]
},
...
]
}
}
搜索日期范圍
如果要跨時(shí)間段或IP范圍進(jìn)行搜索，請(qǐng)使用范圍查詢叽唱。
GET logs-my_app-default/_search
{
"query": {
"range": {
"@timestamp": {
"gte": "2099-05-05",
"lt": "2099-05-08"
}
}
},
"fields": [
"@timestamp"
],
"_source": false,
"sort": [
{
"@timestamp": "desc"
}
]
}
您可以使用日期數(shù)學(xué)來(lái)定義相對(duì)時(shí)間范圍屈呕。下面的查詢搜索過(guò)去一天的數(shù)據(jù)，它不會(huì)匹配logs-my_app-default中的任何日志條目
GET logs-my_app-default/_search
{
"query": {
"range": {
"@timestamp": {
"gte": "now-1d/d",
"lt": "now/d"
}
}
},
"fields": [
"@timestamp"
],
"_source": false,
"sort": [
{
"@timestamp": "desc"
}
]
}
從非結(jié)構(gòu)化內(nèi)容中提取字段
You can extract runtime fields from unstructured content, such as log messages, during a search.
Use the following search to extract the source.ip runtime field from event.original. To include it in the response, add source.ip to the fields parameter.
GET logs-my_app-default/_search
{
"runtime_mappings": {
"source.ip": {
"type": "ip",
"script": """
String sourceip=grok('%{IPORHOST:sourceip} .*').extract(doc[ "event.original" ].value)?.sourceip;
if (sourceip != null) emit(sourceip);
"""
}
},
"query": {
"range": {
"@timestamp": {
"gte": "2099-05-05",
"lt": "2099-05-08"
}
}
},
"fields": [
"@timestamp",
"source.ip"
],
"_source": false,
"sort": [
{
"@timestamp": "desc"
}
]
}
組合查詢
可以使用bool查詢組合多個(gè)查詢棺亭。下面的搜索組合了兩個(gè)范圍查詢:一個(gè)在@timestamp上虎眨，一個(gè)在源上。ip運(yùn)行時(shí)镶摘。
GET logs-my_app-default/_search
{
"runtime_mappings": {
"source.ip": {
"type": "ip",
"script": """
String sourceip=grok('%{IPORHOST:sourceip} .*').extract(doc[ "event.original" ].value)?.sourceip;
if (sourceip != null) emit(sourceip);
"""
}
},
"query": {
"bool": {
"filter": [
{
"range": {
"@timestamp": {
"gte": "2099-05-05",
"lt": "2099-05-08"
}
}
},
{
"range": {
"source.ip": {
"gte": "192.0.2.0",
"lte": "192.0.2.240"
}
}
}
]
}
},
"fields": [
"@timestamp",
"source.ip"
],
"_source": false,
"sort": [
{
"@timestamp": "desc"
}
]
}
聚合數(shù)據(jù)
使用聚合將數(shù)據(jù)總結(jié)為指標(biāo)嗽桩、統(tǒng)計(jì)數(shù)據(jù)或其他分析。
下面的搜索使用聚合來(lái)計(jì)算使用http.response.body.bytes運(yùn)行時(shí)字段的average_response_size凄敢。聚合只在與查詢匹配的文檔上運(yùn)行碌冶。
GET logs-my_app-default/_search
{
"runtime_mappings": {
"http.response.body.bytes": {
"type": "long",
"script": """
String bytes=grok('%{COMMONAPACHELOG}').extract(doc[ "event.original" ].value)?.bytes;
if (bytes != null) emit(Integer.parseInt(bytes));
"""
}
},
"aggs": {
"average_response_size":{
"avg": {
"field": "http.response.body.bytes"
}
}
},
"query": {
"bool": {
"filter": [
{
"range": {
"@timestamp": {
"gte": "2099-05-05",
"lt": "2099-05-08"
}
}
}
]
}
},
"fields": [
"@timestamp",
"http.response.body.bytes"
],
"_source": false,
"sort": [
{
"@timestamp": "desc"
}
]
}
響應(yīng)的聚合對(duì)象包含聚合結(jié)果
{
...
"aggregations" : {
"average_response_size" : {
"value" : 12368.0
}
}
}
Explore more search options
To keep exploring, index more data to your data stream and check out Common search options.
步驟5. 清理
完成后，刪除測(cè)試數(shù)據(jù)流及其備份索引
DELETE _data_stream/logs-my_app-default
- To stop your Elasticsearch and Kibana Docker containers, run:
docker stop es01-test
docker stop kib01-test
- To remove the containers and their network, run:
docker network rm elastic
docker rm es01-test
docker rm kib01-test
What’s next?
- Get the most out of your time series data by setting up data tiers and ILM. See Use Elasticsearch for time series data.
- Use Fleet and Elastic Agent to collect logs and metrics directly from your data sources and send them to Elasticsearch. See the Fleet quick start guide.
- Use Kibana to explore, visualize, and manage your Elasticsearch data. See the Kibana quick start guide.
