2020-10-10 架構(gòu)師第15周作業(yè)

架構(gòu)班的小伙伴作業(yè)看這里哦：（學(xué)習(xí)杰哥視頻的作業(yè)第29-30天）

1理张、使用filebeta收集nginx訪問日志怎栽，并通過geoip展示ip歸屬地

（一）測試的環(huán)境

agentd:192.168.7.22

ES：192.168.7.23

kibana：192.168.7.23

采用的拓?fù)洌簂ogstash -->ES-->kibana

（二）實(shí)施步驟:

? ?（1）logstsh具體配置：

1庶近，配置nginx日志格式，采用log_format格式:

log_format ?main ?'$remote_addr?- $remote_user?[$time_local] "$request" '

??????????????????????'$status?$body_bytes_sent?"$http_referer" '

??????????????????????'"$http_user_agent" "$http_x_forwarded_for"';

2,在logstash服務(wù)器下載IP地址歸類查詢庫

[root@localhost config]# cd /usr/local/logstash/config/

[root@localhost config]# ?wget http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz

3,配置logstash客戶端

[root@localhost config]# vim /usr/local/logstash/config/nginx-access.conf

input {

????????file?{

????????????????path =>?"/opt/access.log"

????????????????type?=>?"nginx"

????????????????start_position =>?"beginning"

??????????????}

????????????}

filter {

??????????grok {

????????????????match =>?{?"message"?=>?"%{IPORHOST:remote_addr} - - \[%{HTTPDATE:time_local}\] \"%{WO

RD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\" %{INT:status} %{INT:body_bytes_sent} %

{QS:http_referer} %{QS:http_user_agent}"

????????????????????????????????}

?????????????????????}

?geoip {

?????source?=>?"remote_addr"

?????target =>?"geoip"

?????database =>?"/usr/local/logstash/config/GeoLite2-City.mmdb"

?????add_field =>?["[geoip][coordinates]","%{[geoip][longitude]}"]

?????add_field =>?["[geoip][coordinates]","%{[geoip][latitude]}"]

????????????????}

???}

output {

????????????????elasticsearch {

????????????????????????hosts =>?["192.168.7.23:9200"]

????????????????????????manage_template =>?true?????????????????????????????????????

????????????????????????index =>?"logstash-map-%{+YYYY-MM}"?????????????????????????

??????????????????????????}???????????????????????????????????????????????????????????????????

?????????????}

備注：

geoip: IP查詢插件

source: 需要通過geoip插件處理的field霎槐，一般為ip监右，這里因?yàn)橥ㄟ^控制臺手動輸入的是ip所以直接填message，生成環(huán)境中如果查詢nginx訪問用戶潦牛，需先將客戶端ip過濾出來眶掌，然后這里填remote_addr即可

target: 解析后的Geoip地址數(shù)據(jù)，應(yīng)該存放在哪一個字段中巴碗，默認(rèn)是geoip這個字段

database: 指定下載的數(shù)據(jù)庫文件

add_field: 這里兩行是添加經(jīng)緯度朴爬，地圖中地區(qū)顯示是根據(jù)經(jīng)緯度來識。如果啟動正常的話橡淆，可以在kibana看到geoip相關(guān)的字段寝殴，如下圖:

4,啟動logstash客戶端并加載剛才的配置文件。

[root@localhost config]# /usr/local/logstash/bin/logstash -f nginx-access.conf ?

Sending Logstash's logs to /usr/local/logstash/logs which?is now configured via log4j2.properties[2017-06-20T22:55:23,801][INFO ][logstash.outputs.elasticsearch]?Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://192.168.7.23:9200/]}}[2017-06-20T22:55:23,805][INFO ][logstash.outputs.elasticsearch]?Running health check to see if?an Elasticsearch connection is working {:healthcheck_url=>http://192.168.7.23:9200/, :path=>"/"}[2017-06-20T22:55:23,901][WARN ][logstash.outputs.elasticsearch]?Restored connection to ES instance {:url=>#<URI::HTTP:0x4e54594d URL:http://192.168.7.23:9200/>}[2017-06-20T22:55:23,909][INFO ][logstash.outputs.elasticsearch]?Using mapping template from {:path=>nil}[2017-06-20T22:55:23,947][INFO ][logstash.outputs.elasticsearch]?Attempting to install?template {:manage_template=>{"template"=>"logstash-*", "version"=>50001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"_all"=>{"enabled"=>true, "norms"=>false}, "dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword"}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "include_in_all"=>false}, "@version"=>{"type"=>"keyword", "include_in_all"=>false}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}[2017-06-20T22:55:23,955][INFO ][logstash.outputs.elasticsearch]?New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>[#<URI::Generic:0x1e098115 URL://192.168.7.23:9200>]}[2017-06-20T22:55:24,065][INFO ][logstash.filters.geoip ??]?Using geoip database {:path=>"/usr/local/logstash/config/GeoLite2-City.mmdb"}[2017-06-20T22:55:24,094][INFO ][logstash.pipeline ???????]?Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}[2017-06-20T22:55:24,275][INFO ][logstash.pipeline ???????]?Pipeline main started[2017-06-20T22:55:24,369][INFO ][logstash.agent ??????????]?Successfully started Logstash API endpoint {:port=>9600}

（2）Kibana配置.

1,編輯修改kibana的配置文件kibana.yml在最后添加如下：

# The default locale. This locale can be used in certain circumstances to substitute any missing# translations.#i18n.defaultLocale: "en"

tilemap.url: 'http://webrd02.is.autonavi.com/appmaptile?lang=zh_cn&size=1&scale=1&style=7&x={x}&y={y}&

z={z}'

2,重啟kibana服務(wù)明垢。

[root@localhost bin]# /usr/local/kibana/bin/kibana &[1]?10631[root@localhost bin]# ps -ef|grep kibana ???????????

root ????10631 ?7795 21 10:52 pts/0 ???00:00:02 /usr/local/kibana/bin/../node/bin/node --no-warnings /usr/local/kibana/bin/../src/cli

root ????10643 ?7795 ?0 10:52 pts/0 ???00:00:00 grep?--color=auto kibana[root@localhost bin]# ??log ??[02:52:59.297] [info][status][plugin:kibana@5.4.0] Status changed from uninitialized to green - Ready

??log ??[02:52:59.445]?[info][status][plugin:elasticsearch@5.4.0]?Status changed from uninitialized to yellow - Waiting for?Elasticsearch

??log ??[02:52:59.482]?[info][status][plugin:console@5.4.0]?Status changed from uninitialized to green - Ready

??log ??[02:52:59.512]?[info][status][plugin:elasticsearch@5.4.0]?Status changed from yellow to green - Kibana index ready

??log ??[02:52:59.513]?[info][status][plugin:metrics@5.4.0]?Status changed from uninitialized to green - Ready

??log ??[02:53:00.075]?[info][status][plugin:timelion@5.4.0]?Status changed from uninitialized to green - Ready

??log ??[02:53:00.080]?[info][listening]?Server running at http://192.168.7.23:5601

??log ??[02:53:00.081]?[info][status][ui settings]?Status changed from uninitialized to green - Ready

3，創(chuàng)建nginx的訪問索引lostash-map*市咽。

具體步驟如下：ip:5601--->Management--Index Patterns--->+--->在Index name or pattern中添加logstash-map*--->create痊银。具體如下圖：

4,創(chuàng)建Visualize。具體步驟如下：Visalize--->+--->Maps(Tile Maps)

2施绎、配置nginx虛擬主機(jī)溯革，實(shí)現(xiàn)域名訪問kibana。

由于kibana界面默認(rèn)沒有安全認(rèn)證界面谷醉，為了保證安全致稀，通過nginx進(jìn)行代理并設(shè)置訪問認(rèn)證。

2.1 配置kibana

[root@linux-elk1 ~]# vim /etc/kibana/kibana.yml

server.host: "127.0.0.1"? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?#將監(jiān)聽地址更改為127.0.0.1

[root@linux-elk1 ~]# systemctl restart kibana

[root@linux-elk1 ~]# netstat -nlutp |grep?5601

tcp ???????0??????0?127.0.0.1:5601??????????0.0.0.0:* ??????????????LISTEN ?????72068/node

2.2 部署nginx

1）安裝nginx

[root@linux-elk1 ~]# yum?-y install?nginx httpd-tools

2）配置nginx

[root@linux-elk1 ~]# vim /etc/nginx/conf.d/kibana.conf

upstream kibana_server {

????server 127.0.0.1:5601?weight=1?max_fails=3?fail_timeout=60;

}

server {

????listen 80;

????server_name www.kibana.com;

????auth_basic "Restricted Access";

????auth_basic_user_file /etc/nginx/conf.d/htpasswd.users;

????location /?{

????????proxy_pass http://kibana_server;

????????proxy_http_version 1.1;

????????proxy_set_header Upgrade $http_upgrade;

????????proxy_set_header Connection 'upgrade';

????????proxy_set_header Host $host;

????????proxy_cache_bypass $http_upgrade;

????}

}

[root@linux-elk1 ~]# htpasswd -bc /etc/nginx/conf.d/htpasswd.users admin 123456

Adding password for?user admin

[root@linux-elk1 ~]# cat?/etc/nginx/conf.d/htpasswd.users

admin:$apr1$ro5tQZp9$grhByziZtm3ZpZCsSFzsQ1

[root@linux-elk1 ~]# systemctl start nginx

3）windows上添加hosts, 路徑C:\Windows\System32\drivers\etc\hosts

192.168.7.101? ? www.kibana.com

4）測試驗(yàn)證