本文默認(rèn)讀者已經(jīng)對(duì)Docker有一定了解，且清楚使用Docker進(jìn)行部署的優(yōu)勢(shì)。
1.安裝Docker（Mac）
1.1 下載 Docker for Mac
地址：https://store.docker.com/editions/community/docker-ce-desktop-mac
1.2 下載完成以后呜叫，雙擊打開(kāi)文件Docker.dmg
1.3雙擊Docker.app啟動(dòng)
Mac頂部狀態(tài)欄會(huì)出現(xiàn)鯨魚(yú)圖標(biāo)
1.4點(diǎn)擊鯨魚(yú)圖標(biāo)可以進(jìn)行設(shè)置
1.5 Check versions
$ docker --version
Docker version 18.03, build c97c6d6
$ docker-compose --version
docker-compose version 1.21.2, build 8dd22a9
$ docker-machine --version
docker-machine version 0.14.0, build 9ba6da9
1.6 Hello Word
1.6.1 打開(kāi)命令行終端，通過(guò)運(yùn)行簡(jiǎn)單的Docker映像測(cè)試您的安裝工作赡矢。
$ docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
ca4f61b1923c: Pull complete
Digest: sha256:ca0eeb6fb05351dfc8759c20733c91def84cb8007aa89a5bf606bc8b315b9fc7
Status: Downloaded newer image for hello-world:latest
Hello from Docker!
This message shows that your installation appears to be working correctly.
...
1.6.2 啟動(dòng)Dockerized web server
$ docker run -d -p 80:80 --name webserver nginx
1.6.3 打開(kāi)瀏覽器溅潜，輸入http://localhost/
常用命令：
docker ps 查看正在運(yùn)行的容器
docker stop停止正在運(yùn)行的容器
docker start啟動(dòng)容器
docker ps -a查看終止?fàn)顟B(tài)的容器
docker rm -f webserver命令來(lái)移除正在運(yùn)行的容器
docker list 列出本地鏡像
docker rmi 刪除的鏡像
2.使用Docker安裝Nginx
Docker Store 地址:https://store.docker.com/images/nginx
其實(shí)在上文中Hello World即已經(jīng)安裝了nginx。
2.1 拉取 image
docker pull nginx
3.2 創(chuàng)建Nginx容器
docker run --name mynginx -p 80:80 -v /Users/gaoguangchao/Work/opt/local/nginx/logs:/var/log/nginx -v /Users/gaoguangchao/Work/opt/local/nginx/conf.d:/etc/nginx/conf.d -v /Users/gaoguangchao/Work/opt/local/nginx/nginx.conf:/etc/nginx/nginx.conf:ro -v /Users/gaoguangchao/Work/opt/local/nginx/html:/etc/nginx/html -d nginx
-d 以守護(hù)進(jìn)程運(yùn)行(運(yùn)行在后臺(tái))
--name nginx 容器名稱(chēng);
-p 80:80 端口映射
-v 配置掛載路徑 宿主機(jī)路徑:容器內(nèi)的路徑
關(guān)于掛載
- 為了能直接修改配置文件抑诸，以實(shí)現(xiàn)對(duì)Nginx的定制化烂琴，需要進(jìn)行Docker的相關(guān)目錄掛在宿主機(jī)上。
- 需要掛載的目錄/文件：
/etc/nginx/conf.d/etc/nginx/nginx.conf/etc/nginx/html
- 需要掛載的目錄/文件：
-
- 有一點(diǎn)尤其需要注意蜕乡，當(dāng)掛載的為文件而非目錄時(shí)奸绷，需要注意以下兩點(diǎn)：
- a. 掛載文件命令: -v 宿主機(jī)路徑:容器內(nèi)的路徑:ro
- b.宿主機(jī)需要先創(chuàng)建后文件，無(wú)法自動(dòng)創(chuàng)建层玲，反之將報(bào)錯(cuò)
nginx.conf 示例
#user nobody;
worker_processes 1;
#error_log logs/error.log;
#error_log logs/error.log notice;
#error_log logs/error.log info;
#pid logs/nginx.pid;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
#access_log logs/access.log main;
sendfile on;
#tcp_nopush on;
#keepalive_timeout 0;
keepalive_timeout 65;
#gzip on;
upstream demo {
server 127.0.0.1:8080;
}
server {
listen 80;
server_name request_log;
location / {
root html;
#index index.html index.htm;
proxy_connect_timeout 3;
proxy_send_timeout 30;
proxy_read_timeout 30;
proxy_pass http://demo;
}
#error_page 404 /404.html;
# redirect server error pages to the static page /50x.html
#
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
}
2.3 瀏覽器訪(fǎng)問(wèn)
在調(diào)試過(guò)程中往往不會(huì)很順利号醉，這里的技巧是通過(guò)閱讀
error.log中的異常日志進(jìn)行
2.4 配置反向代理
此處是本機(jī)啟動(dòng)一個(gè) SpringBoot web server反症，端口為：8080，瀏覽器訪(fǎng)問(wèn)：http://localhost:8080/index/hello
按照上節(jié)中nginx.conf示例中的配置方式畔派，增加upstream铅碍、server、proxy_pass相關(guān)配置线椰，對(duì)80端口進(jìn)行監(jiān)聽(tīng)胞谈，重啟nginx容器。
docker restart mynginx
瀏覽器訪(fǎng)問(wèn)：http://localhost/index/hello,可以看到正常訪(fǎng)問(wèn)憨愉。
3.使用Docker安裝Openresty
Openresty是在Nginx基礎(chǔ)上做了大量的定制擴(kuò)展烦绳，其安裝過(guò)程和Nginx基本一致。
Docker Store 地址:https://store.docker.com/community/images/openresty/openresty
3.1 拉取 image
docker pull openresty/openresty
3.2 創(chuàng)建Openresty容器
docker run -d --name="openresty" -p 80:80 -v /Users/gaoguangchao/Work/opt/local/openresty/nginx.conf:/usr/local/openresty/nginx/conf/nginx.conf:ro -v /Users/gaoguangchao/Work/opt/local/openresty/logs:/usr/local/openresty/nginx/logs -v /Users/gaoguangchao/Work/opt/local/openresty/conf.d:/etc/nginx/conf.d -v /Users/gaoguangchao/Work/opt/local/openresty/html:/etc/nginx/html openresty/openresty
注意事項(xiàng)和安裝Nginx基本一致配紫，在此不再贅述径密。
4.使用Docker安裝Kafka
Docker Store 地址:https://store.docker.com/community/images/spotify/kafka
4.1 拉取 image
docker pull spotify/kafka
4.2 創(chuàng)建Kafka容器
運(yùn)行命令：
docker run -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=`127.0.0.1` --env ADVERTISED_PORT=9092 spotify/kafka
2181為zookeeper端口，9092為kafka端口
輸出啟動(dòng)日志：
4.3 Check zookeeper是否啟動(dòng)
可以使用一些可視化客戶(hù)端連接端口躺孝，進(jìn)行監(jiān)控睹晒，如zooInspector、Idea Zookeeper Plugin等
5.使用Docker安裝Kafka Manager
Kafka Manager 是Yahoo開(kāi)源的kafka監(jiān)控和配置的web系統(tǒng)括细，可以進(jìn)行kafka的日常監(jiān)控和配置的動(dòng)態(tài)修改伪很。
Docker Store 地址:https://store.docker.com/community/images/sheepkiller/kafka-manager
5.1 拉取 image
docker pull sheepkiller/kafka-manager
5.2 創(chuàng)建Kafka Manager容器
運(yùn)行命令：
docker run -it --rm -p 9000:9000 -e ZK_HOSTS="127.0.0.1:2181" -e APPLICATION_SECRET=letmein sheepkiller/kafka-manager
2181為上節(jié)中部署的zookeeper端口，9000為kafka-manager的web端口
輸出啟動(dòng)日志：
5.3 訪(fǎng)問(wèn)Kafka Manager
瀏覽器訪(fǎng)問(wèn)：http://localhost:9000
按照頁(yè)面上的操作按鈕進(jìn)行kafka集群的注冊(cè)奋单，具體使用方式再次不做詳細(xì)介紹锉试。
注冊(cè)配置后的界面：
6.Kafka消息生產(chǎn)與消費(fèi)
6.1創(chuàng)建maven項(xiàng)目
** pom依賴(lài)**
<dependencies>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>jcl-over-slf4j</artifactId>
<version>${org.slf4j-version}</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-1.2-api</artifactId>
<version>${log4j2-version}</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
<version>${log4j2-version}</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<version>${log4j2-version}</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>${log4j2-version}</version>
</dependency>
<dependency>
<groupId>com.lmax</groupId>
<artifactId>disruptor</artifactId>
<version>3.2.0</version>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>0.10.1.0</version>
</dependency>
</dependencies>
6.2 增加log4j2配置
配置log4j2為能正常打印debug日志，方便進(jìn)行異常排查 （重要)
在resources目錄下增加log4j2.xml文件
<?xml version="1.0" encoding="UTF-8"?>
<configuration status="WARN">
<Properties>
<Property name="pattern_layout">%d %-5p (%F:%L) - %m%n</Property>
<Property name="LOG_HOME">/logs</Property>
</Properties>
<appenders>
<Console name="CONSOLE" target="SYSTEM_OUT">
<PatternLayout pattern="%d %-5p (%F:%L) - %m%n"/>
</Console>
</appenders>
<loggers>
<root level="debug" includeLocation="true">
<appender-ref ref="CONSOLE"/>
</root>
</loggers>
</configuration>
關(guān)于log4j2的使用览濒，有興趣的可以了解：Log4j1升級(jí)Log4j2實(shí)戰(zhàn)
6.3 創(chuàng)建生產(chǎn)者示例
package com.moko.kafka;
import org.apache.kafka.clients.producer.*;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.Properties;
public class MokoProducer extends Thread {
private static final Logger LOGGER = LoggerFactory.getLogger(MokoProducer.class);
private final KafkaProducer<String, String> producer;
private final String topic;
private final boolean isAsync;
public MokoProducer(String topic, boolean isAsync) {
Properties properties = new Properties();
properties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "78c4f4a0f989:9092");//broker 集群地址
properties.put(ProducerConfig.CLIENT_ID_CONFIG, "MokoProducer");//自定義客戶(hù)端id
properties.put(ProducerConfig.ACKS_CONFIG, "all");
properties.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");//key 序列號(hào)方式
properties.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");//value 序列號(hào)方式
this.producer = new KafkaProducer<String, String>(properties);
this.topic = topic;
this.isAsync = isAsync;
}
@Override
public void run() {
int seq = 0;
while (true) {
String msg = "Msg: " + seq;
if (isAsync) {//異步
producer.send(new ProducerRecord<String, String>(this.topic, msg));
} else {//同步
producer.send(new ProducerRecord<String, String>(this.topic, msg),
new MsgProducerCallback(msg));
}
seq++;
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
/**
* 消息發(fā)送后的回調(diào)函數(shù)
*/
class MsgProducerCallback implements Callback {
private final String msg;
public MsgProducerCallback(String msg) {
this.msg = msg;
}
public void onCompletion(RecordMetadata recordMetadata, Exception e) {
if (recordMetadata != null) {
LOGGER.info(msg + " be sended to partition no : " + recordMetadata.partition());
} else {
LOGGER.info("recordMetadata is null");
}
if (e != null)
e.printStackTrace();
}
}
public static void main(String args[]) {
new MokoProducer("access-log", false).start();//開(kāi)始發(fā)送消息
}
}
簡(jiǎn)單運(yùn)行后呆盖，打印日志如下：
6.4 創(chuàng)建消費(fèi)者示例
package com.moko.kafka;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.Arrays;
import java.util.Properties;
public class MokoCustomer {
private static final Logger LOGGER = LoggerFactory.getLogger(MokoCustomer.class);
public static void main(String args[]) throws Exception {
String topicName = "access-log";
Properties props = new Properties();
KafkaConsumer<String, String> consumer = getKafkaConsumer(props);
consumer.subscribe(Arrays.asList(topicName));
while (true) {
ConsumerRecords<String, String> records = consumer.poll(100);
if (!records.isEmpty()) {
LOGGER.info("=========================");
}
for (ConsumerRecord<String, String> record : records) {
LOGGER.info(record.value());
}
}
}
private static KafkaConsumer<String, String> getKafkaConsumer(Properties props) {
props.put("bootstrap.servers", "172.18.153.41:9092");
props.put("group.id", "group-1");
props.put("enable.auto.commit", "true");
props.put("auto.commit.interval.ms", "1000");
props.put("session.timeout.ms", "30000");
props.put("key.deserializer",
"org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer",
"org.apache.kafka.common.serialization.StringDeserializer");
return new KafkaConsumer<String, String>(props);
}
}
簡(jiǎn)單運(yùn)行后，打印日志如下：
6.5 注意事項(xiàng)
由于是在本機(jī)使用Docker搭建的環(huán)境贷笛，遇到最多的問(wèn)題就是網(wǎng)絡(luò)問(wèn)題应又，如host等的配置，但是只要意識(shí)到這點(diǎn)乏苦，通過(guò)注意分析各種異常日志株扛，便不難排查解決。
7.結(jié)語(yǔ)
致此汇荐，本文就介紹完了如何使用Docker搭建 Nginx/Openresty - Kafka - kafkaManager洞就。
后續(xù)將會(huì)繼續(xù)介紹如何使用Docker搭建一套 nginx+lua+kafka實(shí)現(xiàn)的日志收集的教程，敬請(qǐng)期待掀淘。
歡迎關(guān)注 高廣超的簡(jiǎn)書(shū)博客 與 收藏文章 旬蟋！
歡迎關(guān)注 頭條號(hào)：互聯(lián)網(wǎng)技術(shù)棧 ！
個(gè)人介紹：
高廣超：多年一線(xiàn)互聯(lián)網(wǎng)研發(fā)與架構(gòu)設(shè)計(jì)經(jīng)驗(yàn)革娄，擅長(zhǎng)設(shè)計(jì)與落地高可用倾贰、高性能冕碟、可擴(kuò)展的互聯(lián)網(wǎng)架構(gòu)。
本文首發(fā)在 高廣超的簡(jiǎn)書(shū)博客 轉(zhuǎn)載請(qǐng)注明匆浙！
