Dubbo在Docker中的優(yōu)雅停機(jī)
優(yōu)雅停機(jī)
優(yōu)雅停機(jī)是指在停止應(yīng)用時(shí)栋荸，執(zhí)行的一系列保證應(yīng)用正常關(guān)閉的操作穆端。這些操作往往包括等待已有請(qǐng)求執(zhí)行完成、關(guān)閉線程鸠真、關(guān)閉連接和釋放資源等诗宣，優(yōu)雅停機(jī)可以避免非正常關(guān)閉程序可能造成數(shù)據(jù)異潮炫拢或丟失，應(yīng)用異常等問題召庞。優(yōu)雅停機(jī)本質(zhì)上是JVM即將關(guān)閉前執(zhí)行的一些額外的處理代碼岛心。這個(gè)功能官方是支持的来破，只要正常kill SIGTERM或SIGIN 就可以。
Dubbo服務(wù)關(guān)閉流程
Provider在接收到停機(jī)指令后
- 從注冊(cè)中心上注銷所有服務(wù)忘古；
- 從配置中心取消監(jiān)聽動(dòng)態(tài)配置徘禁；
- 向所有連接的客戶端發(fā)送只讀事件，停止接收新請(qǐng)求髓堪；
- 等待一段時(shí)間以處理已到達(dá)的請(qǐng)求送朱，然后關(guān)閉請(qǐng)求處理線程池；
- 斷開所有客戶端連接旦袋。
Consumer在接收到停機(jī)指令后
- 拒絕新到請(qǐng)求骤菠，直接返回調(diào)用異常；
- 等待當(dāng)前已發(fā)送請(qǐng)求執(zhí)行完畢疤孕，如果響應(yīng)超時(shí)則強(qiáng)制關(guān)閉連接。
源碼分析
參考Dubbo版本2.7.6
- Provider 啟動(dòng)時(shí)注冊(cè)鉤子
private DubboBootstrap() {
DubboShutdownHook.getDubboShutdownHook().register();
ShutdownHookCallbacks.INSTANCE.addCallback(new ShutdownHookCallback() {
@Override
public void callback() throws Throwable {
DubboBootstrap.this.destroy();
}
});
}
public void register() {
if (registered.compareAndSet(false, true)) {
DubboShutdownHook dubboShutdownHook = getDubboShutdownHook();
Runtime.getRuntime().addShutdownHook(dubboShutdownHook);
dispatch(new DubboShutdownHookRegisteredEvent(dubboShutdownHook));
}
}
- 收到退出信號(hào)
//執(zhí)行銷毀
DubboBootstrap.java
public void destroy() {
if (destroyLock.tryLock()) {
try {
DubboShutdownHook.destroyAll();
.
.
.
} finally {
destroyLock.unlock();
}
}
}
//銷毀
DubboShutdownHook.java
public static void destroyAll() {
if (destroyed.compareAndSet(false, true)) {
AbstractRegistryFactory.destroyAll();
destroyProtocols();
}
}
//從注冊(cè)中心刪除央拖，并取消監(jiān)聽
AbstractRegistry.java
public void destroy() {
//刪除節(jié)點(diǎn)
Set<URL> destroyRegistered = new HashSet<>(getRegistered());
if (!destroyRegistered.isEmpty()) {
for (URL url : new HashSet<>(getRegistered())) {
if (url.getParameter(DYNAMIC_KEY, true)) {
try {
unregister(url);
if (logger.isInfoEnabled()) {
logger.info("Destroy unregister url " + url);
}
} catch (Throwable t) {
logger.warn("Failed to unregister url " + url + " to registry " + getUrl() + " on destroy, cause: " + t.getMessage(), t);
}
}
}
}
//取消監(jiān)聽
Map<URL, Set<NotifyListener>> destroySubscribed = new HashMap<>(getSubscribed());
if (!destroySubscribed.isEmpty()) {
for (Map.Entry<URL, Set<NotifyListener>> entry : destroySubscribed.entrySet()) {
URL url = entry.getKey();
for (NotifyListener listener : entry.getValue()) {
try {
unsubscribe(url, listener);
} catch (Throwable t) {
logger.warn("Failed to unsubscribe url " + url + " to registry " + getUrl() + " on destroy, cause: " + t.getMessage(), t);
}
}
}
}
AbstractRegistryFactory.removeDestroyedRegistry(this);
}
//注銷協(xié)議
DubboShutdownHook.java
public static void destroyProtocols() {
ExtensionLoader<Protocol> loader = ExtensionLoader.getExtensionLoader(Protocol.class);
for (String protocolName : loader.getLoadedExtensions()) {
try {
Protocol protocol = loader.getLoadedExtension(protocolName);
if (protocol != null) {
protocol.destroy();
}
} catch (Throwable t) {
logger.warn(t.getMessage(), t);
}
}
}
docker容器中Dubbo不優(yōu)雅停機(jī)
按理說祭阀，我們不用做任何干預(yù)就可以實(shí)現(xiàn)優(yōu)雅停機(jī)了，但是實(shí)際上卻是Dubbo Provider服務(wù)重啟時(shí)鲜戒，總是能收到如下告警专控，且服務(wù)關(guān)閉慢。
Connection refused: /192.168.1.112:20880
at com.alibaba.dubbo.remoting.transport.netty.NettyClient.doConnect(NettyClient.java:124)
at com.alibaba.dubbo.remoting.transport.AbstractClient.connect(AbstractClient.java:280)
at com.alibaba.dubbo.remoting.transport.AbstractClient$1.run(AbstractClient.java:145)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused: /192.168.1.112:20880
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 common frames omitted
通過分析得知是Dockerfile中啟動(dòng)java進(jìn)程命令導(dǎo)致的遏餐，原始：
CMD java ${CMD_JAVA_ARGS} -jar /opt/app/application.jar 2>&1
改為
CMD ["java",$CMD_JAVA_ARGS,"-jar","/opt/app/${JobFile}","2>&1"]
兩種CMD啟動(dòng)方式
CMD ["executable","param1","param2"] (直接啟動(dòng)程序伦腐，這是推薦Docker官方推薦用法)
CMD command param1 param2 （通過shell啟動(dòng)）
按照方式一 ,進(jìn)程結(jié)構(gòu)為：
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 Apr17 ? 00:11:21 java -jar application.jar
按照方式二，進(jìn)程結(jié)構(gòu)為
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 Mar24 ? 00:00:00 /bin/sh -c java -jar application.jar
root 5 1 0 Mar24 ? 01:37:21 java -jar application.jar
此時(shí)失都，我們的sh進(jìn)程是我們的java程序的父進(jìn)程柏蘑。
Docker停止容器
docker stop，當(dāng)我們用docker stop命令來停掉容器的時(shí)候粹庞，docker默認(rèn)會(huì)允許容器中的應(yīng)用程序有10秒的時(shí)間用以終止運(yùn)行咳焚。在docker stop命令執(zhí)行的時(shí)候，會(huì)先向容器中PID為1的進(jìn)程發(fā)送系統(tǒng)信號(hào)SIGTERM庞溜，然后等待容器中的應(yīng)用程序終止執(zhí)行革半，如果等待時(shí)間達(dá)到設(shè)定的超時(shí)時(shí)間，或者默認(rèn)的10秒流码，會(huì)繼續(xù)發(fā)送SIGKILL的系統(tǒng)信號(hào)強(qiáng)行kill掉進(jìn)程又官。在容器中的應(yīng)用程序，可以選擇忽略和不處理SIGTERM信號(hào)漫试，不過一旦達(dá)到超時(shí)時(shí)間六敬，程序就會(huì)被系統(tǒng)強(qiáng)行kill掉，因?yàn)镾IGKILL信號(hào)是直接發(fā)往系統(tǒng)內(nèi)核的商虐，應(yīng)用程序沒有機(jī)會(huì)去處理它觉阅。
那么問題來了崖疤，對(duì)于啟動(dòng)方式二，我們的bash進(jìn)程是1號(hào)進(jìn)程典勇，但它并不會(huì)將SIGTERM信號(hào)傳遞給我們的java進(jìn)程劫哼，我們就沒有機(jī)會(huì)進(jìn)行優(yōu)雅停機(jī)了。
總結(jié)
Dubbo官方是支持優(yōu)雅停機(jī)的割笙，Docker官方也是支持優(yōu)雅停機(jī)的权烧，但是使用需掌握正確的姿勢(shì)。
參考：
https://dubbo.apache.org/zh-cn/blog/dubbo-gracefully-shutdown.html
https://www.infoq.cn/article/2016/01/dumb-init-Docker
https://yeasy.gitbooks.io/docker_practice/image/dockerfile/cmd.html
https://xiaozhou.net/stop-docker-container-gracefully-2016-09-08.html
