起因: 本地搭建了集群, 想著擴(kuò)充節(jié)點(diǎn)數(shù)目, 就復(fù)制了安裝目錄, 改了nodename, 以及port等, 刪除data目錄. 但是操作失誤, 最終還是導(dǎo)致腦裂了, 三個(gè)master節(jié)點(diǎn), 其中有一個(gè)data被刪除了, 另外兩個(gè)竟然分別屬于不同的集群, 無(wú)法選主...

這種情況下, 雖然 data都在, 都存活, 但是集群已經(jīng)不能提供管理服務(wù)了. 而就是刪除了所有的master節(jié)點(diǎn), 重建一個(gè)master集群, 也不能和原先的data節(jié)點(diǎn)們組成集群. 這樣元數(shù)據(jù)就丟失了...

參考:
類(lèi)似場(chǎng)景:
https://discuss.elastic.co/t/how-to-recover-cluster-when-2-master-nodes-have-been-lost/269472/2
https://discuss.elastic.co/t/recover-data-after-the-lost-of-master-7-1-1/187775/5
工具說(shuō)明鏈接:
tool-1 https://www.elastic.co/guide/en/elasticsearch/reference/current/node-tool.html#node-tool-detach-cluster

半數(shù)以上master節(jié)點(diǎn)丟失(確定無(wú)法恢復(fù))

from tool-1

If there is at least one remaining master-eligible node, but it is not possible to restart a majority of them, then the elasticsearch-node unsafe-bootstrap command will unsafely override the cluster’s voting configuration as if performing another cluster bootstrapping process. The target node can then form a new cluster on its own by using the cluster metadata held locally on the target node.

如果集群中還剩半數(shù)以下 的master-eligible, 無(wú)論它的元數(shù)據(jù)是否完整, 是否最新. 我們都可以基于它來(lái)恢復(fù)之前的集群(可能有元數(shù)據(jù)丟失, 落后等, 所以是unsafe), 不過(guò)在執(zhí)行的時(shí)候, 還是可以看下, 哪個(gè)剩余節(jié)點(diǎn)的元數(shù)據(jù)更新, 通過(guò)term和version判斷. 這兩個(gè)值在執(zhí)行工具命令的時(shí)候, 會(huì)看到(見(jiàn)下圖unsafe-bootstrap)

步驟

1. Make sure you have really lost access to at least half of the master-eligible nodes in the cluster, and they cannot be repaired or recovered by moving their data paths to healthy hardware.
2. Stop all remaining nodes.
3. Choose one of the remaining master-eligible nodes to become the new elected master as described above.
4. On this node, run the elasticsearch-node unsafe-bootstrap command as shown below. Verify that the tool reported Master node was successfully bootstrapped.
5. Start this node and verify that it is elected as the master node.
6. Run the elasticsearch-node detach-cluster tool, described below, on every other node in the cluster.
7. Start all other nodes and verify that each one joins the cluster.
8. Investigate the data in the cluster to discover if any was lost during this process.

注意事項(xiàng) :

1. 關(guān)閉 security 功能. (如果開(kāi)啟了, 最好先關(guān)閉security插件功能
   1. 因?yàn)樾聠?dòng)的master節(jié)點(diǎn), 沒(méi)有數(shù)據(jù)節(jié)點(diǎn)(如果只配置了一個(gè)master的角色), 那security插件的初始化無(wú)法完成, 各類(lèi)接口不好調(diào)用
   2. 如果給新啟動(dòng)的mster節(jié)點(diǎn), 配置了master and data, 則security插件會(huì)初始化成功. 會(huì)插入index, 但是這個(gè)index會(huì)和原來(lái)的data節(jié)點(diǎn)上保存的沖突. 不知道怎么解.
2. 能成功, 不代表沒(méi)有數(shù)據(jù)丟失.
3. 在unsafe-bootstrap的節(jié)點(diǎn), 必須是關(guān)閉的
4. 所有其他或者的master-eligible節(jié)點(diǎn)(如果有) 必須是關(guān)閉的. all other master-eligible nodes are also stopped while this tool is running, but the tool does not check this.

unsafe-bootstrap.png

detach.png

所有master節(jié)點(diǎn)丟失(確定無(wú)法恢復(fù))

而, 如果所有的master節(jié)點(diǎn)都丟了, 且確定恢復(fù)不了了. 那只能把原來(lái)的data節(jié)點(diǎn)們分離出來(lái), 加入到一個(gè)新的空集群, 并且fix dangling index.

步驟 和之前類(lèi)似, 不過(guò)是啟動(dòng)一個(gè)新集群.

1. Make sure you have really lost access to every one of the master-eligible nodes in the cluster, and they cannot be repaired or recovered by moving their data paths to healthy hardware.
2. Start a new cluster and verify that it is healthy. This cluster may comprise one or more brand-new master-eligible nodes, or may be an unsafely-bootstrapped cluster formed as described above.
3. Stop all remaining data nodes.
4. On each data node, run the elasticsearch-node detach-cluster tool as shown below. Verify that the tool reported Node was successfully detached from the cluster.
5. If necessary, configure each data node to discover the new cluster.
6. Start each data node and verify that it has joined the new cluster.
7. Wait for all recoveries to have completed, and investigate the data in the cluster to discover if any was lost during this process. Use the Dangling indices API to list, import or delete any dangling indices.

注意事項(xiàng)
與前面略同.

新集群無(wú)法啟動(dòng)排查

無(wú)法啟動(dòng), 可能是因?yàn)樾录翰粔蛐? (雖然它可能是從舊的master-eligible節(jié)點(diǎn)的元數(shù)據(jù)而來(lái))

1. 關(guān)閉 security 安全插件 , 最好, 方便排查
2. 有些來(lái)自舊集群的persistence設(shè)置, 導(dǎo)致無(wú)法啟動(dòng)
   1. 使用elasticsearch-node remove-settings
   2. 使用 elasticsearch-node remove-custom