Docker-Swarm介紹

官網是最好的老師 https://docs.docker.com/engine/swarm/

Cassandra介紹

官網是最好的老師 https://cassandra.apache.org/_/index.html

在Docker-Swarm中部署Cassandra集群

如果沒有穩(wěn)定可靠的分布式文件系統(tǒng)，這就限制很多，而且使用Docker-Swarm的意義并不是特別大腰涧，除非有像我司這種需要交付無運維的使用方，需要用到它的多服務器編排记某。

注意Ｃ苋巍Ｒ愫瘛撒踪！請仔細閱讀下面的主要大坑描述９А！制妄！
注意５Ы省！忍捡！請仔細閱讀下面的主要大坑描述＜！砸脊！
注意！Ｎ诚肌凌埂！請仔細閱讀下面的主要大坑描述！Ｊ摺瞳抓！

網上有很多的解決方案，我?guī)缀醵纪昕戳朔郑瑳]有一個是能交付生產使用的孩哑。主要問題出現在網絡的選擇上。多次踩坑之后我們選擇了使用HostNetwork來進行部署翠桦，如果你們介意使用這個網絡的話可以去看看其他的教程横蜒，等出現各種奇葩問題之后再回來看下面的內容也不遲。

使用非HostNetwork會遇到的坑

• overlay網絡無法固定IP销凑，CASSANDRA_SEEDS需要一個IP(雖然你可以寫域名丛晌，但是啟動腳本還是會給你轉為IP)，一旦重啟后整個集群狀態(tài)就不可用了斗幼。
• 訪問Cassandra的時候需要連接將Cassandra的域名轉為IP去連接澎蛛，IP不固定會拋出Cassandra連不上的問題，除非你監(jiān)聽域名的變更蜕窿，這樣的話Cassandra的封裝底層編碼會更加復雜谋逻。
• 當Cassandra調度飄移后會出現內部CASSANDRA_SEEDS訪問不到的問題呆馁，整個集群的數據同步也會出現各種問題，最差的情況就是變?yōu)槿齻€獨立的Cassandra毁兆。

開始部署Cassandra集群

對Swarm各節(jié)點進行編號

docker node ls
docker node update --label-add cassandra1=true node1 # 確定將Cassandra1調度到哪臺機器節(jié)點
docker node update --label-add cassandra2=true node2 # 確定將Cassandra2調度到哪臺機器節(jié)點
docker node update --label-add cassandra3=true node3 # 確定將Cassandra3調度到哪臺機器節(jié)點

在三臺機器上面建立文件掛載文件夾

這一步如果不做的話會出現問題

sh ./gendir.sh

#! /bin/bash
if [ ! -d "/data/cassandra" ]; then
    mkdir /data/cassandra
    mkdir /data/cassandra/_data
fi

構建編排文件

這里需要將文件命名為docker-compose.template.yml因為后續(xù)我們需要用shell將主機真實IP打入編排文件中浙滤。當然，使用makefile這種方式去做也行荧恍，但是我個人比較喜歡使用簡單方便的shell腳本去實現這一步瓷叫。

cat docker-compose.template.yml

version: '3.8'
services:
  # Cassandra
  cassandra1:
    image: cassandra:4.0.0
    hostname: cassandra1
    cap_add:
      - SYS_NICE
    environment:
      - CASSANDRA_CLUSTER_NAME=cassandra
      - CASSANDRA_SEEDS=${CASSANDRA_SEEDS}
      - JVM_OPTS=-Xmx6144m -Xms2048m # 限制內存大小
    networks:
      host_network:
    volumes:
      - cassandra1_data:/var/lib/cassandra
    deploy:
      mode: replicated
      replicas: 1
      resources:
        limits:
          memory: 6G
        reservations:
          memory: 2G
      placement:
        max_replicas_per_node: 1
        constraints:
          - node.labels.cassandra1==true
      restart_policy:
        condition: on-failure
        delay: 5s
        # max_attempts: 3
        window: 120s
  cassandra2:
    image: cassandra:4.0.0
    hostname: cassandra2
    cap_add:
      - SYS_NICE
    environment:
      - CASSANDRA_CLUSTER_NAME=cassandra
      - CASSANDRA_SEEDS=${CASSANDRA_SEEDS}
      - JVM_OPTS=-Xmx6144m -Xms2048m # 限制內存大小
    networks:
      host_network:
    volumes:
      - cassandra2_data:/var/lib/cassandra
    deploy:
      mode: replicated
      replicas: 1
      resources:
        limits:
          memory: 6G
        reservations:
          memory: 2G
      placement:
        max_replicas_per_node: 1
        constraints:
          - node.labels.cassandra2==true
      restart_policy:
        condition: on-failure
        delay: 5s
        # max_attempts: 3
        window: 120s
  cassandra3:
    image: cassandra:4.0.0
    hostname: cassandra3
    cap_add:
      - SYS_NICE
    environment:
      - CASSANDRA_CLUSTER_NAME=cassandra
      - CASSANDRA_SEEDS=${CASSANDRA_SEEDS}
      - JVM_OPTS=-Xmx6144m -Xms2048m # 限制內存大小
    networks:
      host_network:
    volumes:
      - cassandra3_data:/var/lib/cassandra
    deploy:
      mode: replicated
      replicas: 1
      resources:
        limits:
          memory: 6G
        reservations:
          memory: 2G
      placement:
        max_replicas_per_node: 1
        constraints:
          - node.labels.cassandra3==true
      restart_policy:
        condition: on-failure
        delay: 5s
        # max_attempts: 3
        window: 120s

networks:
  host_network:
    name: host
    external: true
    attachable: true

volumes:
  cassandra1_data:
    driver: local
    driver_opts:
      type: none
      o: bind
      device: /data/cassandra/_data
  cassandra2_data:
    driver: local
    driver_opts:
      type: none
      o: bind
      device: /data/cassandra/_data
  cassandra3_data:
    driver: local
    driver_opts:
      type: none
      o: bind
      device: /data/cassandra/_data

構建啟動腳本

cat ./app.sh

#! /bin/bash

p_name="cassandra"
script=${1:-"status"}

get_lable_addr() {
    nodes=$(docker node ls -q | xargs docker node inspect -f '{{ .Description.Hostname }}:{{ .Status.Addr}}:{{ range $k, $v := .Spec.Labels }}{{ $k }}={{ $v }} {{end}}' | grep cassandra | grep ${1} | awk -F ":" '{print $2}')
    ip_addr=''
    for node in ${nodes[@]}; do
        # tmp=$(ping ${node} -c 1 | sed '1{s/[^(]*(//;s/).*//;q}')
        tmp=${node}
        if [ ! -z ${tmp} ]; then
            if [ ! -z ${ip_addr} ]; then
                ip_addr="${ip_addr},${tmp}"
            else
                ip_addr=${tmp}
            fi
        fi
    done
    echo ${ip_addr}
    return $?
}

create() {
    compose_path="$(pwd)/docker-compose.yml"
    cp ./docker-compose.template.yml ./docker-compose.yml
    cassandra_ip=$(get_lable_addr cassandra)
    echo "$(sed "s/\${CASSANDRA_SEEDS}/${cassandra_ip}/" ./docker-compose.yml)" >./docker-compose.yml
}

start() {
    docker stack deploy -c ./docker-compose.yml --with-registry-auth ${p_name}
}

stop() {
    docker stack rm ${p_name}
}

status() {
    docker stack services ${p_name}
}

ps() {
    docker stack ps --no-trunc ${p_name}
}

update() {
    docker stack deploy --prune -c ./docker-compose.yml --with-registry-auth ${p_name}
}

main() {
    if [ ${script} == "start" ]; then
        start
    elif [ ${script} == "stop" ]; then
        stop
    elif [ ${script} == "update" ]; then
        update
    elif [ ${script} == "status" ]; then
        status
    elif [ ${script} == "ps" ]; then
        ps
    elif [ ${script} == "create" ]; then
        create
    else
        echo 'Instruction does not exist'
    fi
}

main

開放防火墻

由于使用的是HostNetwork所以防火墻需要自行解決
構建防火墻開放文件 open-firewall.sh

cat ./open-firewall.sh

#! /bin/bash

script=${1:-"status"}
directional=${2}

openports=("7000" "9042")

open_firewall_ip() {
    directionalArr=($(echo ${directional} | tr ',' ' '))
    for port in ${openports[@]}; do
        for d in ${directionalArr[@]}; do
            echo "Open firewall -> [$qabe1ba:${port}]"
            firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="$ws5oyt1" port protocol="tcp" port="${port}" accept"
        done
    done
    firewall-cmd --reload
}

close_firewall_ip() {
    directionalArr=($(echo ${directional} | tr ',' ' '))
    for port in ${openports[@]}; do
        for d in ${directionalArr[@]}; do
            echo "Close firewall -> [$abm0xxl:${port}]"
            firewall-cmd --permanent --remove-rich-rule="rule family="ipv4" source address="$36cdccc" port protocol="tcp" port="${port}" accept"
        done
    done
    firewall-cmd --reload
}

open_firewall() {
    for port in ${openports[@]}; do
        echo "Open firewall -> [0.0.0.0:${port}]"
        firewall-cmd --zone=public --add-port=${port}/tcp --permanent
    done
    firewall-cmd --reload
}

close_firewall() {
    for port in ${openports[@]}; do
        echo "Close firewall -> [0.0.0.0:${port}]"
        firewall-cmd --zone=public --remove-port=${port}/tcp --permanent
    done
    firewall-cmd --reload
}

main() {
    if [ ${script} == "close" ]; then
        if [ ! -z ${directional} ]; then
            close_firewall_ip
        else
            close_firewall
        fi
    elif [ ${script} == "open" ]; then
        if [ ! -z ${directional} ]; then
            open_firewall_ip
        else
            open_firewall
        fi
    elif [ ${script} == "status" ]; then
        firewall-cmd --list-rich-rules
        firewall-cmd --list-ports
    else
        echo 'Instruction does not exist'
    fi
}

main

執(zhí)行啟動命令

sh ./open-firewall.sh # 查看目前已經開放的端口

sh ./open-firewall.sh open # 開放端口
sh ./open-firewall.sh open 192.168.1.11 # 定向開放端口
sh ./open-firewall.sh open 192.168.1.11,192.168.1.12 # 定向向多個IP開放端口

sh ./open-firewall.sh close # 關閉端口
sh ./open-firewall.sh close 192.168.1.142 # 關閉定向端口
sh ./open-firewall.sh close 192.168.1.142,0.0.0.0 # 關閉多個定向端口

啟動Cassandra

sh ./app.sh create # 根據模板創(chuàng)建編排文件
sh ./app.sh start # 啟動
sh ./app.sh stop # 停止
sh ./app.sh status # 查看狀態(tài)
sh ./app.sh ps # 查看詳情