前言
因公司需要驗(yàn)證FlinkCDC的能力，而且要求按照最終的生產(chǎn)模式去部署，因此對(duì)安裝過(guò)程做個(gè)記錄躯畴，小白請(qǐng)多指教~
一茫藏、集群規(guī)劃
hadoop01(Master + Slave):JobManager + TaskManager
hadoop02(Master + Slave):JobManager + TaskManager
hadoop03(Slave):TaskManager
二误趴、部署Flink集群
1.版本選擇
地址：https://flink.apache.org/zh/downloads.html
版本：flink-1.14.4-bin-scala_2.12.tgz
2.上傳到hadoop01 主機(jī)
略
3.解壓安裝包
cd /home/hadoop/plat/flink
tar -zxvf flink-1.14.4-bin-scala_2.12.tgz
4.修改配置文件
4.1 flink-conf.yaml
vi /home/hadoop/plat/flink/flink-1.14.4/conf/flink-conf.yaml
基礎(chǔ)配置
# jobManager 的IP地址
jobmanager.rpc.address: hadoop01
# JobManager 的端口號(hào)
jobmanager.rpc.port: 6123
# JobManager JVM heap 內(nèi)存大小
jobmanager.memory.process.size: 1600m
# TaskManager JVM heap 內(nèi)存大小
taskmanager.memory.process.size: 1728m
# 每個(gè) TaskManager 提供的任務(wù) slots 數(shù)量大小
taskmanager.numberOfTaskSlots: 2
# 程序默認(rèn)并行計(jì)算的個(gè)數(shù)
parallelism.default: 1
#支持Web提交Application
web.submit.enable: true
高可用配置信息
# 使用zookeeper搭建高可用
high-availability: zookeeper
# JobManager metadata 的存儲(chǔ)路徑
high-availability.storageDir: hdfs:///flink/ha/
# 配置ZK集群地址
high-availability.zookeeper.quorum: hadoop01:21001,hadoop02:21001,hadoop03:21001
# 保存 存儲(chǔ)和檢查點(diǎn)狀態(tài)的模式
state.backend: filesystem
#啟用 checkpoints 存儲(chǔ) checkpoints 的數(shù)據(jù)文件和元數(shù)據(jù)的默認(rèn)目錄
state.backend.fs.checkpointdir: hdfs://hadoop01:9000/flink/flink-checkpoints
#啟用檢查點(diǎn)，存savepoints的數(shù)據(jù)文件和元數(shù)據(jù)的默認(rèn)目錄 (可選)
state.backend.fs.checkpointdir: hdfs://hadoop01:9000/flink/flink-savepoints
# 基于 Region 的局部重啟
jobmanager.execution.failover-strategy: region
#支持Web提交Application
web.submit.enable: true
HistoryServer
# 保存已完成的作業(yè)的目錄
jobmanager.archive.fs.dir: hdfs:///hadoop01:9000/flink/completed-jobs/
#基于 Web 的 HistoryServer 的地址
historyserver.web.address: hadoop01
# 基于 Web 的 HistoryServer 的端口號(hào)
historyserver.web.port: 8082
# 以逗號(hào)分隔的目錄列表务傲，用于監(jiān)視已完成的作業(yè)
historyserver.archive.fs.dir: hdfs:///hadoop01:9000/flink/completed-jobs/
# 刷新受監(jiān)控目錄的時(shí)間間隔（以毫秒為單位）
historyserver.archive.fs.refresh-interval: 10000
4.2 masters
vi /home/hadoop/plat/flink/flink-1.14.4/conf/masters
hadoop01:8081
4.3 workers
vi /home/hadoop/plat/flink/flink-1.14.4/conf/workers
hadoop01
hadoop02
hadoop03
3.啟動(dòng)Flink集群
1.環(huán)境變量配置
### Flink
export FLINK_HOME=/home/hadoop/plat/flink/flink-1.14.4
export PATH=${FLINK_HOME}/bin:${PATH}
start-cluster.sh
三凉当、問(wèn)題記錄
1.啟動(dòng)后jps沒(méi)有進(jìn)程，沒(méi)有起來(lái)
查看master節(jié)點(diǎn)日志 /home/hadoop/plat/flink/flink-1.14.4/log/flink-hadoop-standalonesession-2-hadoop01.log 有如下報(bào)錯(cuò)信息
解決方案：下載flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar放到${FLINK_HOME}/lib
地址：https://mvnrepository.com/artifact/org.apache.flink/flink-shaded-hadoop-3-uber?repo=cloudera-repos
2 日志有報(bào)錯(cuò)
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.commons.cli.Option.builder(Ljava/lang/String;)Lorg/apache/commons/cli/Option$Builder;
解決方案：下載commons-cli-1.5.0.jar放到${FLINK_HOME}/lib
地址：https://mvnrepository.com/artifact/commons-cli/commons-cli
3 報(bào)錯(cuò)
執(zhí)行 flink run -m yarn-cluster ./examples/batch/WordCount.jar
報(bào)錯(cuò)信息如下：
Exception in thread "Thread-5" java.lang.IllegalStateException: Trying to access closed classloader. Please check if you store classloaders directly or indirectly in static fields. If the stacktrace suggests that the leak occurs in a third party library and cannot be fixed immediately, you can disable this check with the configuration 'classloader.check-leaked-classloader'.
參考帖子
https://blog.csdn.net/wtl1992/article/details/121307695
https://blog.csdn.net/u013982921/article/details/96428258
https://blog.csdn.net/momentni/article/details/114637659
