90905-centos7-flink1.9.0-hadoop2.7.3-源碼編譯
編譯環(huán)境
| 類(lèi)型 | 版本 | 備注 |
|---|---|---|
| OS | centos el7.x86_64 | |
| maven | 3.6.1 | |
| node | v10.14.1 | |
| flink 源碼 | release-1.9.0 | |
| hadoop | 2.7.3.2.6.0.3-8 | |
| ambari-hdp | 2.6.0.3-8 | |
| scala | 2.12 |
編譯過(guò)程（環(huán)境構(gòu)建參考下文）
- flink-release-1.9.0 源碼
cd ~
# 打開(kāi)下面鏈接下載最新的 1.9.0 源碼 （不要下載 release 的 zip 包）
https://codeload.github.com/apache/flink/zip/release-1.9.0
unzip flink-release-1.9.0.zip
- 源碼預(yù)處理
# 進(jìn)入源碼目錄
cd flink-release-1.9.0
# pom 中 移除 test 相關(guān)模塊
vim pom.xml
# 移除模塊如下
<module>flink-tests</module>
<module>flink-test-utils-parent</module>
<module>flink-end-to-end-tests</module>
<module>flink-yarn-tests</module>
<module>flink-fs-tests</module>
<module>flink-docs</module>
# 修改pom 文件
cd flink-filesystems/flink-s3-fs-hadoop
cd flink-filesystems/flink-oss-fs-hadoop
vim pom.xml
# 刪除 flink-fs-hadoop-shaded 下
<scope>test</scope>
<type>test-jar</type>
# vim flink-connectors/flink-hadoop-compatibility/pom.xml
<dependency>
<groupId>commons-cli</groupId>
<artifactId>commons-cli</artifactId>
<version>1.4</version>
</dependency>
# vim flink-connectors/pom.xml
<dependency>
<groupId>commons-net</groupId>
<artifactId>commons-net</artifactId>
<version>3.6</version>
</dependency>
# nodejs 相關(guān)處理
cd flink-runtime-web/web-dashboard
npm ci --cache-max=0 --no-save
rm -rf node_modules/caniuse-lite
rm -rf node_modules/browserslist
npm update
- 編譯
cd flink-release-1.9.0
mvn clean install -Dmaven.test.skip=true -Dcheckstyle.skip=true -Dlicense.skip=true -Drat.skip=true -Drat.ignoreErrors=true -Dfast -Dscala-2.12 -Dhadoop.version=2.7.3
# 如果編譯錯(cuò)誤會(huì)提示 mvn <goals> -rf :flink-hadoop-fs (...) 封寞，這樣寫(xiě)繼續(xù)編譯
mvn clean install -Dmaven.test.skip=true -Dcheckstyle.skip=true -Dlicense.skip=true -Drat.skip=true -Drat.ignoreErrors=true -Dfast -Dscala-2.12 -Dhadoop.version=2.7.3 -rf :flink-hadoop-fs
# 編譯成功后顯示如下結(jié)果
[INFO] flink-metrics-graphite ............................. SUCCESS [ 0.239 s]
[INFO] flink-metrics-influxdb ............................. SUCCESS [ 1.130 s]
[INFO] flink-metrics-prometheus ........................... SUCCESS [ 0.666 s]
[INFO] flink-metrics-statsd ............................... SUCCESS [ 0.264 s]
[INFO] flink-metrics-datadog .............................. SUCCESS [ 0.420 s]
[INFO] flink-metrics-slf4j ................................ SUCCESS [ 0.247 s]
[INFO] flink-python ....................................... SUCCESS [ 0.969 s]
[INFO] flink-dist ......................................... SUCCESS [ 19.897 s]
[INFO] flink-yarn-tests ................................... SUCCESS [ 1.174 s]
[INFO] flink-ml-parent .................................... SUCCESS [ 0.068 s]
[INFO] flink-ml-api ....................................... SUCCESS [ 0.413 s]
[INFO] flink-ml-lib ....................................... SUCCESS [ 0.390 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 11:25 min
[INFO] Finished at: 2019-09-05T20:28:00+08:00
[INFO] ------------------------------------------------------------------------
- 遇到問(wèn)題及解決方案
# 通過(guò) mvn 命令手動(dòng)添加到本地倉(cāng)庫(kù) 下載 flink-shaded-hadoop-2-2.7.5-7.0.jar 等包
# 注意 不要放在 flink 源碼工程中執(zhí)行
- 4.1 Could not find artifact org.apache.flink:flink-shaded-hadoop-2:jar:2.7.3-7.0
flink-hadoop-fs
# 去 maven 中央倉(cāng)庫(kù)搜索 flink-shaded-hadoop-2 , 發(fā)現(xiàn)沒(méi)有我們的版本然评，下載版本最近的 jar
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-shaded-hadoop-2</artifactId>
<version>2.7.5-7.0</version>
<scope>provided</scope>
</dependency>
mvn install:install-file -DgroupId=org.apache.flink -DartifactId=flink-shaded-hadoop-2 -Dversion=2.7.3-7.0 -Dpackaging=jar -Dfile=./flink-shaded-hadoop-2-2.7.5-7.0.jar
- 4.2 Could not find artifact org.apache.flink:flink-fs-hadoop-shaded:jar:tests:1.9.0
flink-s3-fs-hadoop
# 詳細(xì)錯(cuò)誤信息
Failed to execute goal on project flink-s3-fs-hadoop: Could not resolve dependencies for project org.apache.flink:flink-s3-fs-hadoop:jar:1.9.0: Failure to find org.apache.flink:flink-fs-hadoop-shaded:jar:tests:1.9.0 in http://maven.aliyun.com/nexus/content/groups/public was cached in the local repository, resolution will not be reattempted until the update interval of alimaven has elapsed or updates are forced
# pom 文件修改
cd flink-filesystems/flink-s3-fs-hadoop
cd flink-filesystems/flink-oss-fs-hadoop
vim pom.xml
# 刪除 flink-fs-hadoop-shaded 下
<scope>test</scope>
<type>test-jar</type>
- 4.3 Could not find artifact org.apache.flink:flink-shaded-hadoop-2-uber:jar:2.7.3-7.0
flink-yarn-tests
# 去 maven 中央倉(cāng)庫(kù)搜索 flink-shaded-hadoop-2 , 發(fā)現(xiàn)沒(méi)有我們的版本，下載版本最近的 jar
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-shaded-hadoop-2-uber</artifactId>
<version>2.7.5-7.0</version>
<scope>provided</scope>
</dependency>
mvn install:install-file -DgroupId=org.apache.flink -DartifactId=flink-shaded-hadoop-2-uber -Dversion=2.7.3-7.0 -Dpackaging=jar -Dfile=./flink-shaded-hadoop-2-uber-2.7.5-7.0.jar
5 編譯完成后 二進(jìn)制 文件路徑如下
cd flink-release-1.9.0
cd flink-dist/target/flink-1.9.0-bin/flink-1.9.0
環(huán)境構(gòu)建
maven
- 下載安裝包
# 進(jìn)入自己的下載路徑
cd ~
# 下載安裝包
wget http://mirror.olnevhost.net/pub/apache/maven/maven-3/3.6.1/binaries/apache-maven-3.6.1-bin.zip
- 環(huán)境配置
# 解壓
unzip http://mirror.olnevhost.net/pub/apache/maven/maven-3/3.6.1/binaries/apache-maven-3.6.1-bin.zip
# 重命名文件夾
mv apache-maven-3.6.1 maven
# 配置環(huán)境變量
sudo vim /etc/profile
# 添加如下內(nèi)容
export MAVEN_HOME=~/maven
export PATH=$MAVEN_HOME/bin:$PATH
# 刷新環(huán)境變量
source /etc/profile
- 配置 maven 阿里源
# 編輯配置文件
vim ~/maven/conf/settings.xml
# mirrors 節(jié)點(diǎn)加入如下配置項(xiàng)
<mirror>
<id>alimaven</id>
<name>aliyun maven</name>
<url>http://maven.aliyun.com/nexus/content/groups/public</url>
<mirrorOf>central</mirrorOf>
</mirror>
node
- 測(cè)試 maven
# 輸入
mvn
# 看到如下內(nèi)容說(shuō)明環(huán)境配置正常
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.098 s
[INFO] Finished at: 2019-09-05T11:48:27+08:00
[INFO] ------------------------------------------------------------------------
node 環(huán)境安裝
- 環(huán)境安裝
cd ~
wget https://npm.taobao.org/mirrors/node/v10.14.1/node-v10.14.1-linux-x64.tar.gz
tar zxvf node-v10.14.1-linux-x64.tar.gz
mv node-v10.14.1-linux-x64 node
ln -s ~/node/bin/node /usr/local/bin/node
ln -s ~/node/bin/npm /usr/local/bin/npm
#
npm install -g cnpm --registry=https://registry.npm.taobao.org
#
alias cnpm="npm --registry=https://registry.npm.taobao.org \
--cache=$HOME/.npm/.cache/cnpm \
--disturl=https://npm.taobao.org/dist \
--userconfig=$HOME/.cnpmrc"
# 處理 npm 權(quán)限
npm config -g set unsafe-perm
- 測(cè)試
# 輸入
npm
# 顯示如下 則環(huán)境正常
Usage: npm <command>
where <command> is one of:
