flink:第二種模式:多個(gè)yarn session模式

這種方式的好處是一個(gè)任務(wù)會(huì)對(duì)應(yīng)一個(gè)job,即每提交一個(gè)作業(yè)會(huì)根據(jù)自身的情況,向yarn申請(qǐng)資源穴豫,直到作業(yè)執(zhí)行完成铺峭,并不會(huì)影響下一個(gè)作業(yè)的正常運(yùn)行冀自,除非是yarn上面沒(méi)有任何資源的情況下。
注意:client端必須要設(shè)置YARN_CONF_DIR或者HADOOP_CONF_DIR或者HADOOP_HOME環(huán)境變量厨相,通過(guò)這個(gè)環(huán)境變量來(lái)讀取YARN和HDFS的配置信息领曼,否則啟動(dòng)會(huì)失敗
不需要在yarn當(dāng)中啟動(dòng)任何集群,直接提交任務(wù)即可
第一步:直接執(zhí)行命令提交任務(wù)
cd /kkb/install/flink-1.8.1/
bin/flink run -m yarn-cluster -yn 2 -yjm 1024 -ytm 1024 ./examples/batch/WordCount.jar -input hdfs://node01:8020/flink_input -output hdfs://node01:8020/out_result/out_count.txt
第二步:查看輸出結(jié)果
hdfs執(zhí)行以下命令查看輸出結(jié)果
hdfs dfs -text hdfs://node01:8020/out_result/out_count.txt
第三步:查看flink run幫助文檔
我們可以使用--help 來(lái)查看幫助文檔可以添加哪些參數(shù)
cd /kkb/install/flink-1.8.1/
bin/flink run --help
得到結(jié)果內(nèi)容如下
Action "run" compiles and runs a program.

Syntax: run [OPTIONS] <jar-file> <arguments>
"run" action options:
-c,--class <classname> Class with the program entry point
("main" method or "getPlan()" method.
Only needed if the JAR file does not
specify the class in its manifest.
-C,--classpath <url> Adds a URL to each user code
classloader on all nodes in the
cluster. The paths must specify a
protocol (e.g. file://) and be
accessible on all nodes (e.g. by means
of a NFS share). You can use this
option multiple times for specifying
more than one URL. The protocol must
be supported by the {@link
java.net.URLClassLoader}.
-d,--detached If present, runs the job in detached
mode
-n,--allowNonRestoredState Allow to skip savepoint state that
cannot be restored. You need to allow
this if you removed an operator from
your program that was part of the
program when the savepoint was
triggered.
-p,--parallelism <parallelism> The parallelism with which to run the
program. Optional flag to override the
default value specified in the
configuration.
-q,--sysoutLogging If present, suppress logging output to
standard out.
-s,--fromSavepoint <savepointPath> Path to a savepoint to restore the job
from (for example
hdfs:///flink/savepoint-1537).
-sae,--shutdownOnAttachedExit If the job is submitted in attached
mode, perform a best-effort cluster
shutdown when the CLI is terminated
abruptly, e.g., in response to a user
interrupt, such as typing Ctrl + C.
Options for yarn-cluster mode:
-d,--detached If present, runs the job in detached
mode
-m,--jobmanager <arg> Address of the JobManager (master) to
which to connect. Use this flag to
connect to a different JobManager than
the one specified in the
configuration.
-sae,--shutdownOnAttachedExit If the job is submitted in attached
mode, perform a best-effort cluster
shutdown when the CLI is terminated
abruptly, e.g., in response to a user
interrupt, such as typing Ctrl + C.
-yD <property=value> use value for given property
-yd,--yarndetached If present, runs the job in detached
mode (deprecated; use non-YARN
specific option instead)
-yh,--yarnhelp Help for the Yarn session CLI.
-yid,--yarnapplicationId <arg> Attach to running YARN session
-yj,--yarnjar <arg> Path to Flink jar file
-yjm,--yarnjobManagerMemory <arg> Memory for JobManager Container with
optional unit (default: MB)
-yn,--yarncontainer <arg> Number of YARN container to allocate
(=Number of Task Managers)
-ynl,--yarnnodeLabel <arg> Specify YARN node label for the YARN
application
-ynm,--yarnname <arg> Set a custom name for the application
on YARN
-yq,--yarnquery Display available YARN resources
(memory, cores)
-yqu,--yarnqueue <arg> Specify YARN queue.
-ys,--yarnslots <arg> Number of slots per TaskManager
-yst,--yarnstreaming Start Flink in streaming mode
-yt,--yarnship <arg> Ship files in the specified directory
(t for transfer)
-ytm,--yarntaskManagerMemory <arg> Memory per TaskManager Container with
optional unit (default: MB)
-yz,--yarnzookeeperNamespace <arg> Namespace to create the Zookeeper
sub-paths for high availability mode
-z,--zookeeperNamespace <arg> Namespace to create the Zookeeper
sub-paths for high availability mode

Options for default mode:
-m,--jobmanager <arg> Address of the JobManager (master) to which
to connect. Use this flag to connect to a
different JobManager than the one specified
in the configuration.
-z,--zookeeperNamespace <arg> Namespace to create the Zookeeper sub-paths
for high availability mode
3蛮穿、flink run腳本分析
我們提交flink任務(wù)的時(shí)候庶骄,可以加以下這些參數(shù)
1、默認(rèn)查找當(dāng)前yarn集群中已有的yarn-session信息中的jobmanager【/tmp/.yarn-properties-root】:
bin/flink run ./examples/batch/WordCount.jar -input hdfs://hostname:port/hello.txt -output hdfs://hostname:port/result1
2践磅、連接指定host和port的jobmanager:
bin/flink run -m node01:8081 ./examples/batch/WordCount.jar -input hdfs://hostname:port/hello.txt -output hdfs://hostname:port/result1
3单刁、啟動(dòng)一個(gè)新的yarn-session:
bin/flink run -m yarn-cluster -yn 2 ./examples/batch/WordCount.jar -input hdfs://hostname:port/hello.txt -output hdfs://hostname:port/result1
注意:yarn session命令行的選項(xiàng)也可以使用./bin/flink 工具獲得。它們都有一個(gè)y或者yarn的前綴
例如:bin/flink run -m yarn-cluster -yn 2 ./examples/batch/WordCount.jar

?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請(qǐng)聯(lián)系作者
  • 序言:七十年代末府适,一起剝皮案震驚了整個(gè)濱河市羔飞,隨后出現(xiàn)的幾起案子肺樟,更是在濱河造成了極大的恐慌,老刑警劉巖褥傍,帶你破解...
    沈念sama閱讀 218,755評(píng)論 6 507
  • 序言:濱河連續(xù)發(fā)生了三起死亡事件儡嘶,死亡現(xiàn)場(chǎng)離奇詭異,居然都是意外死亡恍风,警方通過(guò)查閱死者的電腦和手機(jī)蹦狂,發(fā)現(xiàn)死者居然都...
    沈念sama閱讀 93,305評(píng)論 3 395
  • 文/潘曉璐 我一進(jìn)店門,熙熙樓的掌柜王于貴愁眉苦臉地迎上來(lái)朋贬,“玉大人凯楔,你說(shuō)我怎么就攤上這事〗跄迹” “怎么了摆屯?”我有些...
    開(kāi)封第一講書(shū)人閱讀 165,138評(píng)論 0 355
  • 文/不壞的土叔 我叫張陵,是天一觀的道長(zhǎng)糠亩。 經(jīng)常有香客問(wèn)我虐骑,道長(zhǎng),這世上最難降的妖魔是什么赎线? 我笑而不...
    開(kāi)封第一講書(shū)人閱讀 58,791評(píng)論 1 295
  • 正文 為了忘掉前任廷没,我火速辦了婚禮,結(jié)果婚禮上垂寥,老公的妹妹穿的比我還像新娘颠黎。我一直安慰自己,他們只是感情好滞项,可當(dāng)我...
    茶點(diǎn)故事閱讀 67,794評(píng)論 6 392
  • 文/花漫 我一把揭開(kāi)白布狭归。 她就那樣靜靜地躺著,像睡著了一般文判。 火紅的嫁衣襯著肌膚如雪过椎。 梳的紋絲不亂的頭發(fā)上,一...
    開(kāi)封第一講書(shū)人閱讀 51,631評(píng)論 1 305
  • 那天戏仓,我揣著相機(jī)與錄音潭流,去河邊找鬼。 笑死柜去,一個(gè)胖子當(dāng)著我的面吹牛灰嫉,可吹牛的內(nèi)容都是我干的。 我是一名探鬼主播嗓奢,決...
    沈念sama閱讀 40,362評(píng)論 3 418
  • 文/蒼蘭香墨 我猛地睜開(kāi)眼讼撒,長(zhǎng)吁一口氣:“原來(lái)是場(chǎng)噩夢(mèng)啊……” “哼!你這毒婦竟也來(lái)了?” 一聲冷哼從身側(cè)響起根盒,我...
    開(kāi)封第一講書(shū)人閱讀 39,264評(píng)論 0 276
  • 序言:老撾萬(wàn)榮一對(duì)情侶失蹤钳幅,失蹤者是張志新(化名)和其女友劉穎,沒(méi)想到半個(gè)月后炎滞,有當(dāng)?shù)厝嗽跇?shù)林里發(fā)現(xiàn)了一具尸體敢艰,經(jīng)...
    沈念sama閱讀 45,724評(píng)論 1 315
  • 正文 獨(dú)居荒郊野嶺守林人離奇死亡,尸身上長(zhǎng)有42處帶血的膿包…… 初始之章·張勛 以下內(nèi)容為張勛視角 年9月15日...
    茶點(diǎn)故事閱讀 37,900評(píng)論 3 336
  • 正文 我和宋清朗相戀三年册赛,在試婚紗的時(shí)候發(fā)現(xiàn)自己被綠了钠导。 大學(xué)時(shí)的朋友給我發(fā)了我未婚夫和他白月光在一起吃飯的照片。...
    茶點(diǎn)故事閱讀 40,040評(píng)論 1 350
  • 序言:一個(gè)原本活蹦亂跳的男人離奇死亡森瘪,死狀恐怖牡属,靈堂內(nèi)的尸體忽然破棺而出,到底是詐尸還是另有隱情扼睬,我是刑警寧澤逮栅,帶...
    沈念sama閱讀 35,742評(píng)論 5 346
  • 正文 年R本政府宣布,位于F島的核電站窗宇,受9級(jí)特大地震影響措伐,放射性物質(zhì)發(fā)生泄漏。R本人自食惡果不足惜军俊,卻給世界環(huán)境...
    茶點(diǎn)故事閱讀 41,364評(píng)論 3 330
  • 文/蒙蒙 一侥加、第九天 我趴在偏房一處隱蔽的房頂上張望。 院中可真熱鬧蝇完,春花似錦官硝、人聲如沸矗蕊。這莊子的主人今日做“春日...
    開(kāi)封第一講書(shū)人閱讀 31,944評(píng)論 0 22
  • 文/蒼蘭香墨 我抬頭看了看天上的太陽(yáng)傻咖。三九已至朋魔,卻和暖如春,著一層夾襖步出監(jiān)牢的瞬間卿操,已是汗流浹背警检。 一陣腳步聲響...
    開(kāi)封第一講書(shū)人閱讀 33,060評(píng)論 1 270
  • 我被黑心中介騙來(lái)泰國(guó)打工, 沒(méi)想到剛下飛機(jī)就差點(diǎn)兒被人妖公主榨干…… 1. 我叫王不留害淤,地道東北人扇雕。 一個(gè)月前我還...
    沈念sama閱讀 48,247評(píng)論 3 371
  • 正文 我出身青樓,卻偏偏與公主長(zhǎng)得像窥摄,于是被迫代替她去往敵國(guó)和親镶奉。 傳聞我的和親對(duì)象是個(gè)殘疾皇子,可洞房花燭夜當(dāng)晚...
    茶點(diǎn)故事閱讀 44,979評(píng)論 2 355

推薦閱讀更多精彩內(nèi)容