Spark 異步Action

What if we want to execute 2 actions concurrently on different RDD’s, Spark actions are always synchronous. Like if we perform two actions one after other they always execute in sequentially like one after other.
Let see example

val rdd = sc.parallelize(List(32, 34, 2, 3, 4, 54, 3), 4)
rdd.collect().map{ x => println("Items in the lists:" + x)}
val rddCount = sc.parallelize(List(434, 3, 2, 43, 45, 3, 2), 4)
println("Number of items in the list" + rddCount.count())

In the above exmaple 2 actions are perform one after other collect and count, both are execute synchronous. So count will always execute after collect will finish. The out of the above code is as follows


Now question is if we want to run spark jobs concurrently in async fashion.
So for above question answer is simple apache spark also provide a asyn action for concurrent execution of jobs, Few Asynchronous actions spark provide as follows
collectAsync() -> Returns a future for retrieving all elements of this RDD.countAsync() -> Returns a future for counting the number of elements in the RDD.foreachAsync(scala.Function1<T,scala.runtime.BoxedUnit> f) -> Applies a function f to all elements of this RDD.foreachPartitionAsync(scala.Function1<scala.collection.Iterator,scala.runtime.BoxedUnit> f) ->Applies a function f to each partition of this RDD.takeAsync(int num) -> Returns a future for retrieving the first num elements of the RDD.
Now let us see what happen when we use async actions.

val rdd = sc.parallelize(List(32, 34, 2, 3, 4, 54, 3), 4)
rdd.collectAsync().map{ x => x.map{x=> println("Items in the list:"+x)} }
val rddCount = sc.parallelize(List(434, 3, 2, 43, 45, 3, 2), 4)
rddCount.countAsync().map { x =>println("Number of items in the list: "+x) }

So output of the above code is as follows


You can see in above output the result of the second job is come first because first job return future and execute second one but still have you noticed that jobs are execute one after other that’s means a job use all resources of cluster so another job will delayed.
So for take full advantage of Asynchronous jobs we need to configure job scheduler.
Job Scheduling
By default spark scheduler run spark jobs in FIFO (First In First Out) fashion. In FIFO scheduler the priority is given to the first job and then second and so on. If the jobs is not using whole cluster then second job is also run parallel but if first job is too big then second job will wait soo long even it take too less to execute. So for solution spark provide fair scheduler, fair scheduler jobs will execute in “round robin” fashion.
To configure job scheduler we need to set configuration for it as follows
val conf = new SparkConf().setAppName("spark_auth").setMaster("local[*]").set("spark.scheduler.mode", "FAIR")
After configure FAIR scheduling you can see both the jobs are running concurrently and share resources of the spark cluster.
So after this the out of the above code is as follows
Screenshot from 2015-10-21 13:35:53
Screenshot from 2015-10-21 13:35:53

You can see in above result both jobs are running concurrently. The result of both the actions are not wait for each other.
For above code you can checkout: https://github.com/knoldus/spark-scala-async

最后編輯于
?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請聯(lián)系作者
  • 序言:七十年代末晕鹊,一起剝皮案震驚了整個濱河市痴脾,隨后出現(xiàn)的幾起案子洪乍,更是在濱河造成了極大的恐慌借嗽,老刑警劉巖躁锁,帶你破解...
    沈念sama閱讀 222,681評論 6 517
  • 序言:濱河連續(xù)發(fā)生了三起死亡事件峡谊,死亡現(xiàn)場離奇詭異崖面,居然都是意外死亡厚骗,警方通過查閱死者的電腦和手機椅寺,發(fā)現(xiàn)死者居然都...
    沈念sama閱讀 95,205評論 3 399
  • 文/潘曉璐 我一進店門浑槽,熙熙樓的掌柜王于貴愁眉苦臉地迎上來,“玉大人配并,你說我怎么就攤上這事括荡。” “怎么了溉旋?”我有些...
    開封第一講書人閱讀 169,421評論 0 362
  • 文/不壞的土叔 我叫張陵畸冲,是天一觀的道長。 經(jīng)常有香客問我观腊,道長邑闲,這世上最難降的妖魔是什么? 我笑而不...
    開封第一講書人閱讀 60,114評論 1 300
  • 正文 為了忘掉前任梧油,我火速辦了婚禮苫耸,結(jié)果婚禮上,老公的妹妹穿的比我還像新娘儡陨。我一直安慰自己褪子,他們只是感情好,可當我...
    茶點故事閱讀 69,116評論 6 398
  • 文/花漫 我一把揭開白布骗村。 她就那樣靜靜地躺著嫌褪,像睡著了一般。 火紅的嫁衣襯著肌膚如雪胚股。 梳的紋絲不亂的頭發(fā)上笼痛,一...
    開封第一講書人閱讀 52,713評論 1 312
  • 那天,我揣著相機與錄音琅拌,去河邊找鬼缨伊。 笑死,一個胖子當著我的面吹牛进宝,可吹牛的內(nèi)容都是我干的刻坊。 我是一名探鬼主播,決...
    沈念sama閱讀 41,170評論 3 422
  • 文/蒼蘭香墨 我猛地睜開眼党晋,長吁一口氣:“原來是場噩夢啊……” “哼紧唱!你這毒婦竟也來了活尊?” 一聲冷哼從身側(cè)響起,我...
    開封第一講書人閱讀 40,116評論 0 277
  • 序言:老撾萬榮一對情侶失蹤漏益,失蹤者是張志新(化名)和其女友劉穎蛹锰,沒想到半個月后,有當?shù)厝嗽跇淞掷锇l(fā)現(xiàn)了一具尸體绰疤,經(jīng)...
    沈念sama閱讀 46,651評論 1 320
  • 正文 獨居荒郊野嶺守林人離奇死亡铜犬,尸身上長有42處帶血的膿包…… 初始之章·張勛 以下內(nèi)容為張勛視角 年9月15日...
    茶點故事閱讀 38,714評論 3 342
  • 正文 我和宋清朗相戀三年,在試婚紗的時候發(fā)現(xiàn)自己被綠了轻庆。 大學時的朋友給我發(fā)了我未婚夫和他白月光在一起吃飯的照片癣猾。...
    茶點故事閱讀 40,865評論 1 353
  • 序言:一個原本活蹦亂跳的男人離奇死亡,死狀恐怖余爆,靈堂內(nèi)的尸體忽然破棺而出纷宇,到底是詐尸還是另有隱情,我是刑警寧澤蛾方,帶...
    沈念sama閱讀 36,527評論 5 351
  • 正文 年R本政府宣布像捶,位于F島的核電站,受9級特大地震影響桩砰,放射性物質(zhì)發(fā)生泄漏拓春。R本人自食惡果不足惜,卻給世界環(huán)境...
    茶點故事閱讀 42,211評論 3 336
  • 文/蒙蒙 一亚隅、第九天 我趴在偏房一處隱蔽的房頂上張望硼莽。 院中可真熱鬧,春花似錦煮纵、人聲如沸懂鸵。這莊子的主人今日做“春日...
    開封第一講書人閱讀 32,699評論 0 25
  • 文/蒼蘭香墨 我抬頭看了看天上的太陽匆光。三九已至,卻和暖如春隘擎,著一層夾襖步出監(jiān)牢的瞬間殴穴,已是汗流浹背凉夯。 一陣腳步聲響...
    開封第一講書人閱讀 33,814評論 1 274
  • 我被黑心中介騙來泰國打工货葬, 沒想到剛下飛機就差點兒被人妖公主榨干…… 1. 我叫王不留,地道東北人劲够。 一個月前我還...
    沈念sama閱讀 49,299評論 3 379
  • 正文 我出身青樓震桶,卻偏偏與公主長得像,于是被迫代替她去往敵國和親征绎。 傳聞我的和親對象是個殘疾皇子蹲姐,可洞房花燭夜當晚...
    茶點故事閱讀 45,870評論 2 361

推薦閱讀更多精彩內(nèi)容

  • **2014真題Directions:Read the following text. Choose the be...
    又是夜半驚坐起閱讀 9,586評論 0 23
  • 今天一個人去超市磨取,晚風陣陣,畢竟已經(jīng)春天了柴墩。去的時候太陽還在戀戀不舍忙厌,回來的時候天空一片灰暗,無星無月江咳。我一直覺得...
    冷璞閱讀 354評論 1 4
  • 人生就是一場漫長的比賽而對手就是你自己是的就是一個人的比賽你可以平心靜氣的也可以心焦氣燥的 就算場上有那么多人那也...
    哈哈同學閱讀 233評論 0 1
  • 1歼指、第一:社群應(yīng)該盡量避免在群內(nèi)交流過多無聊的話題爹土,會直接影響用戶的情緒。 2踩身、第二:多借用工具來管理社群胀茵,例如:...
    明天會是晴天閱讀 557評論 0 1
  • 前些日子琼娘,微信群里聊到大學生這個話題。我特別認慫的說了句赁濒,我不是大學生轨奄。然后簡單幾句,描述了當年的狀況拒炎。 考砸了挪拟,...
    陳若心閱讀 1,967評論 0 1