- 創(chuàng)建RDD
代碼：
def sparkContext(name:String)=
{
val conf = new SparkConf().setAppName(name).setMaster("local")
val sc = new SparkContext(conf)
sc
} - Map
作用：適用于任何集合豁翎，且對(duì)其作用的集合中的每一個(gè)元素循環(huán)遍歷已慢，并調(diào)用其作為參數(shù)的函數(shù)對(duì)每一個(gè)遍歷的元素進(jìn)行具體化處理。
代碼：
def mapTransformation(sc:SparkContext): Unit ={
val nums = sc.parallelize(1 to 10)//根據(jù)集合創(chuàng)建RDD
val mapped = nums.map(item=> 2 * item)
mapped.collect.foreach(print)
}
結(jié)果：2 4 6 8 10 12 14 16 18 20
Filter
作用：遍歷集合中的所有元素，將每個(gè)元素作為參數(shù)放入函數(shù)中進(jìn)行判斷乏屯，將判斷結(jié)果為真的元素篩選出來(lái)梧税。
代碼：
def filterTransformation(sc:SparkContext): Unit ={
val nums = sc.parallelize(1 to 20)//根據(jù)集合創(chuàng)建RDD
val filtered = nums.filter(item => item % 2 == 0)
filtered.collect.foreach(println)
}
結(jié)果：2 4 6 8 10 12 14 16 18 20Flatmap
作用：通過(guò)傳入的作為參數(shù)的函數(shù)來(lái)作用與RDD的每個(gè)字符串進(jìn)行單詞切分镜豹，然后把切分后的結(jié)果合并成一個(gè)大的集合
代碼：
def flatmapTransformation(sc:SparkContext): Unit ={
val bigData = Array("scala","spark","java_Hadoop","java_tachyon")
val bigDataString =sc.parallelize(bigData)
val words= bigDataString.flatMap(line=>line.split(" "))
words.collect.foreach(print)
}
結(jié)果：scala spark java_Hadoop java_tachyongroupByKey
作用:將傳入的tuple數(shù)組生成為RDD，通過(guò)groupByKey方法將RDD通過(guò)key進(jìn)行分組匯總也榄，并生成一個(gè)新的RDD
代碼：
def groupByKeyTransformation(sc:SparkContext): Unit ={
val data = Array(Tuple2(100,"Spark"),Tuple2(100,"Tachyon"),Tuple2(90,"Hadoop"),Tuple2(80,"Kafka"),Tuple2(70,"Scala"))
val dataRDD = sc.parallelize(data)
val group = dataRDD.groupByKey()
group.collect.foreach(pair=>println(pair._1+":"+pair._2))
}
結(jié)果：
100:CompactBuffer(Spark, Tachyon)
90:CompactBuffer(Hadoop)
80:CompactBuffer(Kafka)
70:CompactBuffer(Scala)reduceByKey
作用：對(duì)key相同的元素進(jìn)行value值得相加。
代碼：
def reduceByKeyTransformation(sc:SparkContext): Unit ={
val lines =sc.textFile("C://Users//feng//IdeaProjects//WordCount//src//SparkText.txt",1)
val reduce= lines.map(line=>(line,1)).reduceByKey(+)
reduce.collect.foreach(pair=>println(pair._1+":"+pair._2))
}
文件內(nèi)容：
hadoop hadoop hadoop
spark Flink spark
scala scala object
object spark scala
spark spark
hadoop
hadoop
結(jié)果：
hadoop hadoop hadoop:1
spark Flink spark:1
scala scala object:1
object spark scala:1
spark spark:1
hadoop:2
Join
作用：根據(jù)相同key,把不同的RDD合并為一個(gè)RDD
代碼：
def joinTransformation(sc:SparkContext): Unit ={
//大數(shù)據(jù)中最重要的算子
val studentNames=Array(
Tuple2(1,"Spark"),
Tuple2(2,"Tachyon"),
Tuple2(3,"Hadoop")
)
val studentScore=Array(
Tuple2(1,100),
Tuple2(2,95),
Tuple2(3,65),
Tuple2(2,95),
Tuple2(3,65)
)
val names = sc.parallelize(studentNames)
val scores = sc.parallelize(studentScore)
val studentNameAndScore=names.join(scores)
studentNameAndScore.collect.foreach(println)
}
結(jié)果：
(1,(Spark, 100))
(3,(Hadoop, 65))
(3,(Hadoop, 65))
(2,(Tachyon,95))
(2,(Tachyon,95))cogroup
作用：協(xié)同分組司志，首先將兩個(gè)RDD的內(nèi)容進(jìn)行join,在此基礎(chǔ)上甜紫，以ID為key的情況下將改ID內(nèi)容的所有分?jǐn)?shù)聚合到一起。
代碼：
def cogroupTransformation(sc:SparkContext): Unit ={
val nameList = Array(
Tuple2(1,"Spark"),
Tuple2(2,"Scala"),
Tuple2(3,"Hadoop")
)
val scoreList = Array(
Tuple2(1,100),
Tuple2(2,90),
Tuple2(3,87),
Tuple2(1,80),
Tuple2(2,90),
Tuple2(2,60)
)
val names = sc.parallelize(nameList)
val scores =sc.parallelize(scoreList)
val nameScores= names.cogroup(scores)
nameScores.collect.foreach(println)
}
結(jié)果：
(1,(CompactBuffer(Spark),CompactBuffer(100, 80)))
(3,(CompactBuffer(Hadoop),CompactBuffer(87)))
(2,(CompactBuffer(Scala),CompactBuffer(90, 90, 60)))
