本文講述的是如何在IDEA里編寫一個計算PI的Spark程序笆檀。
主要內(nèi)容：
- 1.安裝Scala
- 2.編寫ScalaPI
相關(guān)文章：
1.Spark之PI本地
2.Spark之WordCount集群
3.SparkStreaming之讀取Kafka數(shù)據(jù)
4.SparkStreaming之使用redis保存Kafka的Offset
5.SparkStreaming之優(yōu)雅停止
6.SparkStreaming之寫數(shù)據(jù)到Kafka
7.Spark計算《西虹市首富》短評詞云
1.安裝Scala
1.安裝Scala
下載地址：傳送門 （2.11.12）
下載完成解壓到任意目錄饰恕，設(shè)置環(huán)境變量旨剥，直接在Path后加入解壓目錄
打開控制臺輸入scala
說明安裝成功
2.IDEA安裝Scala插件
打開IDEA痢艺，F(xiàn)ile->settings->Plugins->Browse Repositoies->搜索Scala->Install然后重啟IDEA即可
2.編寫ScalaPI
2.1.導(dǎo)入依賴
<dependencies>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.3.0</version>
<!--<scope>provided</scope>-->
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql_2.11 -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>2.3.0</version>
<!--<scope>provided</scope>-->
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.scala-tools</groupId>
<artifactId>maven-scala-plugin</artifactId>
<version>2.15.2</version>
<executions>
<execution>
<id>scala-compile-first</id>
<goals>
<goal>compile</goal>
</goals>
<configuration>
<includes>
<include>**/*.scala</include>
</includes>
</configuration>
</execution>
<execution>
<id>scala-test-compile</id>
<goals>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
2.2.編寫代碼
object ScalaPi {
def main(args: Array[String]) {
//創(chuàng)建一個Config
val conf =
new SparkConf()
.setAppName("ScalaPI")
.setMaster("local")
//核心創(chuàng)建SparkContext對象
val sc = new SparkContext(conf)
//計算PI
val count = sc.parallelize(1 to 10000).filter { _ =>
val x = math.random
val y = math.random
x * x + y * y < 1
}.count()
println(s"Pi is roughly ${4.0 * count / 10000}")
//停止SparkContext對象
sc.stop()
}
}
運行結(jié)果如下：
