win10下idea使用sbt方式編譯scala源碼
雖然集群使用Linux疚宇，Ubuntu的桌面版也有非常好的體驗感受告嘲，但是windows下寫碼編譯打包成jar文件然后再提交到集群運行也是不錯的，故本文記錄win10下如何搭建環(huán)境
準(zhǔn)備工作
idea官網(wǎng)下載 https://www.jetbrains.com/idea/
sbt官網(wǎng)下載 https://www.scala-sbt.org/download.html
一些配置
打開idea华嘹，右上角File->Settings
找到Plugins(插件)吧趣，選擇安裝Jetbrains的插件
在搜索框搜索Scala什么版本無所謂，可能就是為了代碼高亮并且有智能提示和聯(lián)想。因為在之后sbt編譯的配置中不一定使用這個版本强挫。
插件下載完成之后會提示restart idea岔霸，那就restart咯~
新建一個Project，選擇sbt
安裝環(huán)境需要java
打開Settings,找到Build Tools俯渤，進(jìn)行對sbt的一些自定義
可以使用在官網(wǎng)下載的sbt-1.1.6.zip呆细，解壓，解壓后的文件路徑請?zhí)砑拥江h(huán)境變量
然后如下圖八匠，Launcher選擇Custom絮爷，選擇剛剛解壓的文件中sbt/bin/sbt-launch.jar
如果能夠用Google的小伙伴這一步無所謂，不能用就換源梨树，可以換成阿里的~
找到解壓sbt的目錄坑夯，進(jìn)入conf文件夾，新建文本文檔改名為repo.properties
[repositories]
local
aliyun: http://maven.aliyun.com/nexus/content/groups/public/
typesafe: http://repo.typesafe.com/typesafe/ivy-releases/, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext], bootOnly
sonatype-oss-releases
maven-central
sonatype-oss-snapshots
完成之后抡四，切換至Terminal
sbt有許多命令
sbt clean
sbt compile
sbt package
sbt assembly
···
我習(xí)慣使用package命令柜蜈，將scala編譯后生成的內(nèi)容打成jar包，提交到集群運行
到此為止指巡，可以鍵入sbt sbtVersion查看sbt的版本淑履，這個過程將陷入漫長的等待。
第一次會比較慢厌处，可以去喝杯咖啡或者開一波黑鳖谈，之后就會比較快了。
鍵入 sbt sbtVersion 的運行結(jié)果如果是這樣就表示成了阔涉！
D:\IDEAProjects\SparkSample>sbt sbtVersion
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
[info] Loading settings from idea.sbt ...
[info] Loading global plugins from C:\Users\zhongfuze\.sbt\1.0\plugins
[info] Loading settings from assembly.sbt ...
[info] Loading project definition from D:\IDEAProjects\SparkSample\project
[info] Loading settings from build.sbt ...
[info] Set current project to SparkSample (in build file:/D:/IDEAProjects/SparkSample/)
[info] 1.1.6
在src/main/scala里新建HelloScala.scala
object HelloScala {
def main(args: Array[String]): Unit = {
println("Hello Scala!")
}
}
鍵入 sbt package 的運行結(jié)果如果是這樣就表示成了缆娃！
生成的jar包 在根目錄/target/scala-2.11/xxxxx.jar
D:\IDEAProjects\SparkSample>sbt package
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
[info] Loading settings from idea.sbt ...
[info] Loading global plugins from C:\Users\zhongfuze\.sbt\1.0\plugins
[info] Loading settings from assembly.sbt ...
[info] Loading project definition from D:\IDEAProjects\SparkSample\project
[info] Loading settings from build.sbt ...
[info] Set current project to SparkSample (in build file:/D:/IDEAProjects/SparkSample/)
[info] Compiling 1 Scala source to D:\IDEAProjects\SparkSample\target\scala-2.11\classes ...
[info] Done compiling.
[info] Packaging D:\IDEAProjects\SparkSample\target\scala-2.11\sparksample_2.11-1.0.jar ...
[info] Done packaging.
[success] Total time: 4 s, completed 2018-7-24 16:12:19
分割線
到這里為止只是能夠能打包而已，但是coding的時候是需要各種各樣的配置的呢
而且為了便于維護(hù)各種各樣的依賴jar瑰排，properties贯要，config，并且有時需要決定某些jar文件需要被打包椭住，有的jar文件集群里已經(jīng)存在了崇渗，不需要打包，要滿足這種需求京郑，就使用sbt assembly命令
在項目根目錄/project下宅广，與build.properties的同級目錄
新建assembly.sbt，內(nèi)容如下
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.7")
Note:sbt-assembly的版本取決于sbt的版本些举，不同的版本區(qū)間版本號不一樣的８！户魏！
接著找到根目錄下build.sbt
這里可以自定義許多內(nèi)容驶臊，添加依賴等等
更具體的用法請參考github-sbt-assembly
https://github.com/sbt/sbt-assembly
name := "SparkSample"
version := "1.0"
organization := "com.zhong.PRM"
scalaVersion := "2.11.8"
assemblyJarName in assembly := "PRM.jar"
test in assembly := {}
assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false)
assemblyMergeStrategy in assembly := {
case PathList("javax", "servlet", xs@_*) => MergeStrategy.first
case PathList(ps@_*) if ps.last endsWith ".class" => MergeStrategy.first
case PathList(ps@_*) if ps.last endsWith ".xml" => MergeStrategy.first
case PathList(ps@_*) if ps.last endsWith ".properties" => MergeStrategy.first
case "application.conf" => MergeStrategy.concat
case "unwanted.txt" => MergeStrategy.discard
case x =>
val oldStrategy = (assemblyMergeStrategy in assembly).value
oldStrategy(x)
}
libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.1.0" % "provided"
排除jar包
provided 表示打包的時候可以不包含這個jar包
sbt-assembly是根據(jù)項目配置的libraryDependencies依賴進(jìn)行打包的挪挤，不需要打包的依賴可以設(shè)置”provided”進(jìn)行排除
[build.sbt]
libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.1.0" % "provided"
排除scala庫的jar包
在項目根目錄下創(chuàng)建assembly.sbt文件并添加以下配置（注：sbt-assembly相關(guān)的配置，可以配置在項目根目錄/build.sbt中关翎，也可以在項目根目錄下的assembly.sbt文件中）：
[assembly.sbt]
assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false)
明確排除某一指定jar包
[assembly.sbt]
assemblyExcludedJars in assembly := {
val cp = (fullClasspath in assembly).value
cp filter {_.data.getName == “compile-0.1.0.jar”}
}
多個文件共享相同的相對路徑
如果多個文件共享相同的相對路徑（例如扛门，多個依賴項JAR中名為application.conf的資源），則默認(rèn)策略是驗證所有候選項具有相同的內(nèi)容纵寝，否則出錯论寨。可以使用以下內(nèi)置策略之一或編寫自定義策略在每個路徑的基礎(chǔ)上配置此行為：
MergeStrategy.deduplicate是上面描述的默認(rèn)值
MergeStrategy.first以類路徑順序選擇第一個匹配文件
MergeStrategy.last選擇最后一個
MergeStrategy.singleOrError在沖突時出現(xiàn)錯誤消息
MergeStrategy.concat簡單地連接所有匹配的文件并包含結(jié)果
MergeStrategy.filterDistinctLines也會連接店雅，但在此過程中會遺漏重復(fù)項
MergeStrategy.rename重命名源自jar文件的文件
MergeStrategy.discard只是丟棄匹配的文件
路徑名稱到合并策略的映射是通過設(shè)置assemblyMergeStrategy完成的政基，可以按如下方式擴(kuò)充
assemblyMergeStrategy in assembly := {
case PathList("javax", "servlet", xs @ _*) => MergeStrategy.first
case PathList(ps @ _*) if ps.last endsWith ".html" => MergeStrategy.first
case "application.conf" => MergeStrategy.concat
case "unwanted.txt" => MergeStrategy.discard
case x =>
val oldStrategy = (assemblyMergeStrategy in assembly).value
oldStrategy(x)
}
放一些配置文件做備忘
[plugins.sbt]
logLevel := Level.Warn
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.3")
[build.sbt]
name := "lanke"
version := "1.0"
scalaVersion := "2.11.8"
assemblyJarName in assembly := "lanke.jar"
test in assembly := {}
assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false)
assemblyMergeStrategy in assembly := {
case PathList("javax", "servlet", xs@_*) => MergeStrategy.first
case PathList(ps@_*) if ps.last endsWith ".class" => MergeStrategy.first
case PathList(ps@_*) if ps.last endsWith ".xml" => MergeStrategy.first
case PathList(ps@_*) if ps.last endsWith ".properties" => MergeStrategy.first
case "application.conf" => MergeStrategy.concat
case "unwanted.txt" => MergeStrategy.discard
case x =>
val oldStrategy = (assemblyMergeStrategy in assembly).value
oldStrategy(x)
}
resolvers ++= Seq(
"kompics" at "http://kompics.sics.se/maven/repository/"
)
javacOptions ++= Seq("-encoding", "UTF-8", "-source", "1.7", "-target", "1.7")
resolvers ++= Seq(
"libs-releases" at "http://artifactory.jd.com/libs-releases",
"libs-snapshots" at "http://artifactory.jd.com/libs-snapshots",
"plugins-releases" at "http://artifactory.jd.com/plugins-releases",
"plugins-snapshots" at "http://artifactory.jd.com//plugins-snapshots"
)
libraryDependencies ++= Seq(
"org.apache.spark" % "spark-core_2.11" % "2.3.0" % "provided",
"org.apache.spark" % "spark-sql_2.11" % "2.3.0" % "provided",
"org.apache.spark" % "spark-streaming_2.11" % "2.3.0" % "provided",
"org.apache.spark" % "spark-hive_2.11" % "2.3.0" % "provided",
"org.apache.spark" % "spark-repl_2.11" % "2.3.0" % "provided",
"org.apache.spark" % "spark-tags_2.11" % "2.3.0" % "provided"
)
libraryDependencies += "com.yammer.metrics" % "metrics-core" % "2.2.0"
libraryDependencies += "com.typesafe" % "config" % "1.2.1"
libraryDependencies += "net.liftweb" % "lift-json_2.11" % "3.0"
libraryDependencies += "com.huaban" % "jieba-analysis" % "1.0.2"
resolvers += "Sonatype OSS Releases" at "http://oss.sonatype.org/content/repositories/releases/"
libraryDependencies += "com.thesamet" %% "kdtree" % "1.0.4"
libraryDependencies += "com.soundcloud" % "cosine-lsh-join-spark_2.10" % "1.0.1"
libraryDependencies += "org.tensorflow" %% "spark-tensorflow-connector" % "1.6.0"
libraryDependencies += "org.scalaj" %% "scalaj-http" % "2.4.0"
[tools/sbt/conf repo.properties]
[repositories]
local
my-ivy-proxy-releases: http://artifactory.jd.com/ivy-release/, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
my-maven-proxy-releases: http://artifactory.jd.com/libs-releases/
[respositories]
[repositories]
local
aliyun: http://maven.aliyun.com/nexus/content/groups/public/
typesafe: http://repo.typesafe.com/typesafe/ivy-releases/, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext], bootOnly
sonatype-oss-releases
maven-central
sonatype-oss-snapshots
