原文地址：http://blog.csdn.net/oopsoom/article/details/41318599
一、問題定義
最近在用sbt打assembly包時出現(xiàn)問題事扭，在package的時候，發(fā)生jar包沖突/文件沖突問題乐横，兩個相同的class來自不同的jar包在classpath內引起沖突求橄。
具體是：我有一個self4j的jar， 還有一個hadoop-common-hdfs的jar包葡公，其中hadoop-common-hdfs.jar內包含了self4j這個jar包谈撒，導致沖突。
此類異常一般是由于打包不規(guī)范和打包疏忽引起的匾南。
（個人認為正確的打包策略是：只打包自己核心功能，不將依賴打包在一起蛔外，但是有時為了方便或者有時不得不打在一起蛆楞，要注意可能會出現(xiàn)上述問題）
異常log如下：
[trace] Stack trace suppressed: run last *:assembly for the full output.
[error] (*:assembly) deduplicate: different file contents found in the following:
[error] C:\Users\shengli.victor\.ivy2\cache\org.slf4j\slf4j-api\jars\slf4j-api-1.7.7.jar:org/slf4j/IMarkerFactory.class
[error] C:\Users\shengli.victor\.ivy2\cache\com.xxx.xx.hdfsfile\hdfscommon\jars\hdfscommon-1.1.jar:org/slf4j/IMarkerFactory.class
[error] Total time: 4 s, completed 2014-11-20 19:07:33
異常很明顯溯乒，來自2個不同的jar包self4j, hdfscommon-1.1.jar里，在org/slf4j/IMarkerFactory.class這個類沖突了豹爹。
如下圖：
二裆悄、解決方案
解決jar包沖突有兩種方案：
1、刪除其中的某個jar,或者說臂聋，在打包的時候光稼，不將2個相同的class打在同一個jar包內的classpath內，即exclude jar孩等。
2艾君、合并沖突
1. Excluding JARs and files
% "provided"
將相同的jar中排除一個，因為重復肄方，可以使用"provided"關鍵字冰垄。
例如spark是一個容器類，編寫spark應用程序我們需要spark core jar. 但是真正打包提交到集群上執(zhí)行权她，則不需要將它打入jar包內虹茶。
這是我們使用 % "provided" 關鍵字來exclude它。
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "0.8.0-incubating" % "provided",
"org.apache.hadoop" % "hadoop-client" % "2.0.0-cdh4.4.0" % "provided"
)
Maven defines "provided" as:
This is much like compile, but indicates you expect the JDK or a container to provide the dependency at runtime. For example, when building a web application for the Java Enterprise Edition, you would set the dependency on the Servlet API and related Java EE APIs to scopeprovided because the web container provides those classes. This scope is only available on the compilation and test classpath, and is not transitive.
2隅要、Merge Strategy
如果在相對路徑下蝴罪，有多個相同的文件或者jar，這時我們可以使用Merge策略步清。
在build.sbt中對assemblyMergeStrategy 進行定義靠娱。
如下：
mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
{
case PathList("org", "slf4j", xs @ _*) => MergeStrategy.first
case PathList(ps @ _*) if ps.last endsWith "axiom.xml" => MergeStrategy.filterDistinctLines
case PathList(ps @ _*) if ps.last endsWith "Log$Logger.class" => MergeStrategy.first
case PathList(ps @ _*) if ps.last endsWith "ILoggerFactory.class" => MergeStrategy.first
case x => old(x)
}
}
解決方法：將org, slf4j 這個下的所有類和文件踪宠，都做合并， 采用的策略是：在classpath里，2選1很泊，選的是classpath順序里第一個self4j。
這里支持多種格式栽烂，例如ps.lat endsWith "axiom.xml" 胚吁，是以axiom.xml為結尾的文件，都采用filterDistinctLines策略书聚，即合并兩個文件唧领，舍去重復的部分。
通過以上修改雌续，終于解決了slf4j沖突的問題斩个，即deduplicate: different file contents found in the following問題。
再次sbt assembly:
[warn] Merging 'META-INF\INDEX.LIST' with strategy 'discard'
[warn] Merging 'META-INF\MANIFEST.MF' with strategy 'discard'
[warn] Merging 'META-INF\maven\log4j\log4j\pom.properties' with strategy 'first'
[warn] Merging 'META-INF\maven\log4j\log4j\pom.xml' with strategy 'first'
[warn] Merging 'META-INF\maven\org.slf4j\slf4j-api\pom.properties' with strategy 'first'
[warn] Merging 'META-INF\maven\org.slf4j\slf4j-api\pom.xml' with strategy 'first'
[warn] Merging 'META-INF\maven\org.slf4j\slf4j-log4j12\pom.properties' with strategy 'first'
[warn] Merging 'META-INF\maven\org.slf4j\slf4j-log4j12\pom.xml' with strategy 'first'
[warn] Merging 'com\esotericsoftware\minlog\Log$Logger.class' with strategy 'first'
[warn] Merging 'com\esotericsoftware\minlog\Log.class' with strategy 'first'
[warn] Merging 'org\apache\log4j\helpers\LogLog.class' with strategy 'first'
[warn] Merging 'org\slf4j\ILoggerFactory.class' with strategy 'first'
[warn] Merging 'org\slf4j\IMarkerFactory.class' with strategy 'first'
[warn] Merging 'org\slf4j\Logger.class' with strategy 'first'
[warn] Merging 'org\slf4j\LoggerFactory.class' with strategy 'first'
[warn] Merging 'org\slf4j\MDC.class' with strategy 'first'
[warn] Merging 'org\slf4j\Marker.class' with strategy 'first'
[warn] Merging 'org\slf4j\MarkerFactory.class' with strategy 'first'
[warn] Merging 'org\slf4j\helpers\BasicMDCAdapter.class' with strategy 'first'
[warn] Merging 'org\slf4j\helpers\BasicMarker.class' with strategy 'first'
[warn] Merging 'org\slf4j\helpers\BasicMarkerFactory.class' with strategy 'first'
[warn] Merging 'org\slf4j\helpers\FormattingTuple.class' with strategy 'first'
[warn] Merging 'org\slf4j\helpers\MarkerIgnoringBase.class' with strategy 'first'
[warn] Merging 'org\slf4j\helpers\MessageFormatter.class' with strategy 'first'
[warn] Merging 'org\slf4j\helpers\NOPLogger.class' with strategy 'first'
[warn] Merging 'org\slf4j\helpers\NOPLoggerFactory.class' with strategy 'first'
[warn] Merging 'org\slf4j\helpers\NOPMDCAdapter.class' with strategy 'first'
[warn] Merging 'org\slf4j\helpers\NamedLoggerBase.class' with strategy 'first'
[warn] Merging 'org\slf4j\helpers\SubstituteLoggerFactory.class' with strategy 'first'
[warn] Merging 'org\slf4j\helpers\Util.class' with strategy 'first'
[warn] Merging 'org\slf4j\impl\Log4jLoggerAdapter.class' with strategy 'first'
[warn] Merging 'org\slf4j\impl\Log4jLoggerFactory.class' with strategy 'first'
[warn] Merging 'org\slf4j\impl\Log4jMDCAdapter.class' with strategy 'first'
[warn] Merging 'org\slf4j\impl\StaticLoggerBinder.class' with strategy 'first'
[warn] Merging 'org\slf4j\impl\StaticMDCBinder.class' with strategy 'first'
[warn] Merging 'org\slf4j\impl\StaticMarkerBinder.class' with strategy 'first'
[warn] Merging 'org\slf4j\spi\LocationAwareLogger.class' with strategy 'first'
[warn] Merging 'org\slf4j\spi\LoggerFactoryBinder.class' with strategy 'first'
[warn] Merging 'org\slf4j\spi\MDCAdapter.class' with strategy 'first'
[warn] Merging 'org\slf4j\spi\MarkerFactoryBinder.class' with strategy 'first'
[warn] Merging 'rootdoc.txt' with strategy 'concat'
[warn] Strategy 'concat' was applied to a file
[info] Strategy 'deduplicate' was applied to 373 files (Run the task at debug level to see details)
[warn] Strategy 'discard' was applied to 2 files
[warn] Strategy 'first' was applied to 38 files
[java] view plain copy 在CODE上查看代碼片派生到我的代碼片
[info] Done packaging.
[success] Total time: 84 s, completed 2014-11-20 19:04:52
合并策略有很多種：
可以參考官方sbt assembly文檔：https://github.com/sbt/sbt-assembly
http://stackoverflow.com/questions/19606243/resolving-dependencies-in-creating-jar-through-sbt-assembly
MergeStrategy.deduplicate is the default described above
MergeStrategy.first picks the first of the matching files in classpath order
MergeStrategy.last picks the last one
MergeStrategy.singleOrError bails out with an error message on conflict
MergeStrategy.concat simply concatenates all matching files and includes the result
MergeStrategy.filterDistinctLines also concatenates, but leaves out duplicates along the way
MergeStrategy.rename renames the files originating from jar files
MergeStrategy.discard simply discards matching files
更多的寫法驯杜，example：
assemblyMergeStrategy in assembly := {
case PathList("javax", "servlet", xs @ _*) => MergeStrategy.first
case PathList(ps @ _*) if ps.last endsWith ".html" => MergeStrategy.first
case "application.conf" => MergeStrategy.concat
case "unwanted.txt" => MergeStrategy.discard
case x =>
val oldStrategy = (assemblyMergeStrategy in assembly).value
oldStrategy(x)
}
Final Qucik Hack：
如果以上寫法都不奏效受啥，還有最好一種，強制默認全部合并，不到萬不得已滚局，不要用居暖。。
mergeStrategy in assembly <<= (mergeStrategy in assembly) { mergeStrategy => {
case entry => {
val strategy = mergeStrategy(entry)
if (strategy == MergeStrategy.deduplicate) MergeStrategy.first
else strategy
}
}}
三藤肢、總結
碰到類似的問題不要慌張太闺，仔細看log描述的是什么意思。
異常報出內容冗余的沖突嘁圈，在看路徑省骂，發(fā)現(xiàn)在classpath內有完全相同的2個類，這是導致問題的根本原因最住。
找出原因钞澳，解決方發(fā)，消除沖突兩種方法温学，一直是去除法略贮，另一種是合并法。
相對于maven和gradle仗岖，sbt的沖突解決方法還是比較接近底層逃延。如果沒記錯的話，maven和gradle都能自動解決沖突轧拄。
本文僅針對該問題提出解決方案和思路揽祥，具體的各個配置還需要繼續(xù)研究。
