背景

使用idea調(diào)試spark + hive sql 程序時(shí)候，經(jīng)常會(huì)碰到這個(gè)問(wèn)題包晰，比較難以解決顷歌。顧此處我總結(jié)了一下常見(jiàn)的解決方案。

異常

Caused by: java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode0(Ljava/lang/String;I)V
    at org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode0(Native Method)
    at org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode(NativeIO.java:524)
    at org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:465)
    at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:518)
    at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:496)
    at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:316)
    at org.apache.hadoop.hive.ql.session.SessionState.createPath(SessionState.java:639)
    at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:567)
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:508)
    at org.apache.spark.sql.hive.client.HiveClientImpl.newState(HiveClientImpl.scala:185)
    at org.apache.spark.sql.hive.client.HiveClientImpl.<init>(HiveClientImpl.scala:118)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:271)
    at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:384)
    at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:286)
    at org.apache.spark.sql.hive.HiveExternalCatalog.client$lzycompute(HiveExternalCatalog.scala:66)
    at org.apache.spark.sql.hive.HiveExternalCatalog.client(HiveExternalCatalog.scala:65)
    at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply$mcZ$sp(HiveExternalCatalog.scala:215)
    at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply(HiveExternalCatalog.scala:215)
    at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply(HiveExternalCatalog.scala:215)
    at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97)

代碼

// 是否需要做kerberos認(rèn)證锣险，需要自己實(shí)現(xiàn)
System.load("D:\\hadoop2.6.0\\bin\\hadoop.dll");
SparkSession.Builder builder = SparkSession.builder().appName("test");
builder.master("local");
builder.enableHiveSupport();
SparkSession spark = builder.getOrCreate();
Configuration conf = spark.sparkContext().hadoopConfiguration();
conf.addResource("core-site.xml");
conf.addResource("hdfs-site.xml");
conf.addResource("yarn-site.xml");
conf.addResource("hive-site.xml");
Dataset<Row> df = spark.sql("select * from test.a ");
df.show();

注意：hive-site.xml等xml文件（生產(chǎn)或者測(cè)試環(huán)境中的配置）要放在項(xiàng)目根目錄下，原因是 new Hiveconf()時(shí)候會(huì)默認(rèn)讀取根目錄下的配置文件

解決方案

• 確保本地安裝hadoop览闰，我使用的版本是2.6.0芯肤，hadoop路徑是D:\hadoop-2.6.0。配置環(huán)境變量HADOOP_HOME=D:\hadoop-2.6.0压鉴。
• 確保已經(jīng)下載hadoop window版winutils集成包（請(qǐng)注意有版本區(qū)別）崖咨，github下載地址https://github.com/steveloughran/winutils∮涂裕或者大家直接在GitHub上搜索winutils 也會(huì)有很多的击蹲。
• 確保winutils解壓后的bin包覆蓋hadoop-2.6.0中的bin包，并將bin目錄下的hadoop.dll文件拷貝到C:\Windows\System32下
• 如果UnsatisfiedLinkErro還存在婉宰，你需要在idea的debug/run配置 vm options 中添加 -Djava.library.path="D:\hadoop-2.6.0"歌豺，這樣保證程序運(yùn)行的時(shí)候加載的是2.6.0中的jar包而不是其他地方的卒茬。
• OK了盆顾，我的程序能夠正常運(yùn)行了。

關(guān)于大數(shù)據(jù)方面技術(shù)問(wèn)題可以咨詢盹愚，替你解決你的苦惱。微信 hainanzhongjian