在實(shí)施方案前，假設(shè)讀者已經(jīng)基本熟悉以下技術(shù) (不細(xì)說(shuō))
- Java毒费，maven
- hdfs，kerberos
方案實(shí)施
-
最后目錄大概如下
目錄結(jié)構(gòu).png 新建maven工程，pom.xml配置, 添加以下
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.6.5</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.6.5</version>
</dependency>
- 拷貝kdc下的krb5.conf文件
[libdefaults]
default_realm = W.COM
dns_lookup_realm = false
dns_lookup_kdc = false
ticket_lifetime = 24h
renew_lifetime = 7d
forwardable = true
[realms]
W.COM = {
kdc = node1:88
admin_server = node1:749
default_domain = W.COM
kdc = bi-slave1
}
#這里的kdc做了HA坑傅，需要把從kdc也加進(jìn)realm里面
#默認(rèn)的端口號(hào)也可以去掉
- 生成keytab文件
kadmin.local: xst -norandkey -k deng_yb.keytab deng_yb@W.COM
-
下載core-site.xml文件
下載hdfs客戶(hù)端.png
拿到core-site.xml文件
core-site-xml.png
- HDFSUtils
package deng.yb.hdfsUtils;
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.security.UserGroupInformation;
public class HDFSUtils {
/**
* 加載hdfs客戶(hù)端配置
* @return
*/
public static Configuration initConfiguration() {
Configuration configuration = new Configuration();
configuration.addResource(new Path(getPath("core-site.xml")));
return configuration;
}
/**
* 初始化Kerberos環(huán)境
*/
public static void initKerberosENV(Configuration conf) {
System.setProperty("java.security.krb5.conf",
getPath("krb5.conf"));
System.setProperty("javax.security.auth.useSubjectCredsOnly", "false");
// System.setProperty("sun.security.krb5.debug", "true");
try {
UserGroupInformation.setConfiguration(conf);
UserGroupInformation.loginUserFromKeytab("deng_yb@W.COM",
getPath("deng_yb.keytab"));
System.out.println(UserGroupInformation.getCurrentUser());
} catch (IOException e) {
e.printStackTrace();
}
}
public static String getPath(String file) {
if (null == file) {
return null;
}
return Thread.currentThread().getContextClassLoader().getResource(file).getPath();
}
}
- 測(cè)試
package deng.yb.hdfsUtils;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
/**
* 鏈接HA,kerberos 集群
*
*/
public class App
{
public static void main( String[] args )
{
//初始化HDFS Configuration 配置
Configuration configuration = HDFSUtils.initConfiguration();
//初始化Kerberos環(huán)境
HDFSUtils.initKerberosENV(configuration);
try {
//其中nameservice1是HA別名
Path dstPath = new Path("hdfs://nameservice1/");
FileSystem fileSystem = FileSystem.get(configuration);
FileStatus[] listStatus = fileSystem.listStatus(dstPath);
for (FileStatus fileStatus : listStatus) {
Path path = fileStatus.getPath();
System.out.println(path);
}
} catch (Exception e){
e.printStackTrace();
}
}
}
-
執(zhí)行main方法，結(jié)果如下
結(jié)果.png
