基于hadoop HDFS的存儲(chǔ)系統(tǒng)（web 網(wǎng)盤）
1. HDFS的優(yōu)勢
1.1 源碼注釋說很透徹：
Hadoop DFS is a multi-machine system that appears as a single
disk. It's useful because of its fault tolerance and potentially
very large capacity.
解釋：HDFS是一個(gè)多機(jī)系統(tǒng)，對外作為一個(gè)整體的磁盤存在畔师。由于它的容錯(cuò)性和大容量娶靡，具有很大的可用性
1.2 對外作為一個(gè)整體 和 容錯(cuò)性 的原理
1）整體性
NameNode保存元數(shù)據(jù)，作為整個(gè)系統(tǒng)打大管家
2）容錯(cuò)性
數(shù)據(jù)數(shù)據(jù)多機(jī)備份看锉、數(shù)據(jù)校驗(yàn)姿锭、心跳檢測、數(shù)據(jù)塊報(bào)告伯铣、讀寫容錯(cuò)
2. 代碼實(shí)現(xiàn)
這里代碼是基于maven呻此、springboot書寫。當(dāng)然方式多種多樣腔寡，都可以考慮焚鲜。
2.1 依賴重要的包
hadoop-common包，這是一個(gè)對hadoop其余模塊的公告支持庫放前。
包含conf忿磅、FS、IO和IPC等部分
hadoop-common源碼地址如下:
https://github.com/apache/hadoop/tree/trunk/hadoop-common-project
2.1 上傳功能
1）接收用戶的上傳請求
package com.hdlw.controls;
@Controller
public class UploadMap {
...
@RequestMapping(value = "/upload")
@ResponseBody
public ReturnMessage upload(@RequestParam("file") MultipartFile file, HttpServletRequest httpServletRequest) {
...
int result = UploadDownload.upload(file.getInputStream(), path);
...
}
}
2）處理上傳存儲(chǔ)請求
public class UploadDownload {
...
public static int upload(InputStream inputStream, String path) {
int result = -1;
//借助FileSystem 來聯(lián)系/操作HDFS系統(tǒng)
//這里針對每個(gè)請求實(shí)例化一個(gè)局部變量來處理
FileSystem fileSystem = checkAndInit();
if (fileSystem == null) {
return result;
}
OutputStream outputStream = null;
try {
outputStream = fileSystem.create(new Path(path), true);
result = IOUtils.copy(inputStream, outputStream);
} catch (Exception e) {
e.printStackTrace();
} finally {
close(inputStream,outputStream);
}
return result;
}
...
}
2.2 下載功能
1）接收用戶的下載請求
@GetMapping("/download")
public String downloadFile(HttpServletRequest request, HttpServletResponse response, @RequestParam String filenameName) {
.....
try {
response.setContentType("application/force-download");
response.addHeader("Content-Disposition", "attachment;fileName=" + filenameName);
int result = UploadDownload.download(response.getOutputStream(), path);
System.out.println(result);
if (result != -1) {
returnMessage.setMsg(constants.operate_ok);
}
} catch (IOException e) {
e.printStackTrace();
}
return null;
}
1）下載請求處理
public static int download(OutputStream outputStream, String path){
int result = -1;
FileSystem fileSystem = checkAndInit();
if (fileSystem == null) {
return result;
}
InputStream inputStream = null;
Path filePath = new Path(path);
try {
if (!fileSystem.exists(filePath)) {
return result;
}
inputStream = fileSystem.open(new Path(path));
result = IOUtils.copy(inputStream, outputStream);
} catch (Exception e) {
e.printStackTrace();
} finally {
close(inputStream,outputStream);
}
return result;
}
結(jié)語
要實(shí)現(xiàn)復(fù)雜的功能還是有不少工作和細(xì)節(jié)需要去做：
權(quán)限管理凭语、用戶管理葱她、上傳、查詢似扔、下載览效、、刪除虫几、目錄（文件夾）操作、hdfs系統(tǒng)管理
