隸屬于文章系列：大數(shù)據(jù)安全實(shí)戰(zhàn) http://www.reibang.com/p/76627fd8399c
- hdfs和s3之間的帶寬是復(fù)制速度的上限考阱。
- -m <num_maps>
- bandwidth 帶寬
- 增加mapper的個(gè)數(shù)矢渊，和每個(gè)mapper的帶寬可能會(huì)提高傳輸速度，但是 mapper個(gè)數(shù) * bandwidth選項(xiàng)，不會(huì)超過hdfs和s3帶寬的上限浑玛。
- hadoop集群離s3越遠(yuǎn)甲锡，帶寬越小，復(fù)制越慢中燥。在同一區(qū)域的s3 和主機(jī)往往能提高速度寇甸。
- 但是即便是hadoop同樣也部署云基礎(chǔ)設(shè)施中，有可能因觸發(fā)s3的速度限制而導(dǎo)致數(shù)據(jù)復(fù)制慢疗涉。對(duì)一個(gè)目錄樹的負(fù)載過重拿霉，S3可能會(huì)可能會(huì)拖延處理或者拒絕。
- 很多mapper的大數(shù)據(jù)量的復(fù)制操作咱扣，可能會(huì)減慢向S3的上傳绽淘。
- 當(dāng)增減mapper的個(gè)數(shù)實(shí)際上在減慢復(fù)制速度的時(shí)候，可能的原因就是觸發(fā)了限制闹伪。
-
-p[rbugpcaxt]選項(xiàng)對(duì)s3來說沒有意義沪铭，但是用了之后會(huì)讓每次數(shù)據(jù)傳輸都是全量復(fù)制。
hive@cdh-slave01:/tmp$ hadoop distcp -Dfs.s3a.access.key=fdsfdsfdsfsdIMNH6ZBsdf -Dfs.s3a.secret.key=fasdfds -update/user/hive/warehouse/ads.db s3a://bucket/user/hive/warehouse/ads.db
19/04/10 06:59:01 INFO tools.OptionsParser: parseChunkSize: blocksperchunk false
19/04/10 06:59:02 WARN impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties
19/04/10 06:59:02 INFO impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
19/04/10 06:59:02 INFO impl.MetricsSystemImpl: s3a-file-system metrics system started
19/04/10 06:59:03 INFO Configuration.deprecation: fs.s3a.server-side-encryption-key is deprecated. Instead, use fs.s3a.server-side-encryption.key
19/04/10 06:59:03 INFO tools.DistCp: Input Options: DistCpOptions{atomicCommit=false, syncFolder=true, deleteMissing=false, ignoreFailures=false, overwrite=false, append=false, useDiff=false, useRdiff=false, fromSnapshot=null, toSnapshot=null, skipCRC=false, blocking=true, numListstatusThreads=0, maxMaps=20, mapBandwidth=100, sslConfigurationFile='null', copyStrategy='uniformsize', preserveStatus=[], preserveRawXattrs=false, atomicWorkPath=null,logPath=null, sourceFileListing=null, sourcePaths=[/user/hive/warehouse/ads.db], targetPath=s3a://bucket/user/hive/warehouse/ads.db, targetPathExists=true, filtersFile='null', blocksPerChunk=0, copyBufferSize=8192}
19/04/10 06:59:04 INFO tools.SimpleCopyListing: Paths (files+dirs) cnt = 2843; dirCnt = 19
19/04/10 06:59:04 INFO tools.SimpleCopyListing: Build file listing completed.
19/04/10 06:59:04 INFO Configuration.deprecation: io.sort.mb is deprecated. Instead, use mapreduce.task.io.sort.mb
19/04/10 06:59:04 INFO Configuration.deprecation: io.sort.factor is deprecated. Instead, use mapreduce.task.io.sort.factor
19/04/10 06:59:04 INFO tools.DistCp: Number of paths in the copy list: 2843
19/04/10 06:59:04 INFO tools.DistCp: Number of paths in the copy list: 2843
19/04/10 06:59:04 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm721
19/04/10 06:59:04 INFO mapreduce.JobSubmitter: number of splits:21
19/04/10 06:59:04 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1554779117367_2179
19/04/10 06:59:05 INFO impl.YarnClientImpl: Submitted application application_1554779117367_2179
19/04/10 06:59:05 INFO mapreduce.Job: The url to track the job: http://cdh-master2:8088/proxy/application_1554779117367_2179/
19/04/10 06:59:05 INFO tools.DistCp: DistCp job-id: job_1554779117367_2179
19/04/10 06:59:05 INFO mapreduce.Job: Running job: job_1554779117367_2179
19/04/10 06:59:10 INFO mapreduce.Job: Job job_1554779117367_2179 running in uber mode : false
19/04/10 06:59:10 INFO mapreduce.Job: map 0% reduce 0%
19/04/10 06:59:18 INFO mapreduce.Job: map 10% reduce 0%
19/04/10 06:59:19 INFO mapreduce.Job: map 19% reduce 0%
19/04/10 06:59:20 INFO mapreduce.Job: map 52% reduce 0%
19/04/10 06:59:21 INFO mapreduce.Job: map 67% reduce 0%
19/04/10 06:59:22 INFO mapreduce.Job: map 86% reduce 0%
19/04/10 06:59:23 INFO mapreduce.Job: map 95% reduce 0%
19/04/10 06:59:24 INFO mapreduce.Job: map 100% reduce 0%
19/04/10 06:59:25 INFO mapreduce.Job: Job job_1554779117367_2179 completed successfully
19/04/10 06:59:25 INFO mapreduce.Job: Counters: 40
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=3304256
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=985334
HDFS: Number of bytes written=413302
HDFS: Number of read operations=5797
HDFS: Number of large read operations=0
HDFS: Number of write operations=42
S3A: Number of bytes read=0
S3A: Number of bytes written=32647
S3A: Number of read operations=3014
S3A: Number of large read operations=0
S3A: Number of write operations=37
Job Counters
Launched map tasks=21
Other local map tasks=21
Total time spent by all maps in occupied slots (ms)=183711
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=183711
Total vcore-milliseconds taken by all map tasks=183711
Total megabyte-milliseconds taken by all map tasks=564360192
Map-Reduce Framework
Map input records=2843
Map output records=2822
Input split bytes=2394
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=9536
CPU time spent (ms)=136920
Physical memory (bytes) snapshot=9398771712
Virtual memory (bytes) snapshot=56006500352
Total committed heap usage (bytes)=15602810880
File Input Format Counters
Bytes Read=950293
File Output Format Counters
Bytes Written=413302
DistCp Counters
Bytes Copied=32647
Bytes Expected=32647
Bytes Skipped=1800151
Files Copied=21
Files Skipped=2822
19/04/10 06:59:25 INFO impl.MetricsSystemImpl: Stopping s3a-file-system metrics system...
19/04/10 06:59:25 INFO impl.MetricsSystemImpl: s3a-file-system metrics system stopped.
19/04/10 06:59:25 INFO impl.MetricsSystemImpl: s3a-file-system metrics system shutdown complete.
#在命令行查看目錄個(gè)數(shù) 文件個(gè)數(shù) 目錄總的字節(jié)數(shù)
hive@cdh-slave01:/tmp$ hdfs dfs -count /user/hive/warehouse/ads.db
20 2824 1832798 /user/hive/warehouse/ads.db
當(dāng)distcp計(jì)數(shù)器中有skipped這一行的時(shí)候偏瓤，才是復(fù)制的增量杀怠，不然就是復(fù)制全量。
驗(yàn)證方法：
- 運(yùn)行輸出中：
Number of paths in the copy list: 2843包含了文件和目錄的總數(shù)=Files Copied=+Files Skipped=files num+dir num
目錄總的字節(jié)數(shù)=Bytes Copied=+Bytes Skipped
參考
-
hadoop fs -count < hdfs path >統(tǒng)計(jì)hdfs對(duì)應(yīng)路徑下的目錄個(gè)數(shù)厅克，文件個(gè)數(shù)赔退，文件總計(jì)大小
顯示為目錄個(gè)數(shù)，文件個(gè)數(shù)证舟，文件總計(jì)大小硕旗，輸入路徑
例如：
hadoop fs -count /data/dltb3yi/ 1 24000 253953854502 /data/dltb3yi/ 獲得24000個(gè)文件
