想學習運行一個mapreduce程序試試菊霜，記錄如下

本地運行word count

新建maven項目，添加hadoop-client，版本比如3.1.2
官方的wordcount直接拿來用就可以
需要把winutils.exe和hadoop.dll放到環(huán)境變量HADOOP_HOME中，這兩個文件夾需要在bin子文件夾中张弛，下載鏈接

添加兩個運行參數，一個輸入文件名酪劫，一個輸出文件名吞鸭，直接就可以運行了

單服務器配置（偽分布式

服務器使用centos安裝，主機名c1覆糟，靜態(tài)ip地址
使用rpm安裝了oracle的jdk8刻剥，解壓hadoop-3.1.2到/hadoop/文件夾下

ssh-keygen
ssh-copy-id localhost

core-site.xml如下

<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://c1:8020</value>
</property>
</configuration>

hdfs-site.xml

<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/hadoop/tmp</value>
</property>
</configuration>

yarn-site.xml

<configuration>

<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>c1</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>

這時候運行還是會出錯，找不到MRAppMaster類滩字，類似如下：

Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster

Please check whether your etc/hadoop/mapred-site.xml contains the below configuration:
<property>
  <name>yarn.app.mapreduce.am.env</name>
  <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
</property>
<property>
  <name>mapreduce.map.env</name>
  <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
</property>
<property>
  <name>mapreduce.reduce.e nv</name>
  <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
</property>

修改如下
mapred-site.xml

<configuration>
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=/hadoop/hadoop-3.1.2</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=/hadoop/hadoop-3.1.2</value>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=/hadoop/hadoop-3.1.2</value>
</property>
</configuration>

修改workers或slaves文件造虏，不同版本的hadoop文件名不一樣
把localhost改成主機名，比如c1

要在/etc/hosts加入c1的靜態(tài)地址麦箍，否則漓藕，就可能連不上nodemanager，比如

192.168.1.111 c1

啟動dfs/yarn服務

格式化只需要第一次運行一下挟裂。

bin/hadoop namenode -format
bin/start-dfs.sh
bin/start-yarn.sh

stop之后想再start可能需要等待文件列表同步享钞，正常要等30秒，可以在網頁startup信息看诀蓉。

開發(fā)機可以看到c1的50070（hadoop2）或9870（hadoop3）端口就說明hdfs起來了
可以看到8088端口說明yarn起來了

修改程序嫩与，加入YARN配置

wordcount程序需要服務端的xml配置寝姿，可以放到conf文件夾交排，把這個文件夾在IDE里點右鍵設置為Resources划滋，不然程序寫相對路徑也訪問不到
修改wordcount程序，加入xml配置

        Configuration conf = new Configuration();
        conf.addResource("core-site.xml");
        conf.addResource("yarn-site.xml");
        conf.addResource("hdfs-site.xml");
        conf.addResource("mapred-site.xml");

        conf.set("mapreduce.app-submission.cross-platform","true");
        conf.set("mapreduce.framework.name","yarn");
        conf.set("mapreduce.job.jar","target\\mr1-1.0-SNAPSHOT.jar");

在maven工具中調用package埃篓，生成的jar文件名和路徑处坪，寫到上面程序最后一行

在hdfs中放一個input.txt文件作為輸入，如果有權限問題可以參考命令

hdfs dfs -chown user:group input.txt
hdfs dfs -chmod 777 input.txt

提交到y(tǒng)arn運行程序

結果如下

2019-03-26 16:49:33,942 INFO  [main] client.RMProxy (RMProxy.java:newProxyInstance(133)) - Connecting to ResourceManager at c1/192.168.1.111:8032
2019-03-26 16:49:34,363 WARN  [main] mapreduce.JobResourceUploader (JobResourceUploader.java:uploadResourcesInternal(147)) - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2019-03-26 16:49:34,372 INFO  [main] mapreduce.JobResourceUploader (JobResourceUploader.java:disableErasureCodingForPath(883)) - Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/cdarling/.staging/job_1553590163847_0001
2019-03-26 16:49:34,500 INFO  [main] input.FileInputFormat (FileInputFormat.java:listStatus(292)) - Total input files to process : 1
2019-03-26 16:49:35,349 INFO  [main] mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(202)) - number of splits:1
2019-03-26 16:49:35,824 INFO  [main] mapreduce.JobSubmitter (JobSubmitter.java:printTokens(298)) - Submitting tokens for job: job_1553590163847_0001
2019-03-26 16:49:35,826 INFO  [main] mapreduce.JobSubmitter (JobSubmitter.java:printTokens(299)) - Executing with tokens: []
2019-03-26 16:49:35,932 INFO  [main] conf.Configuration (Configuration.java:getConfResourceAsInputStream(2752)) - resource-types.xml not found
2019-03-26 16:49:35,932 INFO  [main] resource.ResourceUtils (ResourceUtils.java:addResourcesFileToConf(418)) - Unable to find 'resource-types.xml'.
2019-03-26 16:49:36,278 INFO  [main] impl.YarnClientImpl (YarnClientImpl.java:submitApplication(324)) - Submitted application application_1553590163847_0001
2019-03-26 16:49:36,302 INFO  [main] mapreduce.Job (Job.java:submit(1574)) - The url to track the job: http://c1:8088/proxy/application_1553590163847_0001/
2019-03-26 16:49:36,302 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1619)) - Running job: job_1553590163847_0001
2019-03-26 16:49:41,367 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1640)) - Job job_1553590163847_0001 running in uber mode : false
2019-03-26 16:49:41,367 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1647)) -  map 0% reduce 0%
2019-03-26 16:49:45,410 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1647)) -  map 100% reduce 0%
2019-03-26 16:49:49,437 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1647)) -  map 100% reduce 100%
2019-03-26 16:49:49,447 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1658)) - Job job_1553590163847_0001 completed successfully
2019-03-26 16:49:49,510 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1665)) - Counters: 53
    File System Counters
        FILE: Number of bytes read=99
        FILE: Number of bytes written=433951
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=165
        HDFS: Number of bytes written=61
        HDFS: Number of read operations=8
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=2
    Job Counters 
        Launched map tasks=1
        Launched reduce tasks=1
        Data-local map tasks=1
        Total time spent by all maps in occupied slots (ms)=1608
        Total time spent by all reduces in occupied slots (ms)=1685
        Total time spent by all map tasks (ms)=1608
        Total time spent by all reduce tasks (ms)=1685
        Total vcore-milliseconds taken by all map tasks=1608
        Total vcore-milliseconds taken by all reduce tasks=1685
        Total megabyte-milliseconds taken by all map tasks=1646592
        Total megabyte-milliseconds taken by all reduce tasks=1725440
    Map-Reduce Framework
        Map input records=3
        Map output records=13
        Map output bytes=123
        Map output materialized bytes=99
        Input split bytes=94
        Combine input records=13
        Combine output records=8
        Reduce input groups=8
        Reduce shuffle bytes=99
        Reduce input records=8
        Reduce output records=8
        Spilled Records=16
        Shuffled Maps =1
        Failed Shuffles=0
        Merged Map outputs=1
        GC time elapsed (ms)=75
        CPU time spent (ms)=930
        Physical memory (bytes) snapshot=509763584
        Virtual memory (bytes) snapshot=5578809344
        Total committed heap usage (bytes)=410517504
        Peak Map Physical memory (bytes)=294195200
        Peak Map Virtual memory (bytes)=2786607104
        Peak Reduce Physical memory (bytes)=215568384
        Peak Reduce Virtual memory (bytes)=2792202240
    Shuffle Errors
        BAD_ID=0
        CONNECTION=0
        IO_ERROR=0
        WRONG_LENGTH=0
        WRONG_MAP=0
        WRONG_REDUCE=0
    File Input Format Counters 
        Bytes Read=71
    File Output Format Counters 
        Bytes Written=61

Process finished with exit code 0