1 配置jiava環(huán)境
- 查看安裝的java版本
rpm -qa|grep java
- 查看java安裝目錄
which java
- 配置hadoop-env.sh中的JAVA_HOME，復(fù)制上一條命令的結(jié)果，粘貼到JAVA_HOME中，把連帶bin之后的字符刪除
2 配置hadoop四大模塊：common，hdfs，yarn碧注，mapreduce
- 配置common，core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:8020</value>
</property>
#configure temp directory
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/module/hadoop-3.1.2/data/tmp</value>
</property>
</configuration>
- 配置hdfs糖赔，hdfs-site.xml
配置副本數(shù)
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
3 啟動(dòng)文件系統(tǒng)
- 格式化文件系統(tǒng)
bin/hdfs namenode -format
- 啟動(dòng)namenode萍丐，datanode，secondarynamenode
sbin/start-dfs.sh
jps #查看啟動(dòng)的進(jìn)程
9809 SecondaryNameNode
3752 DataNode
4171 Jps
3647 NameNode
netstat -ntlp 查看進(jìn)程占用的端口
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:9864 0.0.0.0:* LISTEN 3752/java
tcp 0 0 0.0.0.0:9866 0.0.0.0:* LISTEN 3752/java
tcp 0 0 0.0.0.0:9867 0.0.0.0:* LISTEN 3752/java
tcp 0 0 0.0.0.0:9868 0.0.0.0:* LISTEN 3922/java
tcp 0 0 0.0.0.0:9870 0.0.0.0:* LISTEN 3647/java
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN -
tcp 0 0 192.168.1.8:8020 0.0.0.0:* LISTEN 3647/java
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:33530 0.0.0.0:* LISTEN 3752/java
- 查看namenode狀態(tài)
http://ip:9870/
namenode view - 創(chuàng)建用戶
$ bin/hdfs dfs -mkdir /user
$ bin/hdfs dfs -mkdir /user/<username>
- 在分布式系統(tǒng)上創(chuàng)建輸入文件夾
$ bin/hdfs dfs -put etc/hadoop input
- 運(yùn)行實(shí)例
$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.9.2.jar grep input output 'dfs[a-z.]+'
-獲取放典， 查看日志
$ bin/hdfs dfs -get output output
$ cat output/*
-
在瀏覽器上查看
Browse the file system
下載日志
4 啟動(dòng)yarn
- 配置mapreduce逝变，mapred-site.xml
- 配置yarn，yarn-site.xml
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.env-whitelist</name>
<value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
</property>
- 配置mapreduce刻撒，mapred-site.xml
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.application.classpath</name>
<value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
</property>
- 啟動(dòng)
sbin/start-yarn.sh
jps #查看進(jìn)程
3922 SecondaryNameNode
6276 Jps
3752 DataNode
6056 ResourceManager
3647 NameNode
6191 NodeManager
-
查看集群管理
[http://192.168.1.8:8088/cluster]
管理 執(zhí)行mapreduce實(shí)例
錯(cuò)誤
Error:/bin/bash: /bin/java: No such file or directory
/bin/java不存在骨田，則創(chuàng)建
sudo ln -s /opt/module/jdk1.8.0_162/bin/java /bin/java
- 關(guān)閉
sbin/stop-dfs.sh
sbin/stop-yarn.sh
5 配置
- 指定namenode主機(jī)： core-site.xml
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoop-yarn.cloudyhadoop.com:8020</value>
</property>
- 指定datanode主機(jī)：worker
- 指定secondarynamenode主機(jī)：hdfs-site.xml
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>0.0.0.0:9868</value>
<description>
The secondary namenode http server address and port.
</description>
</property>
- 指定resourcemanager主機(jī)名：yarn-site.xml
<property>
<description>The hostname of the RM.</description>
<name>yarn.resourcemanager.hostname</name>
<value>0.0.0.0</value>
</property>
- 指定nodemanager主機(jī)名：yarn-site.xml
<property>
<description>The hostname of the NM.</description>
<name>yarn.nodemanager.hostname</name>
<value>0.0.0.0</value>
</property>
- 指定historyserver主機(jī)名：mapred-site.xml
<name>mapreduce.jobhistory.admin.address</name>
<value>0.0.0.0:10033</value>
<description>The address of the History server admin interface.</description>
</property>
