有空朴皆，上一個(gè)完整過程捡絮。
| software | version |
|---|---|
| operating system | Ubuntu 18.04.3 LTS |
| jdk version | 1.8.0_232 |
| hadoop version | 3.1.3 |
一昙楚、install java
$ sudo apt-get update
$ sudo apt-get install openjdk-8-jre openjdk-8-jdk -y
$ dpkg -L openjdk-8-jdk | grep '/bin' (or whereis)
$ vim ~/.profile
?```
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
?```
$ source ~/.profile
將以上寫入shell腳本（記得echo時(shí)$符號(hào)前加反斜杠 ' \ '）可能不識(shí)別source命令，則:
$ ls -l \`which sh\` # 發(fā)現(xiàn)sh不對(duì)：bin/sh -> dash
$ sudo dpkg-reconfigure dash # 選擇 no
$ ls -l \`which sh\` # 此時(shí)為 bin/sh -> bash ，即可以在shell腳本中執(zhí)行source
二、install hadoop
$ sudo apt-get install ssh
$ sudo apt-get install pdsh -y
$ wget http://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-3.1.3/hadoop-3.1.3.tar.gz
$ tar -zxvf hadoop-3.1.3.tar.gz
三绒尊、single node配置
將java路徑添加至etc/hadoop/hadoop-env.sh
將Hadoop路徑添加至系統(tǒng)環(huán)境配置 ~/.profile
$ echo "export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64" >> etc/hadoop/hadoop-env.sh
$ echo "export HADOOP_HOME=/home/vickee/bigData/hadoop-3.1.3" >> ~/.profile
$ echo "export PATH=\$PATH:\$HADOOP_HOME/bin:\$HADOOP_HOME/sbin" >> ~/.profile
$ hadoop
四、fully distributed（完全分布式配置）
三臺(tái)主機(jī)：
| role | hostname |
|---|---|
| master | blockchain004 |
| slave1 | blockchain002 |
| slave2 | blockchain003 |
1. Install java && Hadoop for each node（同上文一仔粥、二、蟹但，不再贅述）
2. 配置主機(jī)名（on each node）
$ vim /etc/hostname
$ vim /etc/hosts # IP and hostname of all nodes inthe expected clusetr
3. 關(guān)閉防火墻
- on ubuntu
$ sudo ufw status
$ sudo ufw disable
- on centos
$ sudo systemctl status firewalld
$ sudo systemctl stop firewalld
4. 時(shí)間同步
$ sudo apt-get install ntp
$ ntpdate -u ntp1.aliyun.com
$ date # test if ok
5. 配置ssh免密
- on each node
$ ssh-keygen -t rsa
- on each slave
$ scp .ssh/id_rsa.pub vickee@master.ip.xxx.xxx:.ssh/id_rsa_002.pub
- on master
$ cd .ssh
$ cp id_rsa.pub authorized_keys
$ cat id_rsa_002.pub >> authorized_keys # for each slave pub
$ scp authorized_keys vickee@slave.ip.xxx.xxx:.ssh # for each slave ip
6. 配置Hadoop
在master節(jié)點(diǎn)更改配置文件后后發(fā)送至slave節(jié)點(diǎn)躯泰。
- hadoop-env.sh
$ echo "export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64" >> etc/hadoop/hadoop-env.sh
- * yarn-env.sh
$ echo "export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64" >> etc/hadoop/yarn-env.sh
- core-site.xml
$ vim etc/hadoop/core-site.sh
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://blockchain-004:9000</value>
</property>
<property>
<name>io.file.buffer.size</name>
<value>131072</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>file:/home/vickee/bigData/temp</value>
</property>
</configuration>
- hdfs-site.xml
$ vim etc/hadoop/hdfs-site.sh
<configuration>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>blockchain-004:9001</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/home/vickee/bigData/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/home/vickee/bigData/dfs/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.web.ugi</name>
<value>supergroup</value>
</property>
</configuration>
- mapred-site.xml
$ vim etc/hadoop/mapred-site.sh
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>blockchain-004:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>blockchain-004:19888</value>
</property>
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=/home/vickee/bigData/hadoop-3.1.3</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=/home/vickee/bigData/hadoop-3.1.3</value>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=/home/vickee/bigData/hadoop-3.1.3</value>
</property>
</configuration>
note: 我初始配置時(shí)沒有添加最后三個(gè)屬性，導(dǎo)致運(yùn)行基準(zhǔn)測試工具TestDFSIO時(shí)出現(xiàn)錯(cuò)誤：
Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster
Please check whether your etc/hadoop/mapred-site.xml contains the below configuration:
......
- yarn-site.xml
$ vim etc/hadoop/yarn-site.sh
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>blockchain-004:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>blockchain-004:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>blockchain-004:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>blockchain-004:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>blockchain-004:8088</value>
</property>
</configuration>
- worker
$ vim etc/hadoop/workers
blockchain-002
blockchain-003
- 發(fā)送配置文件夾到salves上
$ scp -r etc/hadoop/ vickee@slave.ip.xxx.xxx:/home/vickee/bigData/hadoop-3.1.3/etc/
- 格式化主節(jié)點(diǎn)hadoop
$ hadoop namenode -format
提示：successfully formatted
- 啟動(dòng)
$ ./sbin/start-all.sh
可通過jps命令在各節(jié)點(diǎn)中查看Hadoop的相關(guān)進(jìn)程
