搭建方式可參考:
http://www.reibang.com/p/faf038923093
參上以上簡書內(nèi)容搭建即可醉蚁,但是存在如下四個(gè)問題需要補(bǔ)充:
- 需要bin目錄替換萧吠,至 https://github.com/steveloughran/winutils下載對(duì)于版本的bin答捕,并添加到hadoop中即可。
- nodemanger無法啟動(dòng)
- 啟動(dòng)nodemanager存在路徑無法打開的問題
- wordcount無法運(yùn)行
解決方案:
針對(duì)2,3,4三個(gè)問題,首先yarn-site.xml的內(nèi)容需要修改為
<?xml version="1.0"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<!-- NodeManager總的可用虛擬CPU個(gè)數(shù) -->
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>1</value>
</property>
<property>
<!-- 每個(gè)節(jié)點(diǎn)可用的最大內(nèi)存 -->
<name>yarn.nodemanager.resource.memory-mb</name>
<value>4096</value>
</property>
<property>
<name>yarn.nodemanager.log-dirs</name>
<value>/C:/hadoop-3.2.2/logs/yarn</value>
</property>
</configuration>
- hadoop啟動(dòng)時(shí)候需要使用管理員方式打開cmd啟動(dòng)start-all
- 每個(gè)節(jié)點(diǎn)的可用最大內(nèi)存需要注意分配的大一些病曾,此處分配為4096,之前分配為2048斤斧,若分配的小了勺美,會(huì)存在map 0 reduce 0 情況