MapReduce官網(wǎng)介紹地址http://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html
Overview
Hadoop MapReduce is a software framework for easily writing applications which process vast amounts of data (multi-terabyte data-sets) in-parallel on large clusters (thousands of nodes) of commodity hardware in a reliable, fault-tolerant manner.
MapReduce以可靠勺美、容錯(cuò)的方式在大型集群的普通硬件上并行處理海量的數(shù)據(jù)
A MapReduce job usually splits the input data-set into independent chunks（塊） which are processed by the map tasks in a completely parallel manner. The framework sorts the outputs of the maps, which are then input to the reduce tasks. Typically both the input and the output of the job are stored in a file-system. The framework takes care of scheduling tasks, monitoring them and re-executes the failed tasks.
MapReduce作業(yè)通常會(huì)把輸入的數(shù)據(jù)集切分成獨(dú)立的塊，交由map進(jìn)行完全的并行的處理。框架會(huì)把map輸出的數(shù)據(jù)進(jìn)行排序然后輸入到reduce官册。作業(yè)的輸入和輸出都存儲(chǔ)在文件系統(tǒng)中硫痰】贪框架負(fù)責(zé)調(diào)度任務(wù)馋记、監(jiān)視任務(wù)和重啟失敗的任務(wù)。
Typically the compute nodes and the storage nodes are the same, that is, the MapReduce framework and the Hadoop Distributed File System (see HDFS Architecture Guide) are running on the same set of nodes. This configuration allows the framework to effectively schedule tasks on the nodes where data is already present, resulting in very high aggregate(高聚合) bandwidth across the cluster.
通常計(jì)算節(jié)點(diǎn)和存儲(chǔ)節(jié)點(diǎn)是相同的肆资，即MapReduce和HDFS運(yùn)行在相同的節(jié)點(diǎn)上矗愧。這種配置允許框架有效地在已存在數(shù)據(jù)的節(jié)點(diǎn)上調(diào)度任務(wù)，從而在集群中產(chǎn)生非常高的聚合帶寬郑原。
The MapReduce framework consists of a single master ResourceManager, one slave NodeManager per cluster-node, and MRAppMaster per application (see YARN Architecture Guide).
MapReduce框架由一個(gè)主節(jié)點(diǎn)RM唉韭，每個(gè)集群節(jié)點(diǎn)一個(gè)NM夜涕，和每個(gè)應(yīng)用程序一個(gè)MR的AM組成
Minimally, applications specify the input/output locations and supply map and reduce functions via implementations of appropriate interfaces and/or abstract-classes. These, and other job parameters, comprise the job configuration.
應(yīng)用程序通過合適的接口實(shí)現(xiàn)類或抽象類來指定輸入輸出位置和實(shí)現(xiàn)map和reduce功能
The Hadoop job client then submits the job (jar/executable etc.) and configuration to the ResourceManager which then assumes the responsibility of distributing the software/configuration to the slaves, scheduling tasks and monitoring them, providing status and diagnostic information to the job-client.
hadoop作業(yè)客戶端提交作業(yè)和配置給RM,RM負(fù)責(zé)分發(fā)程序和配置給從節(jié)點(diǎn)，調(diào)度任務(wù)并監(jiān)視他們属愤，提供狀態(tài)和診斷信息給客戶端
Inputs and Outputs
The MapReduce framework operates exclusively（只） on <key, value> pairs, that is, the framework views the input to the job as a set of <key, value> pairs and produces a set of<key, value> pairs as the output of the job, conceivably of different types.
MapReduce框架只處理鍵值對(duì)數(shù)據(jù)钠乏，作業(yè)的輸入可以看成是一連串的鍵值對(duì)，產(chǎn)生一連串不同類型的鍵值對(duì)作為作業(yè)的輸出春塌。
The key and value classes have to be serializable by the framework and hence need to implement the Writable interface. Additionally, the key classes have to implement the WritableComparable interface to facilitate（促進(jìn)） sorting by the framework.
key和value值必須經(jīng)過框架的序列化，因此需要實(shí)現(xiàn)Writable接口簇捍。key值還需要實(shí)現(xiàn)WritableComparable接口便于框架的排序只壳。
- Writable接口源碼
package org.apache.hadoop.io;
import java.io.DataOutput;
import java.io.DataInput;
import java.io.IOException;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;
/**
* A serializable object which implements a simple, efficient, serialization
* protocol, based on {@link DataInput} and {@link DataOutput}.
*
* <p>Any <code>key</code> or <code>value</code> type in the Hadoop Map-Reduce
* framework implements this interface.</p>
*
* <p>Implementations typically implement a static <code>read(DataInput)</code>
* method which constructs a new instance, calls {@link #readFields(DataInput)}
* and returns the instance.</p>
*
* <p>Example:</p>
* <p><blockquote><pre>
* public class MyWritable implements Writable {
* // Some data
* private int counter;
* private long timestamp;
*
* public void write(DataOutput out) throws IOException {
* out.writeInt(counter);
* out.writeLong(timestamp);
* }
*
* public void readFields(DataInput in) throws IOException {
* counter = in.readInt();
* timestamp = in.readLong();
* }
*
* public static MyWritable read(DataInput in) throws IOException {
* MyWritable w = new MyWritable();
* w.readFields(in);
* return w;
* }
* }
* </pre></blockquote></p>
*/
@InterfaceAudience.Public
@InterfaceStability.Stable
public interface Writable {
/**
* Serialize the fields of this object to <code>out</code>.
*
* @param out <code>DataOuput</code> to serialize this object into.
* @throws IOException
*/
void write(DataOutput out) throws IOException;
/**
* Deserialize the fields of this object from <code>in</code>.
*
* <p>For efficiency, implementations should attempt to re-use storage in the
* existing object where possible.</p>
*
* @param in <code>DataInput</code> to deseriablize this object from.
* @throws IOException
*/
void readFields(DataInput in) throws IOException;
}
- WritableComparable接口源碼
package org.apache.hadoop.io;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;
/**
* A {@link Writable} which is also {@link Comparable}.
*
* <p><code>WritableComparable</code>s can be compared to each other, typically
* via <code>Comparator</code>s. Any type which is to be used as a
* <code>key</code> in the Hadoop Map-Reduce framework should implement this
* interface.</p>
*
* <p>Note that <code>hashCode()</code> is frequently used in Hadoop to partition
* keys. It's important that your implementation of hashCode() returns the same
* result across different instances of the JVM. Note also that the default
* <code>hashCode()</code> implementation in <code>Object</code> does <b>not</b>
* satisfy this property.</p>
*
* <p>Example:</p>
* <p><blockquote><pre>
* public class MyWritableComparable implements WritableComparable<MyWritableComparable> {
* // Some data
* private int counter;
* private long timestamp;
*
* public void write(DataOutput out) throws IOException {
* out.writeInt(counter);
* out.writeLong(timestamp);
* }
*
* public void readFields(DataInput in) throws IOException {
* counter = in.readInt();
* timestamp = in.readLong();
* }
*
* public int compareTo(MyWritableComparable o) {
* int thisValue = this.value;
* int thatValue = o.value;
* return (thisValue < thatValue ? -1 : (thisValue==thatValue ? 0 : 1));
* }
*
* public int hashCode() {
* final int prime = 31;
* int result = 1;
* result = prime * result + counter;
* result = prime * result + (int) (timestamp ^ (timestamp >>> 32));
* return result
* }
* }
* </pre></blockquote></p>
*/
@InterfaceAudience.Public
@InterfaceStability.Stable
public interface WritableComparable<T> extends Writable, Comparable<T> {
}
