通過hadoop源碼編譯阱飘，安裝snappy并生成hadoop native本地庫谬盐。
編譯步驟：
1程剥、安裝maven
tar -zxvf
apache-maven-3.3.3-bin.tar.gz//解壓
mv apache-maven-3.3.3
maven//修改名稱
vi /etc/profile//添加環(huán)境變量
MAVEN_HOME=/usr/java/maven
PATH=$MAVEN_HOME/bin:$PATH
exportMAVEN_HOME
保存退出
source /etc/profile//使文件生效
mvn–v//查看maven版本
2变过、安裝ProtocolBuffer
tar -zxvfprotobuf-2.5.0.tar.gz
mv protobuf-2.5.0 protobuf
cd protobuf
./configure --prefix=/usr/local/protobuf
make
make check
make install
vi /etc/profile//添加環(huán)境變量
PROTOBUF_HOME=/usr/local/protobuf
PATH=$PROTOBUF_HOME/bin:$PATH
exportPROTOBUF_HOME
保存退出
source /etc/profile//使文件生效
protoc–version查看版本號
3、安裝cmake
yum -y install cmake
4症脂、zlib安裝
yum -y install zlib
5谚赎、安裝openssl-devel
yum -y install openssl-devel
6桶唐、安裝snappy
tar -zxvf snappy-1.1.3.tar.gz
mv snappy-1.1.3 snappy
cd snappy
./configure
make
make install
7憎妙、編譯hadoop源碼hadoop-2.2.0-src
tar–zxvf hadoop-2.2.0-src.tar.gz
cd hadoop-2.2.0-src
mvn package -Pdist,native -DskipTests -Dtar -Dbundle.snappy-Dsnappy.lib=/usr/local/lib
8、編譯完成后把編譯好的hadoop解壓提取native本地庫替換現(xiàn)有機(jī)器上裝的hadoop的native
9楞件、Hadoop配置snappy壓縮
配置hadoop-env.sh
exportHADOOP_HOME=/usr/java/hadoop安裝目錄
exportLD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_HOME/lib/native
配置core-site.xml
io.compression.codecs
org.apache.hadoop.io.compress.SnappyCodec
配置mapred-site.xml
如果MR中間結(jié)果需要使用snappy壓縮兴蒸，修改mapred-site.xml
mapreduce.map.output.compress
true
mapreduce.map.output.compress.codec
org.apache.hadoop.io.compress.SnappyCodec
