Hadoop和HBase安装snappy压缩库(2.4.1) 2015-09-15 21:00

说明

hadoop中经常需要使用snappy这个压缩库。但这个库并未集成成hadoop安装包中,需要额外安装。

安装思路:先在集群中找一台机器,完整安装snappy。其他节点只拷贝相关库过去,并配置hadoop即可。

注意:安装Snappy之前,需要先解决Hadoop 2.4.1的libhadoop.so的位数与运行环境不一致问题(参考:Hadoop源码编译)

安装过程

选取一个节点安装

  • 安装snappy snappy-devel库
yum install -y snappy snappy-devel
cp /usr/lib64/libsnapp* /usr/local/lib
  • 编译snappy-hadoop库
wget https://github.com/electrum/hadoop-snappy/archive/master.zip
unzip master.zip
cd hadoop-snappy-master/
#编译
mvn package
  • 拷贝hadoop snappy库

编译完成后,解压target目录下的hadoop-snappy-0.0.1-SNAPSHOT.tar.gz,并拷贝相关文件到hadoop

cd target
tar -xcvf hadoop-snappy-0.0.1-SNAPSHOT.tar.gz
cp hadoop-snappy-0.0.1-SNAPSHOT/lib/native/Linux-amd64-64/lib* /opt/hadoop/lib/native/
cp hadoop-snappy-0.0.1-SNAPSHOT/lib/hadoop-snappy-0.0.1-SNAPSHOT.jar /opt/hadoop/lib/
  • 配置hadoop

hadoop-env.sh增加:

vi /opt/hadoop/etc/hadoop/hadoop-env.sh
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_HOME/lib/native/:/usr/local/lib/

其他节点

  • 将主节点的相关文件拷贝到其他节点
scp hadoop-snappy-0.0.1-SNAPSHOT/lib/native/Linux-amd64-64/lib* data01:/opt/hadoop/lib/native/
scp hadoop-snappy-0.0.1-SNAPSHOT/lib/native/Linux-amd64-64/lib* data02:/opt/hadoop/lib/native/
scp hadoop-snappy-0.0.1-SNAPSHOT/lib/native/Linux-amd64-64/lib* data03:/opt/hadoop/lib/native/
scp hadoop-snappy-0.0.1-SNAPSHOT/lib/hadoop-snappy-0.0.1-SNAPSHOT.jar data01:/opt/hadoop/lib/
scp hadoop-snappy-0.0.1-SNAPSHOT/lib/hadoop-snappy-0.0.1-SNAPSHOT.jar data02:/opt/hadoop/lib/
scp hadoop-snappy-0.0.1-SNAPSHOT/lib/hadoop-snappy-0.0.1-SNAPSHOT.jar data03:/opt/hadoop/lib/
scp /usr/lib64/libsnapp* data01:/usr/local/lib
scp /usr/lib64/libsnapp* data02:/usr/local/lib
scp /usr/lib64/libsnapp* data03:/usr/local/lib
  • 其他节点配置hadoop
vi /opt/hadoop/etc/hadoop/hadoop-env.sh
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_HOME/lib/native/:/usr/local/lib/
  • 所有节点

在$HADOOP_HOME/etc/hadoop/core-site.xml文件中加入snappy配置:

<property>
    <name>io.compression.codecs</name>
    <value>
      org.apache.hadoop.io.compress.GzipCodec,
      org.apache.hadoop.io.compress.DefaultCodec,
      org.apache.hadoop.io.compress.BZip2Codec,
      org.apache.hadoop.io.compress.SnappyCodec
    </value>
</property>

验证安装结果

hadoop checknative -a | grep snappy

HBase安装Snappy库

安装过程

在上面步骤的基础上,将snappy-hadoop编译出来的lib库拷贝到hbase目录下:

cp target/hadoop-snappy-0.0.1-SNAPSHOT/lib/native/Linux-amd64-64/lib* /opt/hbase/lib/native/Linux-amd64-64/

将Hadoop lib目录下的所有本地库也拷贝到hbase中:

cp /opt/hadoop/lib/native/* /opt/hbase/lib/native/Linux-amd64-64/

测试HBase Snappy压缩可用

hbase org.apache.hadoop.hbase.util.CompressionTest file:///tmp/a.txt snappy

参考文档

  1. Hadoop,HBase 配置 安装 Snappy 终极教程
Tags: #Snappy #HBase    Post on Hadoop