Hadoop源码编译(2.4.1) 2015-09-15 21:02

背景

Hadoop 2.4.1预编译版本自带的libhadoop.so是在32位机器上编译上,导致在64位OS上运行时,总出现如下告警:

WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

查看libhadoop.so的位数,显示是32位的:

file libhadoop.so.1.0.0
libhadoop.so.1.0.0: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, not stripped

为彻底解决此问题,自行编译Hadoop 2.4.1源码。

编译过程

  • 安装相关软件
yum install -y cmake autoconf automake libtool gcc zlib1g-dev pkg-config libssl-dev openssl gcc g++ make maven zlib zlib1g-dev libcurl4-o
  • 安装protobuf-2.5.0
1
2
3
4
5
6
wget https://github.com/google/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gz
tar -zxvf protobuf-2.5.0.tar.gz
cd protobuf-2.5.0
./configure
make
make install
  • 编译Hadoop源码
1
2
3
4
5
wget https://archive.apache.org/dist/hadoop/core/hadoop-2.4.1/hadoop-2.4.1-src.tar.gz
tar -zxvf hadoop-2.4.1-src.tar.gz
cd hadoop-2.4.1-src
export Platform=x64
mvn package -Pdist,native -DskipTests -Dtar

查看编译后的本地库,已经是64位:

1
2
3
cd hadoop-dist/target/hadoop-2.4.1/lib/native
file libhadoop.so.1.0.0
libhadoop.so.1.0.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
  • 拷贝到Hadoop环境中
1
2
3
4
5
cd hadoop-dist/target/hadoop-2.4.1/lib/native
cp * /opt/hadoop/lib/native/
scp * data01:/opt/hadoop/lib/native/
scp * data02:/opt/hadoop/lib/native/
scp * data03:/opt/hadoop/lib/native/
  • 修改环境变量

在etc/hadoop/hadoop-env.sh中增加:

1
2
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_HOME/lib/native/:/usr/local/lib/
export JAVA_LIBRARY_PATH=$JAVA_LIBRARY_PATH:/opt/hadoop/lib/native/

参考文档

  1. Build Hadoop Native Librairies
Tags: #Hadoop    Post on Hadoop