编译Hadoop

Apache Hadoop 生态圈软件下载地址：http://archive.apache.org/dist/hadoop/
hadoop下载地址 http://archive.apache.org/dist/hadoop/common

安装环境虚拟机中的 CentOS 64位

需要安装软件：

jdk: jdk-7u45-linux-x64.rpm

hadoop:hadoop-2.7.2-src.tar.gz

maven:apache-maven-3.0.5-bin.tar.gz

protobuf:protobuf-2.5.0.tar.gz

在hadoop目录下有个BUILDING.txt文件，编译源码需要准备的软件，及编译方法进行了说明，可参考使用

Requirements:

* Unix System

* JDK 1.7+

* Maven 3.0 or later

* Findbugs 1.3. (if running findbugs)

* ProtocolBuffer 2.5.

* CMake 2.6 or newer (if compiling native code), must be 3.0 or newer on Mac

* Zlib devel (if compiling native code)

* openssl devel ( if compiling native hadoop-pipes and to get the best HDFS encryption performance )

* Jansson C XML parsing library ( if compiling libwebhdfs )

* Linux FUSE (Filesystem in Userspace) version 2.6 or above ( if compiling fuse_dfs )

* Internet connection for first build (to fetch all Maven and Hadoop dependencies)

1.下载Hadoop

wget http://apache.opencas.org/hadoop/common/hadoop-2.7.2/hadoop-2.7.2-src.tar.gz

tar -zxvf hadoop-2.7.2-src.tar.gz

2.安装JDK

sudo yum install jdk-7u45-linux-x64.rpm

查看jdk安装位置：

which java

/usr/java/jdk1.7.0_45/bin/java

添加jdk到环境变量(~/.bash_profile)：

export JAVA_HOME=/usr/java/jdk1.7.0_45

export PATH=.:$JAVA_HOME/bin:$PATH

验证：

java -version

java version "1.7.0_45"

Java(TM) SE Runtime Environment (build 1.7.0_45-b18)

Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)

3、安装maven

wgethttp://apache.fayea.com/maven/maven-3/3.0.5/binaries/apache-maven-3.0.5-bin.tar.gz

tar -xzvf apache-maven-3.0.5-bin.tar.gz

添加maven到环境变量(~/.bash_profile):

export MAVEN_HOME=/home/hadoop/app/apache-maven-3.0.5

export PATH=.:$MAVEN_HOME/bin:$PATH

验证：

mvn -version

Apache Maven 3.0.5 (r01de14724cdef164cd33c7c8c2fe155faf9602da; 2013-02-19 05:51:28-0800)

Maven home: /home/hadoop/app/apache-maven-3.0.5

Java version: 1.7.0_45, vendor: Oracle Corporation

Java home: /usr/java/jdk1.7.0_45/jre

Default locale: en_US, platform encoding: UTF-8

OS name: "linux", version: "2.6.32-358.el6.x86_64", arch: "amd64", family: "unix"

如果设置代理，需要修改maven配置文件。

<proxy>

<id>optional</id>

</proxy>

下载失败可以换镜像

<name>OSChina Central</name>

<url>http://maven.oschina.net/content/groups/public/</url>

<mirrorOf>central</mirrorOf>

</mirror>

4、安装protobuf

protobuf的官方地址貌似上不了，自行下载protobuf安装包；为了编译安装protobuf，需要先gcc/gcc-c++/make

sudo apt-get install gcc

sudo apt-get install c++

sudo apt-get install cmake

sudo apt-get install gcc-c++

tar -zvxf protobuf-2.5.0.tar.gz

cd protobuf-2.5.0

./configure --prefix=/usr/local/protoc/

sudo make

sudo make install

添加protobuf到环境变量(~/.bash_profile):

export PATH=.:/usr/local/protoc/bin:$PATH

验证：

protoc --version

libprotoc 2.5.0

5、安装其他依赖

sudo yum install cmake

sudo yum install openssl-devel

sudo yum install ncurses-devel

6、编译hadoop源代码

cd ~/app/hadoop-2.7.2-src

mvn package -DskipTests -Pdist,native

编译后的代码在hadoop-2.7.2-src/hadoop-dist/target/hadoop-2.7.2下

问题：

一、编译过程中下载jar失败。

java.net.UnknownHostException: archive.apache.org

解决方式：

1.手动下载放到如下目录：

hadoop-common-project\hadoop-kms\downloads

hadoop-hdfs-project\hadoop-hdfs-httpfs\downloads

2.删除配置文件中的下载步骤：

hadoop-common-project\hadoop-kms\target\antrun\build-main.xml

二、Maven编译出现“java.lang.OutOfMemoryError: Java heap space”

解决方式：

.profile 中添加

export MAVEN_OPTS='-Xms256m -Xmx1024m'

三、编译出现分配内存空间失败

扩大虚拟机内存

四、CMake Error Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the

sudo apt-get install libssl-dev

巴特西