hadoop2.5.1搭建（一）

1.1配置

1.1.1修改hosts

vi /etc/hosts 

192.168.220.64  cluster4 

192.168.220.63 cluster3

1.2安装jdk

rpm安装

rpm -ivh jdk-7u17-linux-x64.rpm

环境变量

vi /etc/profile

#set java environment

JAVA_HOME=/usr/java/jdk1..0_17

CLASSPATH=.:$JAVA_HOME/lib.tools.jar

PATH=$JAVA_HOME/bin:$PATH

export JAVA_HOME CLASSPATH PATH

环境变量生效

source /etc/profile

链接

ln -s -f /usr/java/jdk1..0_17/jre/bin/java

ln -s -f /usr/java/jdk1..0_17/bin/javac

测试

java -version

java -version

java version "1.7.0_17"

Java(TM) SE Runtime Environment (build 1.7.0_17-b02)

Java HotSpot(TM) -Bit Server VM (build 23.7-b01, mixed mode)

安装jdk报错，但是后续貌似不影响，暂时跳过。

[root@cluster3 java]# rpm -ivh jdk-7u17-linux-x64.rpm

Preparing...                ########################################### [%]

   :jdk                    ########################################### [%]

Unpacking JAR files...

        rt.jar...

Error: Could not open input file: /usr/java/jdk1..0_17/jre/lib/rt.pack

        jsse.jar...

Error: Could not open input file: /usr/java/jdk1..0_17/jre/lib/jsse.pack

        charsets.jar...

Error: Could not open input file: /usr/java/jdk1..0_17/jre/lib/charsets.pack

        tools.jar...

Error: Could not open input file: /usr/java/jdk1..0_17/lib/tools.pack

        localedata.jar...

Error: Could not open input file: /usr/java/jdk1..0_17/jre/lib/ext/localedata.pack

1.3配置ssh公钥密钥自动登录

[root@cluster3 ~]# cd .ssh

[root@cluster3 .ssh]# ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key (/root/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /root/.ssh/id_rsa.

Your public key has been saved in /root/.ssh/id_rsa.pub.

The key fingerprint is:

::2a:0f:ce::a6:9d:d7:ed:0b:4b::6b:3d: root@cluster3

[root@cluster3 .ssh]# cat id_rsa.pub >> authorized_keys

[root@cluster3 .ssh]# scp authorized_keys root@192.168.220.64:/root/.ssh/

root@192.168.220.64's password:

authorized_keys                               %       .4KB/s   :

[root@cluster3 .ssh]# ssh root@cluster3

Last login: Thu Oct  ::  from localhost.localdomain

[root@cluster3 ~]# ssh root@cluster4

Last login: Tue Oct  ::  from 192.168.220.1

2安装hadoop

解压hadoop

tar -zxf hadoop-2.5..tar.gz

2.1修改core-site.xml

vi core-site.xml

<configuration>

    <property>

        <name>fs.defaultFS</name>

        <value>hdfs://cluster3:9000</value>

    </property>

    <property>

        <name>hadoop.tmp.dir</name>

        <value>/usr/local/hadoop/hadoop-2.5./tmp</value>

        <description>Abase for other temporary directories.</description>

    </property>

    <property>

        <name>io.file.buffer.size</name>

        <value></value>

    </property>

</configuration>

2.2修改hdfs-site.xml

vi hdfs-site.xml

<configuration>

    <property>

        <name>dfs.nameservices</name>

        <value>hadoop-cluster3</value>

    </property>

    <property>

        <name>dfs.namenode.secondary.http-address</name>

        <value>cluster3:</value>

    </property>

    <property>

        <name>dfs.namenode.name.dir</name>

        <value>file:///usr/local/hadoop/hadoop-2.5.1/dfs/name</value>

    </property>

    <property>

        <name>dfs.namenode.data.dir</name>

        <value>file:///usr/local/hadoop/hadoop-2.5.1/dfs/data</value>

    </property>

    <property>

        <name>dfs.replication</name>

        <value></value>

    </property>

    <property>

        <name>dfs.webhdfs.enabled</name>

        <value>true</value>

    </property>

</configuration>

2.3修改mapred-site.xml

cp mapred-site.xml.template mapred-site.xml

vi mapred-site.xml

<configuration>

    <property>

        <name>mapreduce.framework.name</name>

        <value>yarn</value>

    </property>

    <property>

        <name>mapreduce.jobtracker.http.address</name>

        <value>cluster3:</value>

    </property>

    <property>

        <name>mapreduce.jobhistory.address</name>

        <value>cluster3:</value>

    </property>

    <property>

        <name>mapreduce.jobhistory.webapp.address</name>

        <value>cluster3:</value>

    </property>

</configuration>

2.4修改yarn-site.xml

vi yarn-site.xml

<configuration>

<!-- Site specific YARN configuration properties -->

    <property>

        <name>yarn.nodemanager.aux-services</name>

        <value>mapreduce_shuffle</value>

    </property>

    <property>

        <name>yarn.resourcemanager.address</name>

        <value>cluster3:</value>

    </property>

    <property>

        <name>yarn.resourcemanager.scheduler.address</name>

        <value>cluster3:</value>

    </property>

    <property>

        <name>yarn.resourcemanager.resource-tracker.address</name>

        <value>cluster3:</value>

    </property>

    <property>

        <name>yarn.resourcemanager.admin.address</name>

        <value>cluster3:</value>

    </property>

    <property>

        <name>yarn.resourcemanager.webapp.address</name>

        <value>cluster3:</value>

    </property>

</configuration>

2.5修改slaves

vi slaves

cluster4

3.1修改JAVA_HOME

分别在文件hadoop-env.sh和yarn-env.sh中添加JAVA_HOME配置

vi hadoop-env.sh

export JAVA_HOME=/usr/java/jdk1..0_17

vi yarn-env.sh

export JAVA_HOME=/usr/java/jdk1..0_17

3.2hadoop环境变量

登录Master，配置Hadoop环境变量。

vi /etc/profile

export HADOOP_HOME=/usr/local/hadoop/hadoop-2.5.

export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

source /etc/profile

3.3格式化文件系统

hdfs namenode -format

3.4启动、停止服务

[root@cluster3 sbin]# start-dfs.sh

[root@cluster3 sbin]# start-yarn.sh

[root@cluster3 sbin]# stop-dfs.sh

[root@cluster3 sbin]# stop-yarn.sh

4验证

[root@cluster3 hadoop]# jps

 Jps

 ResourceManager

 NameNode

[root@cluster4 hadoop]# jps

 NodeManager

 DataNode

 Jps

查看日志

[root@cluster3 logs]# tail -n200 /usr/local/hadoop/hadoop-2.5./logs/yarn-root-resourcemanager-cluster3.log

4.2浏览器访问：

http://192.168.220.63:50070

http://192.168.220.63:8088

巴特西

hadoop2.5.1搭建（一）

最新文章

热门文章