Hadoop2学习记录（1） |HA完全分布式集群搭建

准备

系统：CentOS 6或者RedHat 6（这里用的是64位操作）

软件：JDK 1.7、hadoop-2.3.0、native64位包（可以再csdn上下载，这里不提供了）

部署规划

192.168.1.11 C6H1 NameNode、DataNode、ResourceManager、NodeManager、JournalNode

192.168.1.12 C6H2 NameNode、DataNode、JournalNode、NodeManager

192.168.1.13 C6H3 DataNode、JourNode、NodeManager

配置过程

1、关闭相关服务、配置HOSTS文件、解压缩包

chkconfig iptables off

service iptables stop #关闭防火墙

vi /etc/selinux/config

SELINUX=disabled #注销以前的，添加这个或者直接改。

:wq

setenforce 0 #强制关闭selinux

#设置hosts，每台都设置

vi /etc/hosts

192.168.1.11 C6H1

192.168.1.12 C6H2

192.168.1.13 C6H3

tar –zxvf hadoop-2.3.0.tar.gz –C /usr/local

目录改名为hadoop2

2、安装JDK

tar –zxvf jdk-1.7.xx.tar.gz –C /usr/src

cd /usr/src

mv /usr/src/jdk-1.7.xx /usr/local/jdk

vi /etc/profile

添加环境变量，这里一次性把所有的环境变量都添加了

export JAVA_HOME=/usr/local/jdk

export ZOOKEEPER_HOME=/usr/local/zk

export HADOOP_HOME=/usr/local/hadoop2

export PATH=.:$JAVA_HOME/bin:$ZOOKEEPER_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

:wq #保存退出

source /etc/profile #立即生效

验证

java –version

java version "1.7.0_51"

Java(TM) SE Runtime Environment (build 1.7.0_51-b13)

Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)

3、配置密钥登陆

ssh-keygen –t rsa #生成密钥，一路回车。每台机器上都执行一遍

scp /root/.ssh/id_rsa.pub root@C6H1:/root/C6H2_key #分别将C6H2\C6H3上的公钥传到C6H1中。

在C6H1上操作：

cat /root/.ssh/id_rsa.pub > /root/.ssh/authorized_keys

cat /root/C6H2_key >> /root/.ssh/authorized_keys #>>代表追加，一个>覆盖了内容

cat /root/C6H3_key >> /root/.ssh/authorized_keys

将C6H1中的文件拷贝到C6H2\C6H3机器的/root/.ssh/目录下，这样机器之间免密码可以登陆了。

4、配置core-site.xml

<!—设置集群的名称 -- >

<name>fs.defaultFS</name>

<value>hdfs://cluster1</value>

</property>

<! – 设置目录存储的位置，默认namenode、datanode都存储在这里目录下 -- >

<name>hadoop.tmp.dir</name>

<value>/data/dfs/hadoop</value>

</property>

</configuration>

5、配置hdfs-site.xml

<! – 副本数，默认3个 -- >

<name>dfs.replication</name>

</property>

<! – 设置集群名称 -- >

<name>dfs.nameservices</name>

<value>cluster1</value>

</property>

<! – 设置集群中的NameNode节点-- >

<name>dfs.ha.namenodes.cluster1</name>

</property>

<! –- 设置集群中的C6H1的namenode的rpc访问地址和端口 -- >

<name>dfs.namenode.rpc-address.cluster1.C6H1</name>

</property>

<! –- 设置集群中的C6H2的namenode的rpc访问地址和端口 -- >

<name>dfs.namenode.rpc-address.cluster1.C6H2</name>

</property>

<! –- 设置集群中的C6H1的namenode的http访问地址和端口 -- >

<name>dfs.namenode.http-address.cluster1.C6H1</name>

</property>

<! –- 设置集群中的C6H2的namenode的rpc访问地址和端口 -- >

<name>dfs.namenode.http-address.cluster1.C6H2</name>

</property>

<! –- 设置namenode的元数据信息都保存在journal集群中 -- >

<name>dfs.namenode.shared.edits.dir</name>

<value>qjournal://C6H1:8485;C6H2:8485;C6H3:8485/cluster1</value>

</property>

<!-- 设置cluster1故障时，哪一个实现类指定故障切换 -- >

<name>dfs.client.failover.proxy.provider.cluster1</name>

<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>

</property>

<! -- 设置NameNode切换的操作方式，使用ssh操作 -- >

<name>dfs.ha.fencing.methods</name>

<value>sshfence</value>

</property>

<!-- 设置密钥保存位置 -- >

<name>dfs.ha.fencing.ssh.private-key-file</name>

</property>

<! -- 指定journalNode集群对NameNode的目录进行共享时，自己存储在磁盘的路径-- >

<name>dfs.journalnode.edits.dir</name>

<value>/data/dfs/journal</value>

</property>

<! -- 设置namenode存储在磁盘的路径 -->

<name>dfs.namenode.name.dir</name>

</property>

<! -- 设置datanode存储在磁盘的路径 -- >

<name>dfs.datanode.data.dir</name>

</property>

<! -- 开启web端访问FS -- >

<name>dfs.webhdfs.enabled</name>

</property>

</configuration>

6、配置mapred-site.xml

<! -- 与Hadoop1不一样的这里设置yarn方式执行mapreduce -- >

<name>mapreduce.framework.name</name>

</property>

</configuration>

7、配置yarn-site.xml

<! -- 设置reourcemanager主机，这里只能设置一个，有单点隐患！ -- >

<name>yarn.resourcemanager.hostname</name>

</property>

<! -- 设置aux-services，mapreduce_shuffle -- >

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

</configuration>

8、配置yarn-env.sh

export JAVA_HOME=/usr/local/jdk #设置hadoop调用的JAVA_HOME路径

9、配置mapred-env.sh

export JAVA_HOME=/usr/local/jdk #设置hadoop调用的JAVA_HOME路径

10、配置hadoop-env.sh

export JAVA_HOME=/usr/local/jdk #设置hadoop调用的JAVA_HOME路径

11、配置slaves

vi /usr/local/hadoop2/etc/hadoop/slaves

C6H1

C6H2

C6H3

每行一个主机名

12、第一次初始化启动过程

初始化跟hadoop1不同，按照步骤来操作，如果重复格式化需要删除 /data/dfs/中的所有目录，也就是hadoop.tmp.dir设置的路径。

1、分别在三台机器上启动JournalNode

hadoop-daemon.sh start journalnode

2、在C6H1格式化NameNode

hdfs namenode –format

3、在C6H1上启动namenode

hadoop-daemon.sh start namenode

4、在C6H2上格式化另一个NameNode，需要同步C6H1上的NameNode数据。

hdfs namenode –bootstrapStandby

5、启动另一个NameNode

hadoop-daemon.sh start namenode

6、关闭NameNode，启动所有的hadoop所有服务

stop-all.sh

start-all.sh #以后启动直接使用这个命令就行，第一次初始化必须按照以上步骤操作。

启动HDFS 的HA自动切换

hdfs haadmin –failover –forceactive CH61 C6H2

Failover from C6H1 to C6H2 successful

13、测试HDFS

SHELL测试创建文件夹

hadoop fs –mkdir /data

hadoop fs –ls /

14、测试MapReduce

vi /root/word.text

hello you

hello me

上传一个文本文件

hadoop fs –put /root/word.text /

使用自带的测试包测试wordcount

格式 hadoop jar jar包路径 wordcount hdfs输入路径输出路径（必须不存在的，会自动创建）

[root@C6H1 hadoop]# hadoop jar /usr/local/hadoop2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.3.0.jar wordcount /word.text /word_out1

14/03/16 09:36:21 INFO client.RMProxy: Connecting to ResourceManager at C6H1/192.168.1.11:8032

14/03/16 09:36:22 INFO input.FileInputFormat: Total input paths to process : 1

14/03/16 09:36:22 INFO mapreduce.JobSubmitter: number of splits:1

14/03/16 09:36:23 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1394933446304_0001

14/03/16 09:36:23 INFO impl.YarnClientImpl: Submitted application application_1394933446304_0001

14/03/16 09:36:23 INFO mapreduce.Job: The url to track the job: http://C6H1:8088/proxy/application_1394933446304_0001/

14/03/16 09:36:23 INFO mapreduce.Job: Running job: job_1394933446304_0001

14/03/16 09:36:31 INFO mapreduce.Job: Job job_1394933446304_0001 running in uber mode : false

14/03/16 09:36:31 INFO mapreduce.Job: map 0% reduce 0%

14/03/16 09:36:38 INFO mapreduce.Job: map 100% reduce 0%

14/03/16 09:36:44 INFO mapreduce.Job: map 100% reduce 100%

14/03/16 09:36:45 INFO mapreduce.Job: Job job_1394933446304_0001 completed successfully

14/03/16 09:36:45 INFO mapreduce.Job: Counters: 49

File System Counters

FILE: Number of bytes read=48

FILE: Number of bytes written=173817

FILE: Number of read operations=0

FILE: Number of large read operations=0

FILE: Number of write operations=0

HDFS: Number of bytes read=108

HDFS: Number of bytes written=26

HDFS: Number of read operations=6

HDFS: Number of large read operations=0

HDFS: Number of write operations=2

Job Counters

Launched map tasks=1

Launched reduce tasks=1

Data-local map tasks=1

Total time spent by all maps in occupied slots (ms)=4262

Total time spent by all reduces in occupied slots (ms)=3556

Total time spent by all map tasks (ms)=4262

Total time spent by all reduce tasks (ms)=3556

Total vcore-seconds taken by all map tasks=4262

Total vcore-seconds taken by all reduce tasks=3556

Total megabyte-seconds taken by all map tasks=4364288

Total megabyte-seconds taken by all reduce tasks=3641344

Map-Reduce Framework

Map input records=2

Map output records=4

Map output bytes=34

Map output materialized bytes=48

Input split bytes=90

Combine input records=4

Combine output records=4

Reduce input groups=4

Reduce shuffle bytes=48

Reduce input records=4

Reduce output records=4

Spilled Records=8

Shuffled Maps =1

Failed Shuffles=0

Merged Map outputs=1

GC time elapsed (ms)=152

CPU time spent (ms)=1330

Physical memory (bytes) snapshot=308592640

Virtual memory (bytes) snapshot=1708167168

Total committed heap usage (bytes)=136450048

Shuffle Errors

BAD_ID=0

CONNECTION=0

IO_ERROR=0

WRONG_LENGTH=0

WRONG_MAP=0

WRONG_REDUCE=0

File Input Format Counters

Bytes Read=18

File Output Format Counters

Bytes Written=26

集群搭建参考吴超-沉思录博客,转载请注明出处,谢谢!

巴特西

Hadoop2学习记录（1） |HA完全分布式集群搭建

准备

部署规划

配置过程

1、关闭相关服务、配置HOSTS文件、解压缩包

2、安装JDK

3、配置密钥登陆

4、配置core-site.xml

5、配置hdfs-site.xml

6、配置mapred-site.xml

7、配置yarn-site.xml

8、配置yarn-env.sh

9、配置mapred-env.sh

10、配置hadoop-env.sh

11、配置slaves

12、第一次初始化启动过程

13、测试HDFS

14、测试MapReduce

最新文章

热门文章