Hadoop入门学习笔记-第三天（Yarn高可用集群配置及计算案例）

什么是mapreduce

首先让我们来重温一下 hadoop 的四大组件：
HDFS：分布式存储系统
MapReduce：分布式计算系统
YARN： hadoop 的资源调度系统
Common：以上三大组件的底层支撑组件，主要提供基础工具包和 RPC 框架等

Mapreduce 是一个分布式运算程序的编程框架，是用户开发“基于 hadoop 的数据分析应用”的核心框架
Mapreduce 核心功能是将用户编写的业务逻辑代码和自带默认组件整合成一个完整的分布式运算程序，并发运行在一个 hadoop 集群上

一、yarn单机版配置
1.修改mapred-site.xml 文件指定调度方式为yarn
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
2.修改yarn-site.xml 配置yarn服务器
<property>
<name>yarn.resourcemanager.hostname</name>
<value>nameNode10.com</value>
</property>

<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
3.在指定机器上面启动./start-yarn.sh

二、yarn ha高可用集群配置
1.修改yarn-site.xml 配置yarn集群

<property>
<name>yarn.resourcemanager.connect.retry-interval.ms</name>
<value>2000</value>
</property>

<property>
<name>yarn.resourcemanager.ha.enabled</name>
<value>true</value>
</property>

<property>
<name>yarn.resourcemanager.ha.automatic-failover.enabled</name>
<value>true</value>
</property>

<property>
<name>yarn.resourcemanager.ha.automatic-failover.embedded</name>
<value>true</value>
</property>

<property>
<name>yarn.resourcemanager.cluster-id</name>
<value>yarn-rm-cluster</value>
</property>

<property>
<name>yarn.resourcemanager.ha.rm-ids</name>
<value>rm1,rm2</value>
</property>

<property>
<name>yarn.resourcemanager.hostname.rm1</name>
<value>nameNode10.com</value>
</property>

<property>
<name>yarn.resourcemanager.hostname.rm2</name>
<value>dateNode30.com</value>
</property>

<property>
<name>yarn.resourcemanager.recovery.enabled</name>
<value>true</value>
</property>

<property>
<name>yarn.resourcemanager.zk.state-store.address</name>
<value>nameNode10.com:2181,dataNode20.com:2181,nameNode40.com:2181</value>
</property>

<property>
<name>yarn.resourcemanager.zk-address</name>
<value>nameNode10.com:2181,dataNode20.com:2181,nameNode40.com:2181</value>
</property>

<property>
<name>yarn.resourcemanager.address.rm1</name>
<value>nameNode10.com:8032</value>
</property>

<property>
<name>yarn.resourcemanager.scheduler.address.rm1</name>
<value>nameNode10.com:8034</value>
</property>

<property>
<name>yarn.resourcemanager.webapp.address.rm1</name>
<value>nameNode10.com:8088</value>
</property>

<property>
<name>yarn.resourcemanager.address.rm2</name>
<value>nodeNode30.com:8032</value>
</property>

<property>
<name>yarn.resourcemanager.scheduler.address.rm2</name>
<value>nodeNode30.com:8034</value>
</property>

<property>
<name>yarn.resourcemanager.webapp.address.rm2</name>
<value>nodeNode30.com:8088</value>
</property>

<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>

2.输入hadoop classpath 会出现以下路径(这步不配置，执行自带案例demo时会报错)
/software/hadoop-3.2.0/etc/hadoop:/software/hadoop-3.2.0/share/hadoop/common/lib/*:/software/hadoop-3.2.0/share/hadoop/common/*:/software/hadoop-3.2.0/share/hadoop/hdfs:/software/hadoop-3.2.0/share/hadoop/hdfs/lib/*:/software/hadoop-3.2.0/share/hadoop/hdfs/*:/software/hadoop-3.2.0/share/hadoop/mapreduce/lib/*:/software/hadoop-3.2.0/share/hadoop/mapreduce/*:/software/hadoop-3.2.0/share/hadoop/yarn:/software/hadoop-3.2.0/share/hadoop/yarn/lib/*:/software/hadoop-3.2.0/share/hadoop/yarn/*
将上面路径拷贝到 yarn-site.xml文件中
<property>
<name>yarn.application.classpath</name>
<value>/software/hadoop-3.2.0/etc/hadoop:/software/hadoop-3.2.0/share/hadoop/common/lib/*:/software/hadoop-3.2.0/share/hadoop/common/*:/software/hadoop-3.2.0/share/hadoop/hdfs:/software/hadoop-3.2.0/share/hadoop/hdfs/lib/*:/software/hadoop-3.2.0/share/hadoop/hdfs/*:/software/hadoop-3.2.0/share/hadoop/mapreduce/lib/*:/software/hadoop-3.2.0/share/hadoop/mapreduce/*:/software/hadoop-3.2.0/share/hadoop/yarn:/software/hadoop-3.2.0/share/hadoop/yarn/lib/*:/software/hadoop-3.2.0/share/hadoop/yarn/*</value>
</property>

3.单步骤启动
yarn-daemon.sh start resourcemanager
yarn-daemon.sh start nodemanager
4.集成启动 yarn-start.sh

5.访问控制台查看(访问备用节点会自动跳转到主节点控制台)
http://datanode30.com:8088/cluster/
http://namenode10.com:8088/cluster/
主备状态查看
yarn rmadmin -getServiceState rm1

6.执行例子
1.计算圆周率 /software/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.0.jar pi 10 10
2.统计文中每个单词个数
hadoop fs -mkdir /input
hadoop fs -put -f ./file/file1.txt /input
hadoop jar /software/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.0.jar wordcount /input /output
hadoop fs -ls /output
hadoop fs -cat /output/part-r-00000

巴特西

Hadoop入门学习笔记-第三天（Yarn高可用集群配置及计算案例）

最新文章

热门文章