6、Hadoop 2.6.0 运行
2024-09-06 03:20:39
运行方式
- Local (Standalone) Mode
- Pseudo-Distributed Mode
- Fully-Distributed Mode
Standalone Operation
$ sudo mkdir input
$ sudo cp etc/hadoop/*.xml input
$ sudo bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar grep input output 'dfs[a-z.]+'
$ cat output/*
Pseudo-Distributed Operation
Configuration
Use the following:
etc/hadoop/core-site.xml:
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://hw-TM1612:8020</value>
</property>
</configuration>
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/hadoop-2.6.0-cdh5.5.0/data/tmp</value>
</property>
</configuration>
etc/hadoop/hdfs-site.xml:
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
sudo bin/hdfs namenode -format ##格式化namenode
sudo sbin/hadoop-daemon.sh start namenode ##启动namenode
sudo sbin/hadoop-daemon.sh start datanode ##启动datanode
jps
sudo bin/hdfs dfs -mkdir -p /user/hw/ ##创建目录
sudo bin/hdfs dfs -ls -R / ##循环列出目录 -R
sudo bin/hdfs dfs -mkdir -p /user/hw/mapreduce/wordcount/input ##创建hdfs文件
sudo bin/hdfs dfs -put wcinput/wc.input /user/hw/mapreduce/wordcount/input ## 上传文件
sudo bin/hdfs dfs -ls /user/hw/mapreduce/wordcount/input
sudo bin/hdfs dfs -cat /user/hw/mapreduce/wordcount/input/wc.input ##查看
sudo bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-cdh5.5.0.jar wordcount /user/hw/mapreduce/wordcount/input/ /user/hw/mapreduce/wordcount/output ##hdfs下运行
sudo bin/hdfs dfs -cat /user/hw/mapreduce/wordcount/output/part-r-00000
- 默认web端口号：50070
YARN on Single Node
etc/hadoop/mapred-site.xml:
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
etc/hadoop/yarn-site.xml:
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
sudo gedit etc/hadoop/yarn-env.sh
sudo gedit etc/hadoop/yarn-site.xml
sudo gedit etc/hadoop/slaves ##更改主机名
sudo sbin/yarn-daemon.sh start resourcemanager ##启动resourcemanager
jps
sudo sbin/yarn-daemon.sh start nodemanager ##启动nodemanager
jps
sudo gedit etc/hadoop/mapred-env.sh
sudo mv etc/hadoop/mapred-site.xml.template etc/hadoop/mapred-site.xml
sudo gedit etc/hadoop/mapred-site.xml
sudo bin/hdfs dfs -rm -r /user/hw/mapreduce/wordcount/output
sudo bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-cdh5.5.0.jar wordcount /user/hw/mapreduce/wordcount/input/ /user/hw/mapreduce/wordcount/output
- 默认web端口号：8088
启动历史服务器
sudo sbin/mr-jobhistory-daemon.sh start historyserver
最新文章
- 虚基类&虚继承
- jstl param url redirect import
- [linux] 线程和wait命令，sleep命令
- 什么是JavaScript闭包终极全解之一——基础概念
- NEERC 2013, Eastern subregional contest
- 互联网产品设计常用文档类型-BRD、MRD、PRD、FSD (
- Table of Contents - JAXB
- php框架－yii
- linux系统中中断已连接的用户
- linux之线程之互斥
- 一天搞定CSS：盒模型content、padding、border、margin--06
- input type="radio" 赋值问题
- BZOJ 2565: 最长双回文串 [Manacher]
- MYSQL数据库学习九 数据的操作
- Object.clone()方法与对象的深浅拷贝
- Codeforces Global Round 2
- UNIX高级环境编程（14）文件IO - O_DIRECT和O_SYNC详解 < 海棠花溪 >
- Oracle递归查询父子兄弟节点
- Centos6.4安装配置sendmail
- Java 8 – Convert List to Map