琐碎-hadoop1.X和2.X的区别
2024-10-16 06:09:19
1. jobtracker做了分离，分成了resourceManager和nodemanager；
2. MR变成了和HBase和Hive等一样的yarn上面的一个应用；
3. 1.x的默认块大小为64M，2.x的默认块大小为128M；
4. 在2.x中除了datanode要向namenode报告status，nodemanager也要向ResourceManager报告status
5. MR API差别
旧的WordCount
package org.apache.hadoop.mapred;
... ...
public class WordCount extends Configured implements Tool {
public static class MapClass extends MapReduceBase
implements Mapper<LongWritable, Text, Text, IntWritable> {
... ...
public void map(LongWritable key, Text value,
OutputCollector<Text, IntWritable> output,
Reporter reporter) throws IOException {
... ...
}
}
public static class Reduce extends MapReduceBase
implements Reducer<Text, IntWritable, Text, IntWritable> {
public void reduce(Text key, Iterator<IntWritable> values,
OutputCollector<Text, IntWritable> output,
Reporter reporter) throws IOException {
... ...
}
}
static int printUsage() {
System.out.println("wordcount [-m <maps>] [-r <reduces>] <input> <output>");
ToolRunner.printGenericCommandUsage(System.out);
return -1;
}
public int run(String[] args) throws Exception {
... ...
return 0;
}
public static void main(String[] args) throws Exception {
int res = ToolRunner.run(new Configuration(), new WordCount(), args);
System.exit(res);
}
}
新的WordCount
package org.apache.hadoop.examples;
... ...
public class WordCount {
public static class TokenizerMapper
extends Mapper<Object, Text, Text, IntWritable>{
... ...
public void map(Object key, Text value, Context context
) throws IOException, InterruptedException {
... ...
}
}
public static class IntSumReducer
extends Reducer<Text,IntWritable,Text,IntWritable> {
private IntWritable result = new IntWritable();
public void reduce(Text key, Iterable<IntWritable> values,
Context context
) throws IOException, InterruptedException {
... ...
}
}
public static void main(String[] args) throws Exception {
... ...
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}
6.
最新文章
- 使用jOrgChart插件， 异步加载生成组织架构图
- Python3 Socket网络编程
- evil program
- CSS3如何去除 inline block 元素之间多出的空格
- 关于java按位操作运算
- Oracle函数题
- Linux指令--cat,tac
- 读取FTP 图片文件，并显示，非下载
- 深度优先遍历（DFS）（转）
- linux 进阶命令
- 前端里面如何进行搜索引擎优化(SEO)
- 【学习】数据处理基础知识（汇总和计算描述统计）【pandas】
- Partition Numbers的计算
- 进程池的map方法
- SQLServer SQL连接查询深度探险(摘录
- Android笔记之 网络http通信
- NLayerAppV3--DDD之领域层
- SQL Server 跨服务器快速数据转移
- 图解在Eclipse中如何上传项目到svn
- vim添加复制（crtl+c）,粘贴（ctrl+v）ctrl+A 等快捷键