mapreduce新旧api对比
对比:hadoop版本1.x 新版,hadoop版本0.x 旧版
1.新api引用包一般是mapreduce ,旧版api引用的包一般是mapred
2.新api使用Job,旧版api使用JobConf
3.新api类的包名使用mapreduce,旧版api使用mapred
4.新api使用job.waitForCompletion(true)提交作业,旧版api使用JobClient.runJob(job);
5.新api:extends Mapper,旧版api:extends MapRedcueBase implements Mapper
6.新api:使用上下文Context ctx ctx.write(),旧版api:使用OutputCollector<>collector Reporter collector.collect();
-------------------------
新版:
public class kpi extends Configured implements Tool
{
static class MyMapper extends Mapper<LongWritable, Text, Text, kpiwritable>
{
protected void map(LongWritable k1, Text v1, Context context)
context.write(k2, v2);
}
static class MyReducer extends Reducer<Text, kpiwritable, Text, kpiwritable>
{
protected void reduce(Text k2, java.lang.Iterable<kpiwritable> v2s, Context ctx)
context.write(k2, v2);
}
public int run(String[] arg0)
{Job job=new Job(conf,"kpi");
job.waitForCompletion(true);
return 0;
}
public static void main(String[] args) throws Exception
{
ToolRunner.run(new kpi(), args);
}
}
---------------------------------------------------------
旧版:
public class old_api
{
static class MyMapper extends MapReduceBase implements Mapper<LongWritable, Text, Text, LongWritable>{
public void map(LongWritable k1, Text v1,
OutputCollector<Text, LongWritable> collector, Reporter reporter){
collector.collect(new Text(word), new LongWritable(1));
}
}
static class MyReducer extends MapReduceBase implements Reducer<Text, LongWritable, Text, LongWritable>{
public void reduce(Text k2, Iterator<LongWritable> v2s,
OutputCollector<Text, LongWritable> collector, Reporter reporter){
collector.collect(k2, new LongWritable(times));
}
}
public static void main(String[] args) {
final JobConf job = new JobConf(conf , old_api.class);
JobClient.runJob(job);
}
}
最新文章
- Tomcat启动报错整理
- Ubuntu16.10 主题flatabulous安装
- BroadCast小结
- git 学习笔记7--branch
- 关于Nodejs的多进程模块Cluster
- git修改已提交记录的注释
- 2014 Hangjs 见闻流水账第一天
- 实时流式计算框架Storm 0.9.0发布通知(中文版)
- 【题解】【字符串】【BFS】【Leetcode】Word Ladder
- 暑假集训(4)第六弹——— 组合(poj1067)
- Java专项面试训练(一)
- Java进阶03 IO基础
- [原创].NET 业务框架开发实战之九 Mapping属性原理和验证规则的实现策略
- WAS 部署 Birt 报表出现 error.CannotStartupOSGIPlatform 和 更新web.xml
- MVP 实践
- Python第十七天 抽象类
- 【LeetCode每天一题】Remove Nth Node From End of List(移除链表倒数第N个节点)
- 第8章 传输层(7)_TCP连接管理
- unwind
- luogu4155/bzoj4444 国旗计划 (倍增)