MapReduce(1): Prepare input for Mappers
According to Wikipedia MapReduce, there are two ways to illustrate MapReduce. One contains three steps: Map, Shuffle and Reduce; Another one with 5 steps is my preference:
a. Prepare the Map() input,
b. Run the user-provided Map() code
c. "Shuffle" the Map output to the Reduce processors,
d. Run the user-provided Reduce() code,
e. Produce the final output
This blog focuses on how to prepare the Map() input:
1. Block and InputSplit:
As shown in the HDFS blogs, super huge dataset is physically stored in HDFS. But Mappers do not directly process physical blocks, instead InputSplits converts the physical representation of the block into logical for the Hadoop Mappers.
InputSplit is the logical representation of data. It describes a unit of work that contains a single map task in a MapReduce program. It is created by InputFormat. FileInputFormat, by default, breaks a file into 128MB chunks (same as blocks in HDFS),framework assigns one split to each Map function. Inputsplit does not contain the input data; it is just a reference to the data.
2. RecordReader:
It determines how an InputSplit is passed into a Map function. The RecordReader instance is defined by the InputFormat. By default, it uses TextInputFormat for converting data into a key-value pair. TextInputFormat provides 2 types of RecordReaders: LineRecordReader, SequenceFileRecordReader
References:
https://hadoopabcd.wordpress.com/2015/03/10/hdfs-file-block-and-input-split/
https://en.wikipedia.org/wiki/MapReduce
https://data-flair.training/blogs/shuffling-and-sorting-in-hadoop/
https://zhuanlan.zhihu.com/p/34849261
https://www.edureka.co/blog/mapreduce-tutorial/
最新文章
- VS2015开发Android，自带模拟器无法调试、加载程序，算是坑吗
- 当Table中td内容为空时，显示边框的办法
- 解决Java接口内部类的main()方法无法打印输出的问题
- JSON.stringify()、JSON.parse()和eval(string)
- 我的WCF Data Service 系列 （一、为什么要有WCF Data Service）
- 分享Kali Linux 2016.2第43周镜像
- Windows CMD命令大全（转）
- 找区间连续值(HDU5247)
- 算法导论----VLSI芯片测试； n个手机中过半是好的，找出哪些是好手机
- JavaScript面向对象编程指南
- hadoop-2.0.0-mr1-cdh4.2.0源码编译总结
- python socketserver框架解析
- 折腾Java设计模式之备忘录模式
- 聚类——FCM
- SMTP 协议
- Python多进程操作同一个文件，文件锁问题
- hdu 4277 USACO ORZ dfs+hash
- 自定义ListView 、GradView 重写onMeasure方法让其正确显示
- JSP概述、API、注释
- collections集合的总括。