pig加载两个不同字段个数的文件?load file with different items(f1有42列,f2有43列读到一个对象中)
我文章提到,加载一个文件的部分列是可行。两列,你只读一列,没问题。
但是,两个文件,f1和f2,f1有42列,f2有43列,同时加载到一个流对象,如何?
答:成功加载。但是无结构(schema unknown),discribe后看到:Schema for origin_cleaned_data unknown。
这种情况类似union,合并两个不同列的对象,会生成一个未知模式对象。
背景:因为老日志42列,新日志多加一列在第20列,因为20列后面不能同名,又要总体日志的用户点击数。所以一起加载,统一统计。
(如果知道不同日期日志的类型,则可以分别读入,指定明确模式,然后用onschema进行uion,在分别统计。可惜接受项目,不确定线上哪天改的)
采样:老日志log_without.txt,新日志log_with_android_ad_id.txt
代码如下
REGISTER piggybank.jar;
DEFINE SequenceFileLoader org.apache.pig.piggybank.storage.SequenceFileLoader();
%default cleanedLog /user/wizad/tmp/log_*
--%default cleanedLog1 /home/wizad/lmj/log_without.txt
--%default cleanedLog 2/home/wizad/lmj/log_with_android_ad_id.txt
origin_cleaned_data = LOAD '$cleanedLog' USING PigStorage(',');
DUMP origin_cleaned_data;
DESCRIBE origin_cleaned_data;
显示结果:
((null) 5,74,48809e40-b8d7-41a4-bf68-d0f8e28140ad,575356365101899146,2014-07-30 10:33:56,2014-07-30 10:33:56,1,57074,2,,,,,,,,151.87.202.1,1,-1,-1,lmj,-1,1ac2c73e-d93a-4801-a7ee-da05473d0585,48809e40-b8d7-41a4-bf68-d0f8e28140ad,02:00:00:00:00:00,1940064625594046032,,,,d70cc494,25100,206,,0,2,2,7.1,,,,42.833298,12.833298,120232210032202)
((null) 5,74,357633052513139,1033882907630785616,2014-07-30 11:15:05,2014-07-30 11:15:05,1,57074,2,,,,,,,,155.128.32.119,1,357633052513139,270f213575a4eda7,lmj,270f213575a4eda7,,,40:0e:85:40:0e:1a,-7537294162085162169,,,,7626e397,62713,206,,2,1,3,4.3,,,,37.774902,-122.4194,023010203333003)
((null) 5,74,e7a4afce-ffd9-4ecd-b916-39f9d793c218,207640323432175503,2014-07-30 10:29:22,2014-07-30 10:29:22,1,57074,2,,,,,,,,111.200.142.163,1,-1,-1,lmj,-1,14ea5e95237f34e278d7ac210173d6b8ad9d5026,e7a4afce-ffd9-4ecd-b916-39f9d793c218,02:00:00:00:00:00,1179719885610920154,,,,d4eeab6e,66104,101,,0,2,2,7.1,1,7,7,39.928894,116.388306,132100103322203)
((null) 5,74,48809e40-b8d7-41a4-bf68-d0f8e28140ad,575356365101899146,2014-07-30 10:33:56,2014-07-30 10:33:56,1,57074,2,,,,,,,,151.87.202.1,1,-1,-1,-1,1ac2c73e-d93a-4801-a7ee-da05473d0585,48809e40-b8d7-41a4-bf68-d0f8e28140ad,02:00:00:00:00:00,1940064625594046032,,,,d70cc494,25100,206,,0,2,2,7.1,,,,42.833298,12.833298,120232210032202)
((null) 5,74,302bd8f1-b974-4af5-8183-1f67d27410d6,367366268601246781,2014-07-30 10:07:57,2014-07-30 10:07:57,1,57074,2,,,,,,,,56.2.255.220,1,-1,-1,-1,c165376f9f76cf68862a505328b7ba7cd0cfa0b0,302bd8f1-b974-4af5-8183-1f67d27410d6,02:00:00:00:00:00,-488564527359896578,,,,103b14d3,25100,206,,0,2,2,7.1,,,,37.774902,-122.4194,023010203333003)
((null) 5,74,e7a4afce-ffd9-4ecd-b916-39f9d793c218,207640323432175503,2014-07-30 10:29:22,2014-07-30 10:29:22,1,57074,2,,,,,,,,111.200.142.163,1,-1,-1,-1,14ea5e95237f34e278d7ac210173d6b8ad9d5026,e7a4afce-ffd9-4ecd-b916-39f9d793c218,02:00:00:00:00:00,1179719885610920154,,,,d4eeab6e,66104,101,,0,2,2,7.1,1,7,7,39.928894,116.388306,132100103322203)
Schema for origin_cleaned_data unknown.
多了一列值为lmj的列。可以看到无结构。
union:合并不同格式的列
(union不去重复行)
A = load 'input1' as (x:int, y:float);
B = load 'input2' as (x:int, y:chararray);
C = union A, B;
describe C;
显示结果:
Schema for C unknown
两个不通列名的变量union用onschema
需要注意:使用onschema,需要所有的输入都要有明确的schema,否则,错误。因为union合并时,比较是按名字和列类型(可以自动从低级到高级类型转换)。
合并后,空出的列会补null。
A = load 'input1' as (w: chararray, x:int, y:float);
B = load 'input2' as (x:int, y:double, z:chararray);
C = union onschema A, B;
describe C;
结果:
C: {w: chararray,x: int,y: double,z: chararray}
给出一个不能union的代码例子
%default cleanedLog1 /home/wizad/lmj/log_without.txt
%default cleanedLog2 /home/wizad/lmj/log_with_android_ad_id.txt
origin1 = LOAD '$cleanedLog1' USING PigStorage(',');
origin2 = LOAD '$cleanedLog2' USING PigStorage(',');
DESCRIBE origin1
DESCRIBE origin2
origin = union origin1,origin2
结果:
origin1和origin2显示Schema for origin2 unknown.
所以origin不能生成
最新文章
- 先说IEnumerable,我们每天用的foreach你真的懂它吗?
- Windows Installer 服务启动错误 14007 的解决办法
- How GitHub Works《Github是如何工作的?》
- XCode的代码块备份
- 转载:node.js socket.io
- linux 删除某种规则命名的文件
- error: ‘for’ loop initial declarations are only allowed in C99 mode
- C#4.0新特性(3):变性 Variance(逆变与协变)
- OGNL学习-静态方法调用
- ListView列表拖拽排序
- Objective-C(十八、谓语使用及实例说明)——iOS开发基础
- PL/SQL 基础编程
- Linux下SVN安装配置全程实录(转)
- python-连接数据库
- spring异常处理
- js中escape对应的C#解码函数 UrlDecode
- VMWare安装
- Android app widget中实现跑马灯效果(非widget部件也实用)
- 微信小程序 - 弹出键盘遮挡住输入框
- Foundation框架中的NSNumber对象详解
热门文章
- java中什么是序列化和反序列化
- git报错:'fatal:remote origin already exists'怎么处理?附上git常用操作以及说明。
- 判断是否是IE9浏览器的最短语句 var ie=!-[1,]
- Android ImgView属性
- Python中的数据类型
- Docker多台物理主机之间的容器互联
- ThreadLocal 遇上线程池的问题及解决办法
- 实验与作业(Python)-文件操作
- 修改hosts立刻生效不必重启
- Swift3中dispatch_once废弃的解决办法