我文章提到,加载一个文件的部分列是可行。两列,你只读一列,没问题。

但是,两个文件,f1和f2,f1有42列,f2有43列,同时加载到一个流对象,如何?

答:成功加载。但是无结构(schema unknown),discribe后看到:Schema for origin_cleaned_data unknown。

这种情况类似union,合并两个不同列的对象,会生成一个未知模式对象。

背景:因为老日志42列,新日志多加一列在第20列,因为20列后面不能同名,又要总体日志的用户点击数。所以一起加载,统一统计。

(如果知道不同日期日志的类型,则可以分别读入,指定明确模式,然后用onschema进行uion,在分别统计。可惜接受项目,不确定线上哪天改的)

采样:老日志log_without.txt,新日志log_with_android_ad_id.txt

代码如下

REGISTER piggybank.jar;

DEFINE SequenceFileLoader org.apache.pig.piggybank.storage.SequenceFileLoader();



%default cleanedLog /user/wizad/tmp/log_*



--%default cleanedLog1 /home/wizad/lmj/log_without.txt

--%default cleanedLog 2/home/wizad/lmj/log_with_android_ad_id.txt



origin_cleaned_data = LOAD '$cleanedLog' USING PigStorage(','); 



DUMP origin_cleaned_data;



DESCRIBE origin_cleaned_data;

显示结果:

((null) 5,74,48809e40-b8d7-41a4-bf68-d0f8e28140ad,575356365101899146,2014-07-30 10:33:56,2014-07-30 10:33:56,1,57074,2,,,,,,,,151.87.202.1,1,-1,-1,lmj,-1,1ac2c73e-d93a-4801-a7ee-da05473d0585,48809e40-b8d7-41a4-bf68-d0f8e28140ad,02:00:00:00:00:00,1940064625594046032,,,,d70cc494,25100,206,,0,2,2,7.1,,,,42.833298,12.833298,120232210032202)

((null) 5,74,357633052513139,1033882907630785616,2014-07-30 11:15:05,2014-07-30 11:15:05,1,57074,2,,,,,,,,155.128.32.119,1,357633052513139,270f213575a4eda7,lmj,270f213575a4eda7,,,40:0e:85:40:0e:1a,-7537294162085162169,,,,7626e397,62713,206,,2,1,3,4.3,,,,37.774902,-122.4194,023010203333003)

((null) 5,74,e7a4afce-ffd9-4ecd-b916-39f9d793c218,207640323432175503,2014-07-30 10:29:22,2014-07-30 10:29:22,1,57074,2,,,,,,,,111.200.142.163,1,-1,-1,lmj,-1,14ea5e95237f34e278d7ac210173d6b8ad9d5026,e7a4afce-ffd9-4ecd-b916-39f9d793c218,02:00:00:00:00:00,1179719885610920154,,,,d4eeab6e,66104,101,,0,2,2,7.1,1,7,7,39.928894,116.388306,132100103322203)

((null) 5,74,48809e40-b8d7-41a4-bf68-d0f8e28140ad,575356365101899146,2014-07-30 10:33:56,2014-07-30 10:33:56,1,57074,2,,,,,,,,151.87.202.1,1,-1,-1,-1,1ac2c73e-d93a-4801-a7ee-da05473d0585,48809e40-b8d7-41a4-bf68-d0f8e28140ad,02:00:00:00:00:00,1940064625594046032,,,,d70cc494,25100,206,,0,2,2,7.1,,,,42.833298,12.833298,120232210032202)

((null) 5,74,302bd8f1-b974-4af5-8183-1f67d27410d6,367366268601246781,2014-07-30 10:07:57,2014-07-30 10:07:57,1,57074,2,,,,,,,,56.2.255.220,1,-1,-1,-1,c165376f9f76cf68862a505328b7ba7cd0cfa0b0,302bd8f1-b974-4af5-8183-1f67d27410d6,02:00:00:00:00:00,-488564527359896578,,,,103b14d3,25100,206,,0,2,2,7.1,,,,37.774902,-122.4194,023010203333003)

((null) 5,74,e7a4afce-ffd9-4ecd-b916-39f9d793c218,207640323432175503,2014-07-30 10:29:22,2014-07-30 10:29:22,1,57074,2,,,,,,,,111.200.142.163,1,-1,-1,-1,14ea5e95237f34e278d7ac210173d6b8ad9d5026,e7a4afce-ffd9-4ecd-b916-39f9d793c218,02:00:00:00:00:00,1179719885610920154,,,,d4eeab6e,66104,101,,0,2,2,7.1,1,7,7,39.928894,116.388306,132100103322203)

Schema for origin_cleaned_data unknown.

多了一列值为lmj的列。可以看到无结构。

union:合并不同格式的列

(union不去重复行)

A = load 'input1' as (x:int, y:float);
B = load 'input2' as (x:int, y:chararray);
C = union A, B;
describe C;
显示结果:
Schema for C unknown

两个不通列名的变量union用onschema

需要注意:使用onschema,需要所有的输入都要有明确的schema,否则,错误。因为union合并时,比较是按名字和列类型(可以自动从低级到高级类型转换)。

合并后,空出的列会补null。

A = load 'input1' as (w: chararray, x:int, y:float);
B = load 'input2' as (x:int, y:double, z:chararray);
C = union onschema A, B;
describe C;
结果:
C: {w: chararray,x: int,y: double,z: chararray}

给出一个不能union的代码例子

%default cleanedLog1 /home/wizad/lmj/log_without.txt

%default cleanedLog2 /home/wizad/lmj/log_with_android_ad_id.txt

origin1 = LOAD '$cleanedLog1' USING PigStorage(','); 

origin2 = LOAD '$cleanedLog2' USING PigStorage(',');

DESCRIBE origin1

DESCRIBE origin2



origin = union origin1,origin2

结果:

origin1和origin2显示Schema for origin2 unknown.

所以origin不能生成

最新文章

  1. 先说IEnumerable,我们每天用的foreach你真的懂它吗?
  2. Windows Installer 服务启动错误 14007 的解决办法
  3. How GitHub Works《Github是如何工作的?》
  4. XCode的代码块备份
  5. 转载:node.js socket.io
  6. linux 删除某种规则命名的文件
  7. error: ‘for’ loop initial declarations are only allowed in C99 mode
  8. C#4.0新特性(3):变性 Variance(逆变与协变)
  9. OGNL学习-静态方法调用
  10. ListView列表拖拽排序
  11. Objective-C(十八、谓语使用及实例说明)——iOS开发基础
  12. PL/SQL 基础编程
  13. Linux下SVN安装配置全程实录(转)
  14. python-连接数据库
  15. spring异常处理
  16. js中escape对应的C#解码函数 UrlDecode
  17. VMWare安装
  18. Android app widget中实现跑马灯效果(非widget部件也实用)
  19. 微信小程序 - 弹出键盘遮挡住输入框
  20. Foundation框架中的NSNumber对象详解

热门文章

  1. java中什么是序列化和反序列化
  2. git报错:'fatal:remote origin already exists'怎么处理?附上git常用操作以及说明。
  3. 判断是否是IE9浏览器的最短语句 var ie=!-[1,]
  4. Android ImgView属性
  5. Python中的数据类型
  6. Docker多台物理主机之间的容器互联
  7. ThreadLocal 遇上线程池的问题及解决办法
  8. 实验与作业(Python)-文件操作
  9. 修改hosts立刻生效不必重启
  10. Swift3中dispatch_once废弃的解决办法