HUE中Oozie执行Sqoop

Oozie执行Sqoop，传入参数（注意，在使用--query时，参数中一定不要带有空格，否则出错）
1. 新建一个workflow

2. 拖入一个sqoop

3. sqoop抽取命令如下（建议先在命令行试下，因为oozie的错误提示很委婉，基本看不出错误原因）

sqoop import --connect jdbc:mysql://localhost:3306/spider_new --username root --password 1234qwer --target-dir /user/fengz/brand/spider_data/amac/amac_fund_${date} --delete-target-dir --fields-terminated-by '`' --query select * from amac_fund where update_time between '${start_time}' and '${end_time}' and \$CONDITIONS -m 1

这是在命令行可执行的命令，其中有三个动态参数，${date}、${start_time}、${end_time}，此处做了一个import的导入操作，把指定数据抽取到HDFS路径下，用了--query写SQL语句抽取想要的数据，此处需要注意，如果使用--query，必须要带\$CONDITIONS，否则执行失败。
4. 填入oozie-sqoop中，需要填入的内容如下

import --connect jdbc:mysql://localhost:3306/spider_new --username root --password 1234qwer --target-dir /user/fengz/brand/spider_data/amac/amac_fund_${date} --delete-target-dir --fields-terminated-by ` --query select * from amac_fund where update_time between '${start_time}' and '${end_time}' and \$CONDITIONS -m 1

注意：指定分隔符时，要把单引号去掉，下边还有一个大坑
5. 添加参数

6. 启动，填入参数

7. 那么问题来了，报错是必须的
oozie在解析sqoop语句的时候，是根据空格来分割的，那就恶心了，日期字段中间带有空格，就像这样

Setting up log4j2

log4j2 configuration file created at /data1/yarn/nm/usercache/fengz/appcache/application_1552354446915_0759/container_1552354446915_0759_01_000001/sqoop-log4j2.xml

Sqoop command arguments :

import

--connect

jdbc:mysql://localhost:3306/spider_new

--username

root

--password

********

--target-dir

/user/fengz/brand/spider_data/amac/amac_fund_20190315

--delete-target-dir

--fields-terminated-by

`

--query

"select

*

from

amac_fund

where

update_time

between

'2019-03-14

::'

and

'2019-03-14

::'

and

\$CONDITIONS"

-m

Fetching child yarn jobs

tag id : oozie-3e829587c1f3a050998935d2c2ac3ed4

再看下oozie的错误提示，呵呵呵....发现一点鸟用都没有

No child hadoop job is executed.

java.lang.reflect.InvocationTargetException

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)

at java.lang.reflect.Method.invoke(Method.java:)

at org.apache.oozie.action.hadoop.LauncherAM.runActionMain(LauncherAM.java:)

at org.apache.oozie.action.hadoop.LauncherAM.access$(LauncherAM.java:)

at org.apache.oozie.action.hadoop.LauncherAM$.run(LauncherAM.java:)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:)

at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:)

at org.apache.oozie.action.hadoop.LauncherAM$.run(LauncherAM.java:)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:)

at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:)

Caused by: java.lang.SecurityException: Intercepted System.exit()

at org.apache.oozie.action.hadoop.security.LauncherSecurityManager.checkExit(LauncherSecurityManager.java:)

at java.lang.Runtime.exit(Runtime.java:)

at java.lang.System.exit(System.java:)

at org.apache.sqoop.Sqoop.main(Sqoop.java:)

at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:)

at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:)

at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:)

at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:)

...  more

Intercepting System.exit()

Failing Oozie Launcher, Main Class [org.apache.oozie.action.hadoop.SqoopMain], exit code []

以后用的时候注意下就好了。那这个问题怎么解决？用shell，shell脚本里边写sqoop抽取。

巴特西

HUE中Oozie执行Sqoop

最新文章

热门文章