oozie4.3.0的安装与配置 + hadoop2.7.3
安装步骤
- mysql的配置
- oozie的安装
- oozie的配置
- oozie的启动与登录
- 常用oozie的命令
创建oozie数据库,创建oozie用户名和密码,并赋值oozie账号登陆oozie数据库的权限。
mysql -u root -proot
create database oozie;
//(设置oozie数据库的访问权限)
grant all privileges on oozie.* to 'oozie'@'%' identified by 'password';
FLUSH PRIVILEGES;
注意要删除msyql中两个空的用户,否则会一直报用户无权限。
2. oozie的安装
2.1 oozie的下载与编译
https://mirrors.tuna.tsinghua.edu.cn/apache/oozie/4.3.0/oozie-4.3.0.tar.gz
官方提供的oozie只是源码,需要自己编译,解压缩,使用以下命令编译:
./mkdistro.sh -Phadoop- -Dhadoop.auth.version=2.7. -Ddistcp.version=2.7. -Dhadoop.version=2.7. -Dsqoop.version=1.4. -DskipTests
编译过程中会有3处错误(点这里参考这篇文章),都是镜像库中包找不到,所以需要自己下载,然后放到本地maven仓库目录下即可。
编译成功后,打包的文件路径为:oozie-4.3.0/distro/target/oozie-4.3.0-distro.tar.gz
2.2 oozie的安装
1) 解压缩包oozie-4.3.0-distro.tar.gz到/usr/local/目录下,然后进入目录oozie-4.3.0,分别解压缩里面的三个压缩包oozie-client-4.3.0.tar.gz、oozie-examples.tar.gz、oozie-sharelib-4.3.0.tar.gz
完成后的文件列表如下:
2) 在hdfs上创建/user/oozie目录,然后将share目录上传到hdfs中的/user/oozie目录。
将mysql驱动和oracle驱动放到share/lib目录下, 后面使用sqoop的时候,会使用hdfs的/user/oozie/share/lib/sqoop/目录下的jar包。
cp ojdbc*.jar /usr/local/oozie-4.3./share/lib/sqoop/
cp mysql-connector-java-5.1.-bin.jar /usr/local/oozie-4.3./share/lib/sqoop/
hdfs dfs -copyFromLocal /usr/local/oozie-4.3./share/ /user/oozie
3) 在/usr/local/oozie目录下创建libext文件夹,然后复制hadoop的lib目录下的文件到/usr/local/oozie/libext下。
ln -s oozie-4.3. oozie
cd oozie
mkdir libext
cp ${HADOOP_HOME}/share/hadoop/*/*.jar libext/
cp ${HADOOP_HOME}/share/hadoop/*/lib/*.jar libext/
添加ext-2.2.zip和mysql驱动包、oracle驱动包到libext
cp ext-2.2.zip /usr/local/oozie/libext/
cp mysql-connector-java-5.1.-bin.jar /usr/local/oozie/libext/
4) 修改oozie-4.3.0/oozie-server/conf/server.xml文件,注释掉下面的记录
<!--<Listener className="org.apache.catalina.mbeans.ServerLifecycleListener" />-->
5) 打war包
在bin目录下执行命令:
./oozie-setup.sh prepare-war
war文件最终保存在/usr/local/oozie/oozie-server/webapps目录下
3. oozie的配置
3.1 设置环境变量/etc/profile
#180112 oozie path
export OOZIE_HOME=/usr/local/oozie
export PATH=$OOZIE_HOME/bin:$PATH
export OOZIE_CONFIG=/usr/local/oozie/conf
# 这里要注意地址后面要带/oozie,否则报404错误, 踩过的坑只有自己知道痛苦
export OOZIE_URL=http://dwtest-name1:11000/oozie
3.2 修改配置文件/usr/local/oozie/conf/oozie-site.xml
默认conf文件夹下的oozie-site.xml文件都是注释的,需要自己添加以下内容。
<?xml version="1.0"?>
<configuration> <!--
Refer to the oozie-default.xml file for the complete list of
Oozie configuration properties and their default values.
--> <!-- Proxyuser Configuration --> <property>
<name>oozie.service.ProxyUserService.proxyuser.hadoop.hosts</name>
<value>*</value>
<description>
List of hosts the '#USER#' user is allowed to perform 'doAs'
operations. The '#USER#' must be replaced with the username o the user who is
allowed to perform 'doAs' operations. The value can be the '*' wildcard or a list of hostnames. For multiple users copy this property and replace the user name
in the property name.
</description>
</property> <property>
<name>oozie.service.ProxyUserService.proxyuser.hadoop.groups</name>
<value>*</value>
<description>
List of groups the '#USER#' user is allowed to impersonate users
from to perform 'doAs' operations. The '#USER#' must be replaced with the username o the user who is
allowed to perform 'doAs' operations. The value can be the '*' wildcard or a list of groups. For multiple users copy this property and replace the user name
in the property name.
</description>
</property> <!-- 20180110 add -->
<property>
<name>oozie.service.JPAService.create.db.schema</name>
<value>false</value>
</property>
<property>
<name>oozie.service.JPAService.jdbc.driver</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<property>
<name>oozie.service.JPAService.jdbc.url</name>
<value>jdbc:mysql://dwtest-name1:33061/oozie?createDatabaseIfNotExist=true</value>
</property> <property>
<name>oozie.service.JPAService.jdbc.username</name>
<value>oozie</value>
</property> <property>
<name>oozie.service.JPAService.jdbc.password</name>
<value>password</value>
<description>
DB user password.
IMPORTANT: if password is emtpy leave a 1 space string, the service trims the value,
if empty Configuration assumes it is NULL.
</description>
</property> <property>
<name>oozie.service.HadoopAccessorService.hadoop.configurations</name>
<value>*=/usr/local/hadoop/etc/hadoop</value>
</property> <property>
<name>oozie.service.HadoopAccessorService.action.configurations</name>
<value>*=/usr/local/hadoop/etc/hadoop</value>
</property> <property>
<name>oozie.service.SparkConfigurationService.spark.configurations</name>
<value>*=/usr/local/spark/conf</value>
</property> <!-- 这里是保存在hdfs上的路径 -->
<property>
<name>oozie.service.WorkflowAppService.system.libpath</name>
<value>/user/oozie/share/lib</value>
</property> <property>
<name>oozie.use.system.libpath</name>
<value>true</value>
<description>
Default value of oozie.use.system.libpath. If user haven't specified =oozie.use.system.libpath=
in the job.properties and this value is true and Oozie will include sharelib jars for workflow.
</description>
</property> <property>
<name>oozie.subworkflow.classpath.inheritance</name>
<value>true</value>
</property> </configuration>
3.3 创建元数据表
在/usr/local/oozie/bin目录下执行以下命令生成sql文件,并创建元数据表
bin/ooziedb.sh create -sqlfile oozie.sql -run
可以在mysql的oozie数据库中看到以下表被创建:
3.4 修改core-site.xml文件(视具体环境设置)
如果调用oozie job的账号与hadoop不一致,则需要修改hadoop的配置文件core-site.xml,添加调用oozie job的用户组,由于我使用的都是hadoop账户,则不需要做任何修改。
修改完后,拷贝到Second name节点。使用以下命令来刷新,不需要重启hadoop集群。
scp core-site.xml hadoop@dwtest-name2:/usr/local/hadoop/etc/hadoop/
hdfs dfsadmin -refreshSuperUserGroupsConfiguration
yarn rmadmin -refreshSuperUserGroupsConfiguration
4. oozie的启动与登录
启动与停止的脚本
bin/oozied.sh start
bin/oozied.sh stop
启动时,显示如下:
登录: http://dwtest-name1:11000/oozie/
不兼容包的删除:
oozie启动后会自动将war包解压生成一个oozie文件夹。
此时需要将/usr/local/oozie/oozie-server/webapps/oozie/WEB-INF/lib中包含hadoop2.6.0版本的包删除或者移走,
否则启动job会提示错误Error, java.lang.NoSuchFieldError: HADOOP_CLASSPATH
5. 常用oozie的命令
desc |
command |
查看共享库pig包 |
oozie admin -oozie http://localhost:11000/oozie -shareliblist spark |
查看共享库 |
oozie admin -oozie http://localhost:11000/oozie -shareliblist |
提交任务 |
oozie job -oozie http://localhost:11000/oozie -config job.properties -submit |
执行任务 |
oozie job -oozie http://localhost:11000/oozie -config job.properties -run |
杀死任务 |
oozie job -oozie http://localhost:11000/oozie -kill jobid |
重新运行任务 |
oozie job -oozie http://localhost:11000/oozie -config job.properties -rerun jobid |
改变作业参数 |
oozie job -oozie http://localhost:11000/oozie -change jobid -value concurrency=1000;endtime=2018-01-10 |
检查作业状态 |
oozie job -oozie http://localhost:11000/oozie -info jobid |
查看作业日志 |
oozie job -oozie http://localhost:11000/oozie -log jobid |
检查XML是否符合规范 |
oozie calidate myapp/workflow.xml |
help查询:
oozie help calidata //查询所有命令
oozie help admin
oozie help job
最新文章
- 【转】TCP/IP协议栈及OSI参考模型详解
- CentOS搭建NodeJS环境
- LINQ To SQL
- Ubuntu进不去,显示error:unknown filesystem (最简单解决方案总结)
- 【转】select和epoll模型的差异
- JPA-一对多关系
- jquery ajax发送delete(use in jquery file upload delete file)
- 21个js 技巧收藏
- App Store 审核 IPv6 问题
- [.Net跨平台]部署DTCMS到Jexus遇到的问题及解决思路--验证码
- 原生JS实现轮播效果
- Python排序算法——快速排序
- MongoDB索引的种类与使用
- ArcGIS Python编程案例-电子资料链接
- 9款原型设计工具与Sketch的强强组合,轻松构建交互原型!
- Pytest 生成Report
- 使用Intent在活动之间穿梭
- Shell脚本编写3---Shell 传递参数
- ArcGIS Pro体验01——申请、下载、安装
- Win7操作系统安装IE10提示“安装前需要更新与安装程序版本”