软件安装路径

软件名称 版本 安装路径
jdk 1.6.0_12

/usr/java/jdk1.6.0_12
maven 3.1.0

/usr/local//apache-maven-3.1.0
Oozie 3.3.1

/usr/local/oozie-3.3.1/distro/target/oozie-3.3.1-distro/oozie-3.3.1
hadoop 1.0.3

/usr/local/hadoop-1.0.3
mysql 5.5.7

/usr/local/mysql

环境变量.bash_profile

export HADOOP_HOME="/home/tnuser/hadoop"

export HADOOP_PREFIX=$HADOOP_HOME

export HBASE_HOME="/home/tnuser/hbase"

export HIVE_HOME="/home/tnuser/hive"

export OOZIE_HOME="/home/tnuser/oozie"

export JAVA_HOME="/usr/java/jdk1.6.0_12"

export JRE_HOME=$JAVA_HOME/jre

export MAVEN_HOME="/usr/local/apache-maven-3.1.0"

export CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar:$JRE_HOME/lib

export PATH=$JAVA_HOME/bin:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HBASE_HOME/bin:$SQOOP_HOME/bin:$PIG_HOME/bin:$HIVE_HOME/bin:$MAVEN_HOME/bin:$OOZIE_HOME/bin:/usr/bin:/sbin:/usr/sbin/:$PATH

export PIG_CLASSPATH=/usr/java/jdk1..0_12/lib/tools.jar:$PIG_CLASSPATH

export HIVE_LIB=$HIVE_HOME/lib/hive-hbase-handler-0.10..jar,$HIVE_HOME/lib/hbase-0.92..jar,$HIVE_HOME/lib/zookeeper-3.4..jar,$HIVE_HOME/lib/guava-r09.jar

export "OOZIE_URL=http://localhost:11000/oozie"

家目录设置

$ ls -lh /home/tnuser/

total

lrwxrwxrwx  tnuser appuser  Jun  : hadoop -> /usr/local/hadoop-1.0.

lrwxrwxrwx  tnuser appuser  Jun  : hbase -> /usr/local/hbase-0.92.

lrwxrwxrwx  tnuser appuser  Jun  : hdfs -> /usr/local/data

drwxr-xr-x  tnuser appuser  Jun  : logs

lrwxrwxrwx  tnuser appuser  Jun  : oozie -> /usr/local/oozie-3.3./distro/target/oozie-3.3.-distro/oozie-3.3.

1. maven安装

$ wget https://archive.apache.org/dist/maven/maven-3/3.1.0/binaries/apache-maven-3.1.0-bin.tar.gz

$ tar -zxf apache-maven-3.1.-bin.tar.gz

2.hadoop安装

略

3.mysql安装

略

4. 生成Oozie安装包

$ wget https://archive.apache.org/dist/oozie/3.3.1/oozie-3.3.1.tar.gz

$ tar -zxf oozie-3.3..tar.gz

$ cd /usr/local/contentplatform/oozie-3.3./bin

$ ./mkdistro.sh -DskipTests

报错及解决

报错

ERROR] Failed to execute goal on project oozie-sharelib-hive: Could not resolve dependencies for project org.apache.oozie:oozie-sharelib-hive:jar:3.3.1: Failed to collect dependencies at org.apache.hive:hive-cli:jar:0.9.0 -> org.apache.hive:hive-metastore:jar:0.9.0 -> javax.jdo:jdo2-api:jar:2.3-ec: Failed to read artifact descriptor for javax.jdo:jdo2-api:jar:2.3-ec: Could not transfer artifact javax.jdo:jdo2-api:pom:2.3-ec from/to Codehaus repository (http://repository.codehaus.org/): repository.codehaus.org: Unknown host repository.codehaus.org -> [Help 1]

[ERROR]

[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

[ERROR] Re-run Maven using the -X switch to enable full debug logging.

[ERROR]

[ERROR] For more information about the errors and possible solutions, please read the following articles:

[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException

[ERROR]

[ERROR] After correcting the problems, you can resume the build with the command

[ERROR]   mvn <goals> -rf :oozie-sharelib-hive

ERROR, Oozie distro creation failed

解决

将安装根目录的pom.xml如下代码

        <repository>

            <id>Codehaus repository</id>

            <url>http://repository.codehaus.org/</url>

            <snapshots>

                <enabled>false</enabled>

            </snapshots>

        </repository>

替换为

        <repository>

            <id>Codehaus repository</id>

            <name>codehaus-mule-repo</name>

            <url>https://repository-master.mulesoft.org/nexus/content/groups/public/</url>

            <layout>default</layout>

        </repository>

重新运行

./mkdistro.sh -DskipTests

出现以下结果说明生成安装包成功

Oozie distro created, DATE[2019.06.18-01:40:02GMT] VC-REV[unavailable], available at [/usr/local/oozie-3.3.1/distro/target]

5.安装Oozie

5.1 启用Oozie Web Console

wget http://archive.cloudera.com/gplextras/misc/ext-2.2.zip

解压到$OOZIE_HOME根目录

5.2 复制$HADOOP_HOME以及$HADOOP_HOME/lib所有jar包到$OOZIE_HOME/libext

$ mkdir $OOZIE_HOME/libext

$ scp  $HADOOP_HOME/*.jar $OOZIE_HOME/libext

$ scp $HADOOP_HOME/lib/*.jar $OOZIE_HOME/libext

5.3 添加mysql数据库驱动到Oozie

$ wget https://cdn.mysql.com//Downloads/Connector-J/mysql-connector-java-5.1.47.tar.gz

$ mv mysql-connector-java-5.1..tar.gz $SQOOP_HOME/libext

5.4 安装Oozie

$ oozie-setup.sh -hadoop 0.20. $HADOOP_HOME -extjs ext-2.2

输出log

  setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"

INFO: Adding extension: /home/tnuser/oozie/libext/asm-3.2.jar

INFO: Adding extension: /home/tnuser/oozie/libext/aspectjrt-1.6.5.jar

INFO: Adding extension: /home/tnuser/oozie/libext/aspectjtools-1.6.5.jar

INFO: Adding extension: /home/tnuser/oozie/libext/commons-beanutils-1.7.0.jar

INFO: Adding extension: /home/tnuser/oozie/libext/commons-beanutils-core-1.8.0.jar

INFO: Adding extension: /home/tnuser/oozie/libext/commons-cli-1.2.jar

INFO: Adding extension: /home/tnuser/oozie/libext/commons-codec-1.4.jar

INFO: Adding extension: /home/tnuser/oozie/libext/commons-collections-3.2.1.jar

INFO: Adding extension: /home/tnuser/oozie/libext/commons-configuration-1.6.jar

INFO: Adding extension: /home/tnuser/oozie/libext/commons-daemon-1.0.1.jar

INFO: Adding extension: /home/tnuser/oozie/libext/commons-digester-1.8.jar

INFO: Adding extension: /home/tnuser/oozie/libext/commons-el-1.0.jar

INFO: Adding extension: /home/tnuser/oozie/libext/commons-httpclient-3.0.1.jar

INFO: Adding extension: /home/tnuser/oozie/libext/commons-lang-2.4.jar

INFO: Adding extension: /home/tnuser/oozie/libext/commons-logging-1.1.1.jar

INFO: Adding extension: /home/tnuser/oozie/libext/commons-logging-api-1.0.4.jar

INFO: Adding extension: /home/tnuser/oozie/libext/commons-math-2.1.jar

INFO: Adding extension: /home/tnuser/oozie/libext/commons-net-1.4.1.jar

INFO: Adding extension: /home/tnuser/oozie/libext/core-3.1.1.jar

INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-ant-1.0.2.jar

INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-capacity-scheduler-1.0.2.jar

INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-client-1.0.2.jar

INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-core-1.0.2.jar

INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-examples-1.0.2.jar

INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-fairscheduler-1.0.2.jar

INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-minicluster-1.0.2.jar

INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-test-1.0.2.jar

INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-thriftfs-1.0.2.jar

INFO: Adding extension: /home/tnuser/oozie/libext/hadoop-tools-1.0.2.jar

INFO: Adding extension: /home/tnuser/oozie/libext/hsqldb-1.8.0.10.jar

INFO: Adding extension: /home/tnuser/oozie/libext/jackson-core-asl-1.8.8.jar

INFO: Adding extension: /home/tnuser/oozie/libext/jackson-mapper-asl-1.8.8.jar

INFO: Adding extension: /home/tnuser/oozie/libext/jasper-compiler-5.5.12.jar

INFO: Adding extension: /home/tnuser/oozie/libext/jasper-runtime-5.5.12.jar

INFO: Adding extension: /home/tnuser/oozie/libext/jdeb-0.8.jar

INFO: Adding extension: /home/tnuser/oozie/libext/jersey-core-1.8.jar

INFO: Adding extension: /home/tnuser/oozie/libext/jersey-json-1.8.jar

INFO: Adding extension: /home/tnuser/oozie/libext/jersey-server-1.8.jar

INFO: Adding extension: /home/tnuser/oozie/libext/jets3t-0.6.1.jar

INFO: Adding extension: /home/tnuser/oozie/libext/jetty-6.1.26.jar

INFO: Adding extension: /home/tnuser/oozie/libext/jetty-util-6.1.26.jar

INFO: Adding extension: /home/tnuser/oozie/libext/jsch-0.1.42.jar

INFO: Adding extension: /home/tnuser/oozie/libext/junit-4.5.jar

INFO: Adding extension: /home/tnuser/oozie/libext/kfs-0.2.2.jar

INFO: Adding extension: /home/tnuser/oozie/libext/log4j-1.2.16.jar

INFO: Adding extension: /home/tnuser/oozie/libext/mockito-all-1.8.5.jar

INFO: Adding extension: /home/tnuser/oozie/libext/mysql-connector-java-5.1.24-bin.jar

INFO: Adding extension: /home/tnuser/oozie/libext/oro-2.0.8.jar

INFO: Adding extension: /home/tnuser/oozie/libext/servlet-api-2.5-20081211.jar

INFO: Adding extension: /home/tnuser/oozie/libext/slf4j-api-1.4.3.jar

INFO: Adding extension: /home/tnuser/oozie/libext/slf4j-log4j12-1.4.3.jar

INFO: Adding extension: /home/tnuser/oozie/libext/xmlenc-0.52.jar

Injecting following Hadoop JARs

/home/tnuser/hadoop/hadoop-core-1.0.3.jar

New Oozie WAR file with added 'Hadoop JARs, ExtJS library, JARs' at /home/tnuser/oozie/oozie-server/webapps/oozie.war

INFO: Oozie is ready to be started

5.5 生成Oozie数据库初始化文件

$ ooziedb.sh create -sqlfile oozie.sql

5.6 将初始化文件导入mysql数据库

CREATE DATABASE oozie_data DEFAULT CHARACTER SET uft8 ;

CREATE USER 'oozie'@'*' IDENTIFIED BY 'oozie';

GRANT ALL PRIVILEGES ON oozie.* TO 'oozie'@'*'

use oozie;

source oozie.sql;

5.7 配置hadoop core-site.xml，添加以下参数，如之前未添加，需要重启hadoop

<property>

  <name>hadoop.proxyuser.tnuser.hosts</name>

  <value>*</value>

</property>

<property>

  <name>hadoop.proxyuser.tnuser.groups</name>

  <value>*</value>

</property>

5.8 启动Oozie

$ oozie-start.sh

$ netstat -antl | grep

tcp               0.0.0.0:           0.0.0.0:*               LISTEN

$ oozie admin -oozie http://localhost:11000/oozie -status

System mode: NORMAL

5.9 将Oozie共享库导入hadoop

$ cd $OOZIE_HOME

$ tar xvfz oozie-sharelib-3.3..tar.gz

$ HADOOP_HOME/bin/hadoop fs -put share share

$ hadoop fs -ls /user/tnuser/share/lib

Found  items

drwxr-xr-x   - tnuser supergroup           -- : /user/tnuser/share/lib/distcp

drwxr-xr-x   - tnuser supergroup           -- : /user/tnuser/share/lib/hive

drwxr-xr-x   - tnuser supergroup           -- : /user/tnuser/share/lib/mapreduce-streaming

drwxr-xr-x   - tnuser supergroup           -- : /user/tnuser/share/lib/oozie

drwxr-xr-x   - tnuser supergroup           -- : /user/tnuser/share/lib/pig

-rw-r--r--    tnuser supergroup        -- : /user/tnuser/share/lib/sharelib.properties

drwxr-xr-x   - tnuser supergroup           -- : /user/tnuser/share/lib/sqoop

6. 测试Oozie

6.1 修改example包文件对应属性值

$ cd $OOZIE_HOME

$ tar -zxf oozie-examples.tar.gz

$ ls -lh examples/apps/map-reduce/

total .0K

-rw-r--r--  tnuser appuser  Jun  : job.properties

drwxr-xr-x  tnuser appuser    Jun  : lib

-rw-r--r--  tnuser appuser .5K Jan    workflow.xml

将job.properties文件namenode和jobtracker地址修改为实际的地址

$ cat examples/apps/map-reduce/job.properties 

nameNode=hdfs://ec2d-newcntprocnn-01:9000

jobTracker=ec2d-newcntprocnn-:

queueName=default

examplesRoot=examples

oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/apps/map-reduce

outputDir=map-reduce

6.2 上传到hdfs

$ hadoop fs -put examples examples

$ hadoop fs -ls /user/tnuser

Found  items

drwxr-xr-x   - tnuser supergroup           -- : /user/tnuser/examples

drwxr-xr-x   - tnuser supergroup           -- : /user/tnuser/share

6.3 测试Oozie

报错

$ oozie job -oozie http://localhost:11000/oozie -config examples/apps/map-reduce/job.properties -run

Error: E0902 : E0902: Exception occured: [org.apache.hadoop.ipc.RemoteException: User: tnuser is not allowed to impersonate tnuser]

$ oozie job -oozie http://localhost:11000/oozie -config examples/apps/map-reduce/job.properties -run

Error: E0902 : E0902: Exception occured: [org.apache.hadoop.ipc.RemoteException: Unauthorized connection for super-user: tnuser from IP 10.189.102.176]

经测试以下hdfs参数的value值为“*“会出现以上报错，需要修改hdfs参数为对应的真实值

<property>

  <name>hadoop.proxyuser.tnuser.hosts</name>

  <value>ec2d-newcntprocdn-04</value>

</property>

<property>

  <name>hadoop.proxyuser.tnuser.groups</name>

  <value>appuser</value>

</property>

正常运行结果

$ oozie job -oozie http://localhost:11000/oozie -config examples/apps/map-reduce/job.properties -run

job: --oozie-tnus-W

$ oozie job -oozie http://localhost:11000/oozie -info 0000000-190620070628363-oozie-tnus-W

Job ID : --oozie-tnus-W

------------------------------------------------------------------------------------------------------------------------------------

Workflow Name : map-reduce-wf

App Path      : hdfs://ec2d-newcntprocnn-01:9000/user/tnuser/examples/apps/map-reduce

Status        : RUNNING

Run           :

User          : tnuser

Group         : -

Created       : -- : GMT

Started       : -- : GMT

Last Modified : -- : GMT

Ended         : -

CoordAction ID: -

Actions

------------------------------------------------------------------------------------------------------------------------------------

ID                                                                            Status    Ext ID                 Ext Status Err Code

------------------------------------------------------------------------------------------------------------------------------------

--oozie-tnus-W@:start:                                  OK        -                      OK         -

------------------------------------------------------------------------------------------------------------------------------------

--oozie-tnus-W@mr-node                                  START_RETRY-                      -          JA009

------------------------------------------------------------------------------------------------------------------------------------

$ oozie job -oozie http://localhost:11000/oozie -info 0000000-190620070628363-oozie-tnus-W

Job ID : --oozie-tnus-W

------------------------------------------------------------------------------------------------------------------------------------

Workflow Name : map-reduce-wf

App Path      : hdfs://ec2d-newcntprocnn-01:9000/user/tnuser/examples/apps/map-reduce

Status        : SUCCEEDED

Run           :

User          : tnuser

Group         : -

Created       : -- : GMT

Started       : -- : GMT

Last Modified : -- : GMT

Ended         : -- : GMT

CoordAction ID: -

Actions

------------------------------------------------------------------------------------------------------------------------------------

ID                                                                            Status    Ext ID                 Ext Status Err Code

------------------------------------------------------------------------------------------------------------------------------------

--oozie-tnus-W@:start:                                  OK        -                      OK         -

------------------------------------------------------------------------------------------------------------------------------------

--oozie-tnus-W@mr-node                                  OK        job_201906200714_0002  SUCCEEDED  -

------------------------------------------------------------------------------------------------------------------------------------

--oozie-tnus-W@end                                      OK        -                      OK         -

------------------------------------------------------------------------------------------------------------------------------------

最新文章

  1. Android Studio22-NDK-LLDB调试
  2. Lucene系列-索引文件
  3. 移动WebApp利用Chrome浏览器进行调试
  4. Delphi的属性Property
  5. Asp.net MVC应用在IIS7上部署后403错误解决方案
  6. [转]后台页面访问权限:页面基类&amp;内置票据认证 使用方法
  7. poj2193
  8. HDU 5742 It&#39;s All In The Mind （贪心）
  9. 74LS164 for stm32 源码下载
  10. 【原】模式之-适配器Adapter模式
  11. 学习笔记_Java_day14—编码实战___一个注册页面的完整流程
  12. sql第三天
  13. 【js】操作checkbox radio 的操作总结
  14. 201521123015 《Java程序设计》第10周学习总结
  15. Python--Pycharm backup_ver1.py 控制台一直Backup FAILED
  16. Python之线程
  17. kubernetes实践之一：kubernetes二进制包安装
  18. python 操作RabbitMq详解
  19. 小A的位运算-（前缀和+位运算）
  20. JS高级 - 面向对象3（面向过程改写面向对象）

热门文章

  1. 【JZOJ5439】【NOIP2017提高A组集训10.31】Calculate
  2. TTTTTTTTTTTT POJ 2112 奶牛与机器 多重二分匹配 跑最大流 建图很经典！！
  3. [CF1223G/1240E]Wooden Raft 题解
  4. 2-sat基础详解
  5. kali VMWOOLS的安装
  6. eclipse内存溢出 参数配置
  7. JPA 开发写SQL时候遇见的困难点
  8. mysql 查询一个月的数据
  9. Netflix的Ribbon主要负载均衡策略
  10. Dubbo Monitor Simple 监控中心