机器规划：

主机 ip 进程
hadoop1 10.183.225.158 hive server
hadoop2 10.183.225.166 hive client

前置条建：

kerberos部署：http://www.cnblogs.com/kisf/p/7473193.html

Hadoop  HA + kerberos部署：http://www.cnblogs.com/kisf/p/7477440.html

mysql安装：略

添加hive用户名，及数据库。mysql -uhive -h10.112.28.179 -phive123456

hive使用2.3.0版本：

wget http://mirror.bit.edu.cn/apache/hive/hive-2.3.0/apache-hive-2.3.0-bin.tar.gz

添加环境变量：

export HIVE_HOME=/letv/soft/apache-hive-2.3.0-bin

export HIVE_CONF_DIR=$HIVE_HOME/conf

export PATH=\$PATH:\$HIVE_HOME/bin

同步至master2，并 source /etc/profile

解压：　　

tar zxvf apache-hive-2.3.0-bin.tar.gz

　　

kerberos生成keytab:

addprinc -randkey hive/hadoop1@JENKIN.COM

addprinc -randkey hive/hadoop2@JENKIN.COM

xst -k /var/kerberos/krb5kdc/keytab/hive.keytab hive/hadoop1@JENKIN.COM

xst -k /var/kerberos/krb5kdc/keytab/hive.keytab hive/hadoop2@JENKIN.COM

　　

拷贝至hadoop2

scp /var/kerberos/krb5kdc/keytab/hive.keytab hadoop1:/var/kerberos/krb5kdc/keytab/

scp /var/kerberos/krb5kdc/keytab/hive.keytab hadoop2:/var/kerberos/krb5kdc/keytab/

（使用需要kinit）　　

hive server 配置：

hive server hive-env.sh增加：　　

HADOOP_HOME=/xxx/soft/hadoop-2.7.3

export HIVE_CONF_DIR=/xxx/soft/apache-hive-2.3.0-bin/conf

export HIVE_AUX_JARS_PATH=/xxx/soft/apache-hive-2.3.0-bin/lib

　　

hive server上增加hive-site.xml：

<configuration>

    <property>

           <name>hive.metastore.schema.verification</name>

           <value>false</value>

           <description>

              Enforce metastore schema version consistency.

                  True: Verify that version information stored in metastore matches with one from Hive jars.  Also disable automatic

                        schema migration attempt. Users are required to manully migrate schema after Hive upgrade which ensures

                        proper metastore schema migration. (Default)

                  False: Warn if the version information stored in metastore doesn't match with one from in Hive jars.

            </description>

    </property>

    <property>

            <name>hive.metastore.warehouse.dir</name>

            <value>/user/hive/warehouse</value>

            <description>location of default database for the warehouse</description>

    </property>

    <property>

            <name>hive.querylog.location</name>

            <value>/xxx/soft/apache-hive-2.3.0-bin/log</value>

            <description>Location of Hive run time structured log file</description>

    </property>

    <property>

            <name>hive.downloaded.resources.dir</name>

            <value>/xxx/soft/apache-hive-2.3.0-bin/tmp</value>

            <description>Temporary local directory for added resources in the remote file system.</description>

    </property>

    <property>

            <name>javax.jdo.option.ConnectionURL</name>

            <value>jdbc:mysql://10.112.28.179:3306/hive?createDatabaseIfNotExist=true&iuseUnicode=true&characterEncoding=utf-8&useSSL=false</value<configuration>

            <description>JDBC connect string for a JDBC metastore</description>

    </property>

    <property>

            <name>javax.jdo.option.ConnectionDriverName</name>

            <value>com.mysql.jdbc.Driver</value>

            <description>Driver class name for a JDBC metastore</description>

    </property>

    <property>

            <name>javax.jdo.option.ConnectionUserName</name>

            <value>hive</value>

            <description>username to use against metastore database</description>

    </property>

    <property>

            <name>javax.jdo.option.ConnectionPassword</name>

            <value>hive123456</value>

            <description>password to use against metastore database</description>

    </property>

<!-- kerberos config -->

    <property>

        <name>hive.server2.authentication</name>

        <value>KERBEROS</value>

    </property>

    <property>

        <name>hive.server2.authentication.kerberos.principal</name>

        <value>hive/_HOST@JENKIN.COM</value>

    </property>

    <property>

        <name>hive.server2.authentication.kerberos.keytab</name>

        <value>/var/kerberos/krb5kdc/keytab/hive.keytab</value>

        <!-- value>/xxx/soft/apache-hive-2.3.0-bin/conf/keytab/hive.keytab</value -->

    </property>

    <property>

        <name>hive.metastore.sasl.enabled</name>

        <value>true</value>

    </property>

    <property>

        <name>hive.metastore.kerberos.keytab.file</name>

        <value>/var/kerberos/krb5kdc/keytab/hive.keytab</value>

    </property>

    <property>

        <name>hive.metastore.kerberos.principal</name>

        <value>hive/_HOST@JENKIN.COM</value>

    </property>

　　

hadoop namenode core-site.xml增加配置：

<!-- hive congfig  -->

        <property>

                <name>hadoop.proxyuser.hive.hosts</name>

                <value>*</value>

        </property>

        <property>

                <name>hadoop.proxyuser.hive.groups</name>

                <value>*</value>

        </property>

        <property>

                <name>hadoop.proxyuser.hdfs.hosts</name>

                <value>*</value>

        </property>

        <property>

                <name>hadoop.proxyuser.hdfs.groups</name>

                <value>*</value>

        </property>

        <property>

                <name>hadoop.proxyuser.HTTP.hosts</name>

                <value>*</value>

        </property>

        <property>

                <name>hadoop.proxyuser.HTTP.groups</name>

                <value>*</value>

         </property>

　　同步是其他机器。

scp etc/hadoop/core-site.xml master2:/xxx/soft/hadoop-2.7.3/etc/hadoop/

scp etc/hadoop/core-site.xml slave2:/xxx/soft/hadoop-2.7.3/etc/hadoop/

　　

JDBC下载：

wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.44.tar.gz

tar zxvf mysql-connector-java-5.1.44.tar.gz

复制到hive lib目录：

cp mysql-connector-java-5.1.44/mysql-connector-java-5.1.44-bin.jar apache-hive-2.3.0-bin/lib/

客户端配置：

将hive拷贝至hadoop2

scp -r apache-hive-2.3.0-bin/ hadoop2:/xxx/soft/

　　

在hadoop2上(client)：

hive-site.xml

<configuration>

    <property>

        <name>hive.metastore.uris</name>

        <value>thrift://hadoop1:9083</value>

    </property>

    <property>

         <name>hive.metastore.local</name>

         <value>false</value>

    </property>

    <!-- kerberos config -->

    <property>

        <name>hive.server2.authentication</name>

        <value>KERBEROS</value>

    </property>

    <property>

        <name>hive.server2.authentication.kerberos.principal</name>

        <value>hive/_HOST@JENKIN.COM</value>

    </property>

    <property>

        <name>hive.server2.authentication.kerberos.keytab</name>

        <value>/var/kerberos/krb5kdc/keytab/hive.keytab</value>

        <!-- value>/xxx/soft/apache-hive-2.3.0-bin/conf/keytab/hive.keytab</value -->

    </property>

    <property>

        <name>hive.metastore.sasl.enabled</name>

        <value>true</value>

    </property>

    <property>

        <name>hive.metastore.kerberos.keytab.file</name>

        <value>/var/kerberos/krb5kdc/keytab/hive.keytab</value>

    </property>

    <property>

        <name>hive.metastore.kerberos.principal</name>

        <value>hive/_HOST@JENKIN.COM</value>

    </property>

</configuration>

　　

启动hive：

初始化数据：

./bin/schematool -dbType mysql -initSchema

获取票据：

kinit -k -t /var/kerberos/krb5kdc/keytab/hive.keytab hive/hadoop1@JENKIN.COM

启动server：

hive --service metastore &　　

验证：

[root@hadoop1 conf]# netstat -nl | grep 9083

tcp        0      0 0.0.0.0:9083                0.0.0.0:*                   LISTEN

　　

ps -ef | grep metastore

hive

hive>

启动thrift （hive server）

hive --service hiveserver2 &

　

验证thrift（hive server是否启动）　

[root@hadoop1 conf]# netstat -nl | grep 10000

tcp        0      0 0.0.0.0:10000               0.0.0.0:*                   LISTEN

　　

hive客户端hql操作：

DDL参考：https://cwiki.apache.org//confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-Create/Drop/Alter/UseDatabase

DML参考：https://cwiki.apache.org//confluence/display/Hive/LanguageManual+DML

通过hive建的database，tables， 在hdfs 上都能看到。参考hive-site.xml location配置。

hadoop fs -ls /usr/hive/warehouse

　　

beeline客户端连接hive:

beeline -u "jdbc:hive2://hadoop1:10000/;principal=hive/_HOST@JENKIN.COM"

执行sql:

0: jdbc:hive2://hadoop1:10000/> show databases;

+----------------+

| database_name  |

+----------------+

| default        |

| hivetest       |

+----------------+

2 rows selected (0.318 seconds)

　　

hive> create database jenkintest;

OK

Time taken: 0.968 seconds

hive> show databases;

OK

default

hivetest

jenkintest

Time taken: 0.033 seconds, Fetched: 3 row(s)

hive> use jenkintest

    > ;

OK

Time taken: 0.108 seconds

hive> create table test1(columna int, columnb string);

OK

Time taken: 0.646 seconds

hive> show tables;

OK

test1

Time taken: 0.084 seconds, Fetched: 1 row(s)

　　

　hive数据导入：（通过文件导入，在本地建立文件，列按“table”键分开）　

[root@hadoop2 ~]# vim jenkindb.txt

1       jenkin

2       jenkin.k

3       anne

[root@hadoop2 ~]#hive

hive> create table jenkintb (id int, name string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' STORED AS TEXTFILE;

hive> load data local inpath 'jenkindb.txt' into table jenkintb;

hive> select * from jenkintb;

OK

1       jenkin

2       jenkin.k

3       anne

　　

show create table jenkintb;

　　

　　

最新文章

  1. Could not load file or assembly &#39;Microsoft.ReportViewer.WebForms, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a&#39; or one of its de
  2. 【转载】利用Unity自带的合图切割功能将合图切割成子图
  3. sqlserver前面加N解释
  4. 《A First Course in Probability》-chaper2-概率论公理
  5. UVALive 5102 Fermat Point in Quadrangle 极角排序+找距离二维坐标4个点近期的点
  6. NET Core全新的配置管理
  7. sqlserver2008用bat脚本备份时报错因为库名有中横杠【原创】
  8. Linux内存机制以及手动释放swap和内存
  9. NOIP2013花匠(波动序列)
  10. android开发之Tabhost刷新
  11. UGUI 判断元素进入舞台
  12. How to add hyperlink in JLabel
  13. 每天一个linux命令-wc命令
  14. Visual Categorization with Bags of Keypoints
  15. INSERT INTO 语句用于向表格中插入新的行。
  16. qml 关于鼠标穿透的问题
  17. 修改Eclipse中项目在Apache Tomcat中的部署路径
  18. Python简单的制作图片验证码
  19. 七. 多线程编程6.isAlive()和join()的使用
  20. ubuntu openfire Server install

热门文章

  1. 那个你经常用的abs函数（取绝对值）真的总是返回非负数吗？
  2. 【BZOJ3626】[LNOI2014]LCA 离线+树链剖分+线段树
  3. ORACLE WITH AS 用法,创建临时表
  4. 解决：R读取含中文excel文件，read.xlsx乱码问题
  5. Tomcat-----&gt;tomcat配置虚拟主机（搭建网站）mac
  6. lsof,fuser,xargs,print0,cut,paste,cat,tac,rev,exec,{},双引号，单引号，‘（字符串中执行命令）
  7. python基础-第五篇-5.3装饰器
  8. 1.1 - python基础语法 - 总结练习题
  9. kafka-stream数据清洗
  10. 国产服务器离线安装gm