Dream------hive on spark

一、Hive on Spark是Hive跑在Spark上，用的是Spark执行引擎，而不是MapReduce，和Hive on Tez的道理一样。

并且用的是$HIVE_HOME/bin/hive，liunx命令运行客户端

这个时候需要下载spark的源码并且要重新编译，一个不支持hive的版本。

步骤：
1、下载spark1.4.1的源码
https://github.com/apache/spark/tree/v1.4.1

并解压

2、使用编译命令：
./make-distribution.sh --name "hadoop-2.6.0" --tgz "-Dyarn.version=2.6.0 -Dhadoop.version=2.6.0 -Pyarn"

3、配置spark-env.sh文件

export JAVA_HOME=/usr/local/soft/jdk1.7.0

#export SPARK_MASTER_IP=hadoop-spark01

export SPARK_MASTER_WEBUI_PORT=8099

#export SPARK_MASTER_IP=localhost

export SPARK_MASTER_PORT=7077

export SPARK_WORKER_CORES=2

export SPARK_WORKER_INSTANCES=2

export SPARK_WORKER_MEMORY=1g

#export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=FILESYSTEM -Dspark.deploy.recoveryDirectory=/nfs/spark/recovery"

export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=hadoop-spark01:2181,hadoop-spark02:2181,hadoop-spark03:2181 -Dspark.deploy.zookeeper.dir=/home/data/spark/zkdir" （这是spark的HA配置）

export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop

export HIVE_CONF_DIR=$HIVE_HOME/conf

export SPARK_HOME=/usr/local/soft/spark-1.4.1-bin-hadoop-2.6.0

export SPARK_CLASSPATH=/usr/local/soft/sparkclasspath/mysql-connector-java-5.1.38-bin.jar:/usr/local/soft/sparkclasspath/hiv

e-hbase-handler-1.2.1.jar:/usr/local/soft/sparkclasspath/hbase-common-1.1.2.jar:/usr/local/soft/sparkclasspath/hbase-client-1.1.2.jar:/usr/local/soft/sparkclasspath/hbase-protocol-1.1.2.jar:/usr/local/soft/sparkclasspath/hbase-server-1.1.2.jar:/usr/local/soft/sparkclasspath/protobuf-java-2.5.0.jar:/usr/local/soft/sparkclasspath/htrace-core-3.1.0-incubating.jar:/usr/local/soft/sparkclasspath/guava-12.0.1.jar:/usr/local/soft/sparkclasspath/hive-exec-1.2.1.jar

#export SPARK_LIBRARY_PATH=/usr/local/soft/hbase-1.1.2/lib

export SPARK_JAR=/usr/local/soft/spark-1.4.1-bin-hadoop-2.6.0/lib/spark-assembly-1.4.1-hadoop2.6.0.jar

export PATH=$SPARK_HOME/bin:$PATH

4、将spark-assembly-1.4.1-hadoop2.6.0.jar包，拷贝到$HIVE_HOME/lib目录下

5、修改hive-site.xml

<property>

<name>hive.metastore.uris</name>

<value>thrift://hadoop-spark01:9083</value>

<description>Thrift URI forthe remote metastore. Used by metastore client to connect to remote metastore.</description>

</property>

<property>

<name>hive.server2.thrift.min.worker.threads</name>

<value>5</value>

<description>Minimum number of Thrift worker threads</description>

</property>

<property>

<name>hive.server2.thrift.max.worker.threads</name>

<value>500</value>

<description>Maximum number of Thrift worker threads</description>

</property>

<property>

<name>hive.server2.thrift.port</name>

<value>10000</value>

<description>Port number of HiveServer2 Thrift interface. Can be overridden by setting $HIVE_SERVER2_THRIFT_PORT</description>

</property>

<property>

<name>hive.server2.thrift.bind.host</name>

<value>hadoop-spark01</value>

<description>Bind host on which to run the HiveServer2 Thrift interface.Can be overridden by setting$HIVE_SERVER2_THRIFT_BIND_HOST</description>

</property>

<name>spark.serializer</name>

<value>org.apache.spark.serializer.KryoSerializer</value>

</property>

<name>spark.eventLog.enabled</name>

</property>

<name>spark.eventLog.dir</name>

<value>hdfs://founder/sparklog/logs</value>

</property>

<name>spark.master</name>

<value>spark://hadoop-spark01:7077,hadoop-spark02:7077</value>

</property>

还有这些参数也要配置上

1、hive.exec.local.scratchdir

/opt/hive-1.2/tmp

2、hive.downloaded.resources.dir

/opt/hive-1.2/resources

配置Mysql数据库

1、javax.jdo.option.ConnectionPassword

123456

2、javax.jdo.option.ConnectionURL

jdbc:mysql://hadoop-spark01:3306/hive_db

3、javax.jdo.option.ConnectionDriverName

com.mysql.jdbc.Driver

4、javax.jdo.option.ConnectionUserName

root

6、启动

启动spark

./start-all.sh

在backup-master节点上

./start-master

启动hive

./hive

------------------------------------------------------------------------------------------------------------------------------------------------------

二、使用beeline连接，这个比较使用，因为可以使用jdbc让客户端连接

首先特么的这个是不用重新编译spark的源码的，他需要支持hive

1、启动spark

2、启动thriftserver

cd $SPARK_HOME/sbin

./start-thriftserver.sh --master spark://hadoop-spark01:7077 --executor-memory 1g

3、启动hive metastore

hive --service metastore > metastore.log 2>&1 &

使用beeline连接

[root@hadoop-spark01 logs]# beeline

beeline> !connect jdbc:hive2://hadoop-spark01:10000

0: jdbc:hive2://hadoop-spark01:10000> select count(*) from t_trackinfo;

+------+--+

| _c0 |

+------+--+

| 188 |

+------+--+

1 row selected (16.738 seconds)

需要注意的几点：

1、我的hive中的数据是从hbase同步过来的。

2、不需要从新编译hive源码。直接从apache官网上下载就可以了。

3、一般使用的都是thriftserver2这种方式，通过客户端程序通过jdbc操作hive。所以不用编译源码，做好相应的配置就可以了。

这些配置已经过时，并且写在spark-defaults.conf文件里面，就可以了

SPARK_CLASSPATH was detected (set to '/usr/local/soft/sparkclasspath/mysql-connector-java-5.1.38-bin.jar:/usr/local/soft/sparkcla

sspath/hive-hbase-handler-1.2.1.jar:/usr/local/soft/sparkclasspath/hbase-common-1.1.2.jar:/usr/local/soft/sparkclasspath/hbase-client-1.1.2.jar:/usr/local/soft/sparkclasspath/hbase-protocol-1.1.2.jar:/usr/local/soft/sparkclasspath/hbase-server-1.1.2.jar:/usr/local/soft/sparkclasspath/protobuf-java-2.5.0.jar:/usr/local/soft/sparkclasspath/htrace-core-3.1.0-incubating.jar:/usr/local/soft/sparkclasspath/guava-12.0.1.jar:/usr/local/soft/sparkclasspath/hive-exec-1.2.1.jar').This is deprecated in Spark 1.0+.

Please instead use:

- ./spark-submit with --driver-class-path to augment the driver classpath

- spark.executor.extraClassPath to augment the executor classpath

SPARK_WORKER_INSTANCES was detected (set to '2').

This is deprecated in Spark 1.0+.

Please instead use:

- ./spark-submit with --num-executors to specify the number of executors

- Or set SPARK_EXECUTOR_INSTANCES

- spark.executor.instances to configure the number of instances in the spark config.

巴特西

Dream------hive on spark

最新文章

热门文章