1、【Spark】Spark安装

本文基于的环境

Red       Hat Linux Enterprise 7 x86_64

jdk       1.7.0_79

Python    2.7
Spart     spark-1.5.2-bin-hadoop2.6

官方要求的环境如下：

Java 7+

Python 2.6+

R 3.1+.

1、下载Spark(spark-1.5.2-bin-hadoop2.6.tgz)安装包

wget http://mirrors.hust.edu.cn/apache/spark/spark-1.5.2/spark-1.5.2-bin-hadoop2.6.tgz

2、将安装包解压缩

tar –xzvf spark-1.5.2-bin-hadoop2.6.tgz

3、启动spark

./bin/spark-shell

./bin/pyspark

看到这个界面，证明Spark已经启动成功了

Welcome to

      ____              __

     / __/__  ___ _____/ /__

    _\ \/ _ \/ _ `/ __/  '_/

   /__ / .__/\_,_/_/ /_/\_\   version 1.5.2

      /_/

Using Python version 2.7.5 (default, Feb 11 2014 07:46:25)

SparkContext available as sc, HiveContext available as sqlContext.

Q&A:

1、Q:Spark启动的时候报这个错

java.net.BindException: Failed to bind to: /192.168.3.15:0: Service 'sparkDriver' failed after 16 retries!

    at org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272)

    at akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:393)

    at akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:389)

    at scala.util.Success$$anonfun$map$1.apply(Try.scala:206)

    at scala.util.Try$.apply(Try.scala:161)

    at scala.util.Success.map(Try.scala:206)

    at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235)

    at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235)

    at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)

    at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)

    at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)

    at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)

    at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)

    at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)

    at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)

    at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)

    at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)

    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)

    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)

    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)

    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

A:这是由于在没没有配置下面两个变量：

#SPART_HOME/bin/load-spark-env.sh

export SPARK_MASTER_IP=127.0.0.1

export SPARK_LOCAL_IP=127.0.0.1

配置之后，一个单机版的环境就搭建成功了

巴特西

1、【Spark】Spark安装

最新文章

热门文章