CentOS上安装spark standalone mode(转载)
原文链接 http://blog.csdn.net/chenxingzhen001/article/details/11072765
参考:
http://spark.incubator.apache.org/docs/latest/
http://spark.incubator.apache.org/docs/latest/spark-standalone.html
http://www.yanjiuyanjiu.com/blog/20130617/
1.安装JDK
2.安装scala 2.9.3
Spark 0.7.2 依赖 Scala 2.9.3, 我们必须要安装Scala 2.9.3.
下载 scala-2.9.3.tgz 并 保存到home目录(已经在sg206上).
$ tar -zxf scala-2.9.3.tgz
$ sudo mv scala-2.9.3 /usr/lib
$ sudo vim /etc/profile
# add the following lines at the end
export SCALA_HOME=/usr/lib/scala-2.9.3
export PATH=$PATH:$SCALA_HOME/bin
# save and exit vim
#make the bash profile take effect immediately
source /etc/profile
# test
$ scala -version
3.building spark
cd /home
tar -zxf spark-0.7.3-sources.gz
cd spark-0.7.3
sbt/sbt package (需要git环境 yum install git)
4.配置文件
spark-env.sh
############
export SCALA_HOME=/usr/lib/scala-2.9.3
export SPARK_MASTER_IP=172.16.48.202
export SPARK_WORKER_MEMORY=10G
#############
slaves
将从节点IP添加至slaves配置文件
5.启动和停止
bin/start-master.sh - Starts a master instance on the machine the script is executed on.
bin/start-slaves.sh - Starts a slave instance on each machine specified in the conf/slaves file.
bin/start-all.sh - Starts both a master and a number of slaves as described above.
bin/stop-master.sh - Stops the master that was started via the bin/start-master.sh script.
bin/stop-slaves.sh - Stops the slave instances that were started via bin/start-slaves.sh.
bin/stop-all.sh - Stops both the master and the slaves as described above.
最新文章
- CI框架,双层弹出框的样式实现
- 二叉树的实现与一些基本操作(C++环境)
- php中mysql与mysqli的区别
- Uber从Postgres切换到MySQL
- HBase学习笔记-高级(一)
- javaWeb 使用jsp标签进行防盗链
- 转载:JMS-ActiveMQ浅析
- hdu5406 CRB and Apple dp+两个LIS
- 使用JS动态创建含有1000行的表格
- bzoj3864: Hero meet devil
- Idea facet
- PHP面试题汇总参考
- tomcat基础应用
- Codeforces Round #404 (Div. 2)
- 001. Java内存中的字符编码
- html模板导出pdf文件
- AVL树的理解及自写AVL树
- 微信小程序cavas画图并保存
- selenium捕捉视频
- dubbo发布和引用服务
热门文章
- MySQL绿色版的安装(mysql-5.6.22-win32.zip)
- js产生随机数函数
- C++ const用法总结
- HTML5服务器推送消息的各种解决办法
- 从jquery源码中看类型判断和数组的一些操作
- 72. Generate Parentheses &;&; Valid Parentheses
- webpack 往右一点之 “你好,初次见面”
- AX7: Install a deployable package
- Android驱动开发之Hello实例
- Spark Streaming源码解读之Driver容错安全性