① 将 spark-3.0.0-bin-hadoop3.2.tgz 文件上传到 Linux (cd /opt/module路径下)并解压缩 ② 修改spark-3.0.0-bin-hadoop3.2名称为spark-local mv spark-3.0.0-bin-hadoop3.2/ spark-local ③ 进入spark-local cd spark-local/ ④ 启动local环境 bin/spark-shell ⑤ Web页面访问:master:40
def main(args: Array[String]): Unit = { val conf = new SparkConf() conf.set("spark.master", "local") conf.set("spark.app.name", "spark demo") val sc = new SparkContext(conf); // 读取hdfs数据 val textFileRdd = sc.textFil
def main(args: Array[String]): Unit = { val hConf = HBaseConfiguration.create(); hConf.set("hbase.zookeeper.quorum","m6,m7,m8") val tableName = "t_person" hConf.set(TableInputFormat.INPUT_TABLE, tableName) val hAdmin = new HB
SparkContext可以通过parallelize把一个集合转换为RDD def main(args: Array[String]): Unit = { val conf = new SparkConf(); val list = List(1, 2, 3, 4, 5,6); conf.set("spark.master", "local") conf.set("spark.app.name", "spark demo")
尊重版权,原文:http://blog.csdn.net/macyang/article/details/7100523 - Spark是什么? Spark is a MapReduce-like cluster computing framework designed to support low-latency iterative jobs and interactive use from an interpreter. It is written in Scala, a high-le