Spark学习之第一个程序 WordCount

WordCount程序

求下列文件中使用空格分割之后，单词出现的个数

input.txt

java scala python hello world

java pyfysf upuptop wintp top

sfok sf sf

sf java android sf pyfysf upuptop

pyfysf upuptop java android spark

hello world world hello top scala spark

spark spark sql

创建maven项目

pom.xml

<?xml version="1.0" encoding="UTF-8"?>

<project xmlns="http://maven.apache.org/POM/4.0.0"

         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

    <parent>

        <artifactId>SparkStudy</artifactId>

        <groupId>top.wintp.sparkstudy</groupId>

        <version>1.0-SNAPSHOT</version>

    </parent>

    <modelVersion>4.0.0</modelVersion>

    <artifactId>SparkCore</artifactId>

    <dependencies>

        <dependency>

            <groupId>org.apache.spark</groupId>

            <artifactId>spark-core_2.11</artifactId>

            <version>2.1.1</version>

        </dependency>

    </dependencies>

    <build>

        <finalName>WordCount</finalName>

        <plugins>

            <plugin>

                <groupId>net.alchim31.maven</groupId>

                <artifactId>scala-maven-plugin</artifactId>

                <version>3.2.2</version>

                <executions>

                    <execution>

                        <goals>

                            <goal>compile</goal>

                            <goal>testCompile</goal>

                        </goals>

                    </execution>

                </executions>

            </plugin>

            <plugin>

                <groupId>org.apache.maven.plugins</groupId>

                <artifactId>maven-assembly-plugin</artifactId>

                <version>3.0.0</version>

                <configuration>

                    <archive>

                        <manifest>

                            <mainClass>WordCount(修改)</mainClass>

                        </manifest>

                    </archive>

                    <descriptorRefs>

                        <descriptorRef>jar-with-dependencies</descriptorRef>

                    </descriptorRefs>

                </configuration>

                <executions>

                    <execution>

                        <id>make-assembly</id>

                        <phase>package</phase>

                        <goals>

                            <goal>single</goal>

                        </goals>

                    </execution>

                </executions>

            </plugin>

        </plugins>

    </build>

</project>

WordCount.scala

package top.wintp.sparkstudy.sparkcore

import org.apache.spark.{SparkConf, SparkContext}

/**

  * description:

  * <p>

  * author:  upuptop

  * <p>

  * qq: 337081267

  * <p>

  * CSDN:   http://blog.csdn.net/pyfysf

  * <p>

  * cnblogs:   http://www.cnblogs.com/upuptop

  * <p>

  * blog:   http://wintp.top

  * <p>

  * email:  pyfysf@163.com

  * <p>

  * time: 2019/07/2019/7/1

  * <p>

  */

object WordCount {

  def main(args: Array[String]): Unit = {

    //    创建SparkConf

    // setMaster local/local[n]/local[*] 都是本地运行 可以设置远程服务器的Master的地址URL

    val conf = new SparkConf().setMaster("local[*]").setAppName("WordCount")

    //    创建SparkContext

    val sc = new SparkContext(conf)

    //    根据外部文件创建RDD

    val line = sc.textFile("E:/input/input.txt")

    //    flatmap压平操作

    val words = line.flatMap(_.split(" "))

    //    map 组装键值对

    val k2v = words.map((_, 1))

    //    计算结果

    val result = k2v.reduceByKey(_ + _)

    //    保存结果数据到文件中去

    result.saveAsTextFile("E:/output/" + System.currentTimeMillis())

    //    关闭Context

    sc.stop()

  }

}

输出结果

按照如上配置，不会出现以下问题，如不幸出现下面描述问题，请将scal-SDK放到所有依赖的最后



Exception in thread "main"

 java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)

 Lscala/collection/mutable/ArrayOps;

巴特西

Spark学习之第一个程序 WordCount

WordCount程序

最新文章

热门文章