IDEA创建本地Spark程序，并本地运行

1 IDEA创建maven项目进行测试

v创建一个新项目，步骤如下：

选择“Enable Auto-Import”，加载完后：选择“Enable Auto-Import”，加载完后：

添加SDK依赖：

点击OK ok

可以看到scala包加载成功

再修改pox.xml文件

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">

  <modelVersion>4.0.</modelVersion>

  <groupId>com.wangle</groupId>

  <artifactId>LocalSpark</artifactId>

  <version>1.0-SNAPSHOT</version>

    <properties>

        <spark.version>2.1.</spark.version>

        <scala.version>2.11</scala.version>

    </properties>

    <dependencies>

        <dependency>

            <groupId>org.apache.spark</groupId>

            <artifactId>spark-core_${scala.version}</artifactId>

            <version>${spark.version}</version>

        </dependency>

        <dependency>

            <groupId>org.apache.spark</groupId>

            <artifactId>spark-streaming_${scala.version}</artifactId>

            <version>${spark.version}</version>

        </dependency>

        <dependency>

            <groupId>org.apache.spark</groupId>

            <artifactId>spark-sql_${scala.version}</artifactId>

            <version>${spark.version}</version>

        </dependency>

        <dependency>

            <groupId>org.apache.spark</groupId>

            <artifactId>spark-hive_${scala.version}</artifactId>

            <version>${spark.version}</version>

        </dependency>

        <dependency>

            <groupId>org.apache.spark</groupId>

            <artifactId>spark-mllib_${scala.version}</artifactId>

            <version>${spark.version}</version>

        </dependency>

    </dependencies>

    <build>

        <plugins>

            <plugin>

                <groupId>org.scala-tools</groupId>

                <artifactId>maven-scala-plugin</artifactId>

                <version>2.15.</version>

                <executions>

                    <execution>

                        <goals>

                            <goal>compile</goal>

                            <goal>testCompile</goal>

                        </goals>

                    </execution>

                </executions>

            </plugin>

            <plugin>

                <artifactId>maven-compiler-plugin</artifactId>

                <version>3.6.</version>

                <configuration>

                    <source>1.8</source>

                    <target>1.8</target>

                </configuration>

            </plugin>

            <plugin>

                <groupId>org.apache.maven.plugins</groupId>

                <artifactId>maven-surefire-plugin</artifactId>

                <version>2.19</version>

                <configuration>

                    <skip>true</skip>

                </configuration>

            </plugin>

        </plugins>

    </build>

</project>

新建“Demo1.scala”，测试

巴特西

IDEA创建本地Spark程序，并本地运行

1 IDEA创建maven项目进行测试

最新文章

热门文章