[jun@master ~]$ cd scala-2.12./

[jun@master scala-2.12.]$ ls -l

total

drwxrwxr-x.  jun jun  Apr  : bin

drwxrwxr-x.  jun jun   Apr  : doc

drwxrwxr-x.  jun jun  Apr  : lib

drwxrwxr-x.  jun jun   Apr  : man

[jun@master scala-2.12.]$ bin/scala

Welcome to Scala 2.12. (Java HotSpot(TM) -Bit Server VM, Java 1.8.0_171).

Type in expressions for evaluation. Or try :help.

scala> println("hello,world")

hello,world

scala> *

res1: Int = 

scala> *res1

res2: Int = 

scala> :help

All commands can be abbreviated, e.g., :he instead of :help.

:completions <string>    output completions for the given string

:edit <id>|<line>        edit history

:help [command]          print this summary or command-specific help

:history [num]           show the history (optional num is commands to show)

:h? <string>             search the history

:imports [name name ...] show import history, identifying sources of names

:implicits [-v]          show the implicits in scope

:javap <path|class>      disassemble a file or class name

:line <id>|<line>        place line(s) at the end of history

:load <path>             interpret lines in a file

:paste [-raw] [path]     enter paste mode or paste a file

:power                   enable power user mode

:quit                    exit the interpreter

:replay [options]        reset the repl and replay all previous commands

:require <path>          add a jar to the classpath

:reset [options]         reset the repl to its initial state, forgetting all session entries

:save <path>             save replayable session to a file

:sh <command line>       run a shell command (result is implicitly => List[String])

:settings <options>      update compiler options, if possible; see reset

:silent                  disable/enable automatic printing of results

:type [-v] <expr>        display the type of an expression without evaluating it

:kind [-v] <type>        display the kind of a type. see also :help kind

:warnings                show the suppressed warnings from the most recent line which had any

scala> :quit

[jun@master ~]$ cd spark-2.3.-bin-hadoop2./

[jun@master spark-2.3.-bin-hadoop2.]$ ls -l

total

drwxrwxr-x.  jun jun   Jun   : bin

drwxrwxr-x.  jun jun    Jun   : conf

drwxrwxr-x.  jun jun     Jun   : data

drwxrwxr-x.  jun jun     Jun   : examples

drwxrwxr-x.  jun jun  Jun   : jars

drwxrwxr-x.  jun jun     Jun   : kubernetes

-rw-rw-r--.  jun jun  Jun   : LICENSE

drwxrwxr-x.  jun jun   Jun   : licenses

-rw-rw-r--.  jun jun  Jun   : NOTICE

drwxrwxr-x.  jun jun    Jun   : python

drwxrwxr-x.  jun jun     Jun   : R

-rw-rw-r--.  jun jun   Jun   : README.md

-rw-rw-r--.  jun jun    Jun   : RELEASE

drwxrwxr-x.  jun jun   Jun   : sbin

drwxrwxr-x.  jun jun     Jun   : yarn

#spark

export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

export HDFS_CONF_DIR=$HADOOP_HOME/etc/hadoop

export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop

[jun@master conf]$ cp ~/spark-2.3.-bin-hadoop2./conf/spark-env.sh.template   ~/spark-2.3.-bin-hadoop2./conf/spark-env.sh

[jun@master conf]$ gedit ~/spark-2.3.-bin-hadoop2./conf/spark-env.sh

export SPARK_MASTER_IP=192.168.1.100

export JAVA_HOME=/usr/java/jdk1..0_171/

export SCALA_HOME=/home/jun/scala-2.12./

[jun@master conf]$ cp ~/spark-2.3.-bin-hadoop2./conf/slaves.template slaves

[jun@master conf]$ gedit ~/spark-2.3.-bin-hadoop2./conf/slaves

# A Spark Worker will be started on each of the machines listed below.

slave0

slave1

[jun@master conf]$ /home/jun/spark-2.3.-bin-hadoop2./sbin/start-all.sh

starting org.apache.spark.deploy.master.Master, logging to /home/jun/spark-2.3.-bin-hadoop2./logs/spark-jun-org.apache.spark.deploy.master.Master--master.out

slave0: starting org.apache.spark.deploy.worker.Worker, logging to /home/jun/spark-2.3.-bin-hadoop2./logs/spark-jun-org.apache.spark.deploy.worker.Worker--slave0.out

slave1: starting org.apache.spark.deploy.worker.Worker, logging to /home/jun/spark-2.3.-bin-hadoop2./logs/spark-jun-org.apache.spark.deploy.worker.Worker--slave1.out

[jun@master conf]$ jps

 ResourceManager

 SecondaryNameNode

 Master

 NameNode

 Jps

[jun@slave0 ~]$ jps

 DataNode

 Worker

 NodeManager

 Jps

[jun@slave1 ~]$ jps

 DataNode

 Worker

 NodeManager

 Jps

[jun@master conf]$ cp /home/jun/spark-2.3.-bin-hadoop2./examples/jars/spark-examples_2.-2.3..jar /home/jun/spark-2.3.-bin-hadoop2./

[jun@master spark-2.3.-bin-hadoop2.]$ bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster --num-executors  --driver-memory 512m --executor-memory 512m --executor-cores  spark-examples_2.-2.3..jar 

diagnostics: Application application_1532350446978_0001 failed  times due to AM Container for appattempt_1532350446978_0001_000002 exited with  exitCode: -

Failing this attempt.Diagnostics: Container [pid=,containerID=container_1532350446978_0001_02_000001] is running beyond virtual memory limits. Current usage: 289.7 MB of  GB physical memory used; 2.3 GB of 2.1 GB virtual memory used. Killing container.

    <property>

        <name>yarn.nodemanager.vmem-pmem-ratio</name>

        <value>2.5</value>

    </property>

-- :: INFO  Client: -

     client token: N/A

     diagnostics: N/A

     ApplicationMaster host: 192.168.1.101

     ApplicationMaster RPC port:

     queue: default

     start time:

     final status: SUCCEEDED

     tracking URL: http://master:18088/proxy/application_1532352327714_0001/

     user: jun

-- :: INFO  ShutdownHookManager: - Shutdown hook called

-- :: INFO  ShutdownHookManager: - Deleting directory /tmp/spark-1ed5bee9-1aa7--b3ec-80ff2b153192

-- :: INFO  ShutdownHookManager: - Deleting directory /tmp/spark-7349a4e3--4d09-91ff-e1e48cb59b46