spark 执行spark-example

1. 找到CDH 安装spark的目录

执行 which spark-shell

/usr/bin/spark-shell

执行 ll /usr/bin/spark-shell

lrwxrwxrwx 1 root root 29 12月 11 14:29 /usr/bin/spark-shell -> /etc/alternatives/spark-shell

执行ll /etc/alternatives/spark-shell

lrwxrwxrwx 1 root root 68 12月 11 14:29 /etc/alternatives/spark-shell -> /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/spark-shell

所有安装目录是 /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373

 cd /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/spark/examples/jars

 ll

 会看到 spark-examples_2.11-2.4.0-cdh6.2.0.jar

执行

spark-submit --class org.apache.spark.examples.SparkPi --master yarn --executor-memory 1G --total-executor-cores 2 ./spark-examples_2.11-2.4.0-cdh6.2.0.jar 100

会看到终端打印日志：

2/01/04 18:34:48 INFO cluster.YarnScheduler: Removed TaskSet 0.0, whose tasks have all completed, from pool

22/01/04 18:34:48 INFO scheduler.DAGScheduler: ResultStage 0 (reduce at SparkPi.scala:38) finished in 8.309 s

22/01/04 18:34:48 INFO scheduler.DAGScheduler: Job 0 finished: reduce at SparkPi.scala:38, took 8.413772 s

Pi is roughly 3.1415547141554714

22/01/04 18:34:48 INFO server.AbstractConnector: Stopped Spark@687a762c{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}

22/01/04 18:34:48 INFO ui.SparkUI: Stopped Spark web UI at http://hadoop2:4040

22/01/04 18:34:48 INFO cluster.YarnClientSchedulerBackend: Interrupting monitor thread

Pi is roughly 3.1415547141554714

巴特西

spark 执行spark-example

1. 找到CDH 安装spark的目录

最新文章

热门文章