Spark教程——(7)编写spark-sql程序读取HBase定时生成报表
2024-09-04 10:52:04
plugin划红线报错:
maven-scala-plugin maven-shade-plugin
查找Maven仓库,发现一个没有jar包,一个jar包无法解压缩打开,删除Maven中坏的jar包,并Reimport成功,IDEA不再报错:
在IDEA的Maven面板中,设置跳过test,为打包做准备:
打包Spark程序:
打包程序报错:
Failed to execute goal org.scala-tools:maven-scala-plugin:2.15.2:compile wrap: org.apache.commons.exec.ExecuteException error: scala.reflect.internal.MissingRequirementError: object scala.runtime in compiler mirror not found. Re-run Maven using the -X switch to enable full debug logging.
参考相关博客,推断Maven库中jar包有问题,但是报错不具体,无法定位有问题的jar包,设置Maven输出更详细的信息,将Output level由Info调成Debug:
先Clean,后执行Compile:
返回错误信息,经过滤如下:
[FATAL] Non-parseable POM D:\Development\MavenRepository\org\apache\hadoop\hadoop-mapreduce-client-core\2.6.0-cdh5.14.2\hadoop-mapreduce-client-core-2.6.0-cdh5.14.2.pom: end tag name </head> must be the same as start tag <link> from line 21 (position: TEXT seen ...<![endif]-->\r\n</head>... @66:8) @ line 66, column 8 [FATAL] Non-parseable POM D:\Development\MavenRepository\org\apache\phoenix\phoenix-core\4.14.0-cdh5.14.2\phoenix-core-4.14.0-cdh5.14.2.pom: end tag name </head> must be the same as start tag <link> from line 21 (position: TEXT seen ...<![endif]-->\r\n</head>... @66:8) @ line 66, column 8 [FATAL] Non-parseable POM D:\Development\MavenRepository\com\lmax\disruptor\3.3.8\disruptor-3.3.8.pom: end tag name </head> must be the same as start tag <link> from line 21 (position: TEXT seen ...<![endif]-->\r\n</head>... @66:8) @ line 66, column 8 ……
到对应目录下查看jar包是否正常,解压打开报错,说明对应jar包确实有问题,删除Maven中坏的jar包,执行Reimport重新导入jar包:
虽然IDEA依然有红线报错,但是画红线的jar包已经导入并能解压缩打开,尝试执行Compile,Compile成功,执行Package,Package成功,返回如下信息:
[INFO] Replacing original artifact with shaded artifact. [INFO] Replacing D:\Development\asset\statistics-master-190725\target\statistics-1.0-SNAPSHOT.jar with D:\Development\asset\statistics-master-190725\target\statistics-1.0-SNAPSHOT-shaded.jar [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 02:49 min [INFO] Finished at: 2019-07-25T16:41:04+08:00 [INFO] ------------------------------------------------------------------------
上传到服务器:
执行打包的Spark程序:
[root@node2 ~]# spark-submit --master yarn-cluster --driver-memory 4g --num-executors --executor-memory 2g --executor-cores --class statistics.CostAsset --conf spark.driver.extraClassPath=/opt/cloudera/parcels/APACHE_PHOENIX--cdh5./lib/phoenix/lib/* --conf spark.executor.extraClassPath=/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/* /home/microservices/statistics-1.0-SNAPSHOT.jar total model
执行后返回如下信息:
[root@node2 ~]# spark-submit --master yarn-cluster --driver-memory 4g --num-executors --executor-memory 2g --executor-cores --class statistics.CostAsset --conf spark.driver.extraClassPath=/opt/cloudera/parcels/APACHE_PHOENIX--cdh5./lib/phoenix/lib/* --conf spark.executor.extraClassPath=/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/* /home/microservices/statistics-1.0-SNAPSHOT.jar total model 19/07/25 16:54:52 INFO client.RMProxy: Connecting to ResourceManager at node1/10.200.101.131:8032 19/07/25 16:54:52 INFO yarn.Client: Requesting a new application from cluster with 3 NodeManagers 19/07/25 16:54:52 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (40874 MB per container) 19/07/25 16:54:52 INFO yarn.Client: Will allocate AM container, with 4505 MB memory including 409 MB overhead 19/07/25 16:54:52 INFO yarn.Client: Setting up container launch context for our AM 19/07/25 16:54:52 INFO yarn.Client: Setting up the launch environment for our AM container 19/07/25 16:54:52 INFO yarn.Client: Preparing resources for our AM container 19/07/25 16:54:53 INFO yarn.Client: Uploading resource file:/home/microservices/statistics-1.0-SNAPSHOT.jar -> hdfs://node1:8020/user/root/.sparkStaging/application_1563417834812_0018/statistics-1.0-SNAPSHOT.jar 19/07/25 16:54:54 INFO yarn.Client: Uploading resource file:/tmp/spark-2573c8b3-f471-452f-85b1-d3582877290e/__spark_conf__4623511860207833838.zip -> hdfs://node1:8020/user/root/.sparkStaging/application_1563417834812_0018/__spark_conf__4623511860207833838.zip 19/07/25 16:54:54 INFO spark.SecurityManager: Changing view acls to: root 19/07/25 16:54:54 INFO spark.SecurityManager: Changing modify acls to: root 19/07/25 16:54:54 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root) 19/07/25 16:54:54 INFO yarn.Client: Submitting application 18 to ResourceManager 19/07/25 16:54:54 INFO impl.YarnClientImpl: Submitted application application_1563417834812_0018 19/07/25 16:54:55 INFO yarn.Client: Application report for application_1563417834812_0018 (state: ACCEPTED) 19/07/25 16:54:55 INFO yarn.Client: client token: N/A diagnostics: N/A ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: root.users.root start time: 1564044894496 final status: UNDEFINED tracking URL: http://node1:8088/proxy/application_1563417834812_0018/ user: root 19/07/25 16:54:56 INFO yarn.Client: Application report for application_1563417834812_0018 (state: ACCEPTED) 19/07/25 16:54:57 INFO yarn.Client: Application report for application_1563417834812_0018 (state: ACCEPTED) 19/07/25 16:54:58 INFO yarn.Client: Application report for application_1563417834812_0018 (state: ACCEPTED) 19/07/25 16:54:59 INFO yarn.Client: Application report for application_1563417834812_0018 (state: ACCEPTED) 19/07/25 16:55:00 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING) 19/07/25 16:55:00 INFO yarn.Client: client token: N/A diagnostics: N/A ApplicationMaster host: 10.200.101.133 ApplicationMaster RPC port: 0 queue: root.users.root start time: 1564044894496 final status: UNDEFINED tracking URL: http://node1:8088/proxy/application_1563417834812_0018/ user: root 19/07/25 16:55:01 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING) 19/07/25 16:55:02 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING) 19/07/25 16:55:03 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING) 19/07/25 16:55:04 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING) 19/07/25 16:55:05 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING) 19/07/25 16:55:06 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING) 19/07/25 16:55:07 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING) 19/07/25 16:55:08 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING) 19/07/25 16:55:09 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING) 19/07/25 16:55:10 INFO yarn.Client: Application report for application_1563417834812_0018 (state: ACCEPTED) 19/07/25 16:55:10 INFO yarn.Client: client token: N/A diagnostics: N/A ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: root.users.root start time: 1564044894496 final status: UNDEFINED tracking URL: http://node1:8088/proxy/application_1563417834812_0018/ user: root 19/07/25 16:55:11 INFO yarn.Client: Application report for application_1563417834812_0018 (state: ACCEPTED) 19/07/25 16:55:12 INFO yarn.Client: Application report for application_1563417834812_0018 (state: ACCEPTED) 19/07/25 16:55:13 INFO yarn.Client: Application report for application_1563417834812_0018 (state: ACCEPTED) 19/07/25 16:55:14 INFO yarn.Client: Application report for application_1563417834812_0018 (state: ACCEPTED) 19/07/25 16:55:15 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING) 19/07/25 16:55:15 INFO yarn.Client: client token: N/A diagnostics: N/A ApplicationMaster host: 10.200.101.135 ApplicationMaster RPC port: 0 queue: root.users.root start time: 1564044894496 final status: UNDEFINED tracking URL: http://node1:8088/proxy/application_1563417834812_0018/ user: root 19/07/25 16:55:16 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING) 19/07/25 16:55:17 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING) 19/07/25 16:55:18 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING) 19/07/25 16:55:19 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING) 19/07/25 16:55:20 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING) 19/07/25 16:55:21 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING) 19/07/25 16:55:22 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING) 19/07/25 16:55:23 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING) 19/07/25 16:55:24 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING) 19/07/25 16:55:25 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING) 19/07/25 16:55:26 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING) 19/07/25 16:55:27 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING) 19/07/25 16:55:28 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING) 19/07/25 16:55:29 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING) 19/07/25 16:55:30 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING) 19/07/25 16:55:31 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING) 19/07/25 16:55:32 INFO yarn.Client: Application report for application_1563417834812_0018 (state: RUNNING) 19/07/25 16:55:33 INFO yarn.Client: Application report for application_1563417834812_0018 (state: FINISHED) 19/07/25 16:55:33 INFO yarn.Client: client token: N/A diagnostics: N/A ApplicationMaster host: 10.200.101.135 ApplicationMaster RPC port: 0 queue: root.users.root start time: 1564044894496 final status: SUCCEEDED tracking URL: http://node1:8088/proxy/application_1563417834812_0018/ user: root 19/07/25 16:55:33 INFO util.ShutdownHookManager: Shutdown hook called 19/07/25 16:55:33 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-2573c8b3-f471-452f-85b1-d3582877290e
通过界面分析执行的程序:
参考:
https://www.cnblogs.com/nurseryboy/p/6155925.html
https://www.oschina.net/question/1422726_2263380?sort=time
最新文章
- iOS 保存、读取与应用状态
- mahout 安装测试
- 月薪3万的程序员告诉你:这样工作才能拿高薪(转 IT之家)
- 如何安装Oracle Instant Client
- event 内存泄漏
- 使用Codis搭建redis集群服务
- Gamit解算脚本
- RMAN备份FORMAT格式中%的含义
- C#Http编程
- JAVASCRIPT 调用 其他应用程序的方法
- 操作系统中的几种调度算法(JAVA版)
- 十八、泛型 l 注解 l Servlet3.0 l 动态代理 l 类加载器基础加强
- WCF 基础框架
- [Hive_add_9] Hive 的存储格式
- flutter控件之ExpansionPanelList
- 如何写django中form的测试用例
- mybatis generator(MyBatis的逆向工程)
- MongoDB 记录
- 1.svn+apache整合
- C++中去掉string字符串中的\r\n等