【异常】Reason: Executor heartbeat timed out after 140927 ms
2024-10-07 03:04:16
1 详细异常
ERROR scheduler.JobScheduler: Error running job streaming job ms.
org.apache.spark.SparkException: Job aborted due to stage failure: Task in stage 0.0 failed times,
most recent failure: Lost task 0.3 in stage 0.0 (TID , , executor ): ExecutorLostFailure (executor exited caused by one of the running tasks) Reason: Executor heartbeat timed out after ms
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$failJobAndIndependentStages(DAGScheduler.scala:)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$.apply(DAGScheduler.scala:)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$.apply(DAGScheduler.scala:)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$.apply(DAGScheduler.scala:)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$.apply(DAGScheduler.scala:)
at scala.Option.foreach(Option.scala:)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:)
at org.apache.spark.util.EventLoop$$anon$.run(EventLoop.scala:)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$.apply(RDD.scala:)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$.apply(RDD.scala:)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:)
at org.apache.spark.rdd.RDD.foreachPartition(RDD.scala:)
at com.wm.bigdata.phoenix.etl.WmPhoniexEtlToHbase$$anonfun$main$.apply(WmPhoniexEtlToHbase.scala:)
at com.wm.bigdata.phoenix.etl.WmPhoniexEtlToHbase$$anonfun$main$.apply(WmPhoniexEtlToHbase.scala:)
at org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$$$anonfun$apply$mcV$sp$.apply(DStream.scala:)
at org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$$$anonfun$apply$mcV$sp$.apply(DStream.scala:)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$$$anonfun$apply$mcV$sp$.apply$mcV$sp(ForEachDStream.scala:)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$$$anonfun$apply$mcV$sp$.apply(ForEachDStream.scala:)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$$$anonfun$apply$mcV$sp$.apply(ForEachDStream.scala:)
at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$.apply$mcV$sp(ForEachDStream.scala:)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$.apply(ForEachDStream.scala:)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$.apply(ForEachDStream.scala:)
at scala.util.Try$.apply(Try.scala:)
at org.apache.spark.streaming.scheduler.Job.run(Job.scala:)
at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$.apply$mcV$sp(JobScheduler.scala:)
at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$.apply(JobScheduler.scala:)
at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$.apply(JobScheduler.scala:)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:)
at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:)
at java.lang.Thread.run(Thread.java:)
2 查询Stack Overflow里面问答
3 解决
提交spark submit任务的时候,加大超时时间设置
--conf spark.network.timeout --conf spark.executor.heartbeatInterval= --conf spark.driver.maxResultSize=4g
最新文章
- sql练习(mysql版)
- Flapper Bird的学习笔记(一)
- ini_set(";display_errors";,";On";);和error_reporting(E_ALL);
- 【Bootstrap】2.作品展示站点
- JDBC常用接口详解
- 模拟 POJ 1573 Robot Motion
- PHP写一段代码,确保多个进程同时写入一个文件成功
- 点击TableView中某行进入下一级界面(Swift)
- VC++编程中为程序加入启动画面功能
- 转: 透过CAT,来看分布式实时监控系统的设计与实现
- PHP生成表格
- POJ3608(旋转卡壳--求两凸包的最近点对距离)
- cocos 水果机,老Tiger虎机流水灯,博彩大转盘效果
- 下载jar包的网站
- Delphi之Exception获得错误信息
- ADO.NET基础学习 一(连接数据库)
- UrlDecode
- vs2008快捷键一览表
- P1279 字串距离
- Android应用开发-Activity