生产中遇到的spark任务问题
spark版本 2.2.0
日志里面的信息:
WARN RowBasedKeyValueBatch: Calling spill() on RowBasedKeyValueBatch. Will not spill but return 0.
What could be the reason for this warning? Is this something I should care about or can I safely ignore it?
回答1
As indicated here this warning means that your RAM is full and that part of the RAM contents are moved to disk.
See also the Spark FAQ
Does my data need to fit in memory to use Spark?
No. Spark's operators spill data to disk if it does not fit in memory, allowing it to run well on any sized data. Likewise, cached datasets that do not fit in memory are either spilled to disk or recomputed on the fly when needed, as determined by the RDD's storage level.
回答2
I guess this message is worse than a simple warning : it is on the edge of being an error.
Have a look at the source code :
/**
* Sometimes the TaskMemoryManager may call spill() on its associated MemoryConsumers to make
* space for new consumers. For RowBasedKeyValueBatch, we do not actually spill and return 0.
* We should not throw OutOfMemory exception here because other associated consumers might spill
*/
public final long spill(long size, MemoryConsumer trigger) throws IOException {
logger.warn("Calling spill() on RowBasedKeyValueBatch. Will not spill but return 0.");
return 0;
}
here : https://github.com/apache/spark/blob/master/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/RowBasedKeyValueBatch.java
So I would say here you are on an infinite loop of "needing to spill but actually not spilling".
来源:https://stackoverflow.com/questions/46907447/meaning-of-apache-spark-warning-calling-spill-on-rowbasedkeyvaluebatch
引用:https://www.e-learn.cn/topic/3560880
最新文章
- HBase Zookeeper 安装学习
- MyEclipse的项目中把 java EE 5 Libraries 删掉后怎么重新导入
- knockoutjs入门
- 实体写到redis写不进去--误把类当成实体类
- Qt编写自定义控件插件路过的坑及注意事项
- iOS之手势滑动返回功能-b
- Eclipse配置Git
- POJ 1458-Common Subsequence(线性dp/LCS)
- Android 开发笔记 “Sqlite Cursor 使用”
- KindeEditor图片上传插件用法
- C# devExpress GridControl 行中行 子行 多级行
- DNS over TLS到底有多牛?你想知道的都在这儿
- IO流(二)
- Html链接标签:
- Python进阶(一)
- docker部署rabbitMQ
- JAVAFX开发桌面应用
- 按键精灵对APP自动化测试(上)
- scheduling.quartz.CronTriggerBean has interface org.quartz.CronTrigger as super class
- 12.文件系统fs