Hive进行数据统计时报错:org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
2024-10-19 19:24:41
报错详情:
2020-04-09 22:56:58,827 ERROR [Listener at 0.0.0.0/45871] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.NullPointerException
at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:178)
at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:122)
at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:280)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.serviceStart(MRAppMaster.java:979)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1293)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$6.run(MRAppMaster.java:1761)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1757)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1691)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.mapreduce.v2.app.client.MRClientService.getHttpPort(MRClientService.java:177)
at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:159)
... 14 more
2020-04-09 22:56:58,818 INFO [TaskHeartbeatHandler PingChecker] org.apache.hadoop.mapreduce.v2.app.TaskHeartbeatHandler: TaskHeartbeatHandler thread interrupted
2020-04-09 22:56:58,819 INFO [Ping Checker] org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: TaskAttemptFinishingMonitor thread interrupted
阅读源代码,会发现这是由于org.apache.hadoop.yarn.server.webproxy.amfilter.AmFilterInitializer.java这个类中的一段代码引起的,涉及的代码如下:
if (rmIds != null) {
List<String> urls = new ArrayList<>();
for (String rmId : rmIds) {
String url = getUrlByRmId(yarnConf, rmId);
// urls.add(url); // 注释掉这端代码,修改为下面的if语句判断是否为null
if (url != null) {
urls.add(url);
}
}
if (!urls.isEmpty()) {
params.put(RM_HA_URLS, StringUtils.join(",", urls));
}
}
这与yarn-site.xml配置HA的兼容性有关,取决于 yarn.resourcemanager.webapp.address 和 yarn.resourcemanager.webapp.https.address 是否为空。
修改源码的方法:
1、首先将hadoop-yarn-server-web-proxy-3.2.1.jar从服务器上拿下来。
2、使用JD-GUI打开,复制org.apache.hadoop.yarn.server.webproxy.amfilter.AmFilterInitializer.java类中代码;
3、在eclipse新建一个工程,然后新建一个类插入复制的代码,并进行修改。
4、有报错的话就build class,把本地的hadoop相关的包导入
5、将该工程导出为jar包。
6、分别打开hadoop-yarn-server-web-proxy-3.2.1.jar 和刚导出的jar包用解压软件打开,替换原来的org.apache.hadoop.yarn.server.webproxy.amfilter.AmFilterInitializer.class文件。
修改完后替换原来的jar包,重启hdfs和yarn.job运行成功!
最新文章
- MYSQL数据库如何赋予远程某个IP访问权限
- Yocto开发笔记之《Makefile编写》(QQ交流群:519230208)
- 解决Kali Linux没有声音
- Android 学习笔记之AndBase框架学习(一) 实现多功能标题栏
- EFsql笔记
- 【Slickflow学习】.NET开源工作流项目转换(二)
- PHP的魔术方法(简介)
- nRF51822 SDK初体验
- Codeforces 437C The Child and Toy(贪心)
- Apache 关于 mod_rewrite 遇到 %2F或%5C (正反斜杠)等特殊符号导致URL重写失效出现404的问题
- Java NIO之Buffers
- ASP.NET Core WebApi 返回统一格式参数
- jq 获取下一个兄弟原素 下拉箭头旋转
- P2370 yyy2015c01的U盘(二分+背包)
- 小学生四则运算出题软件-基于java控制台的实现
- LNMP环境包安装IonCube教程
- [Error] &#39;exit&#39; was not declared in this scope的解决方法
- SEO从理论到实践
- UITableViewCell 取消分隔线
- Oracle分组取第一条数据