应用程序连接hbase报错:java.net.SocketTimeoutException: callTimeout=60000
2024-08-25 11:34:54
背景说明:
今天对生产环境hbase增加了节点,下午的时候一个同事反馈,应用程序后台报错,如下:
Tue Feb 26 17:35:35 CST 2019, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=68451: row 'SYSTEM.CATALOG,TARGETCUST_DATA,99999999999999' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=host-10-191-36-24,16020,1551146724629, seqNum=0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:276)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:210)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
at org.apache.hadoop.hbase.client.ClientSmallReversedScanner.loadCache(ClientSmallReversedScanner.java:212)
at org.apache.hadoop.hbase.client.ClientSmallReversedScanner.next(ClientSmallReversedScanner.java:186)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1275)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1181)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1165)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1122)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:957)
at org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:83)
at org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:506)
at org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:722)
at org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:692)
at org.apache.hadoop.hbase.client.HTable.getStartKeysInRange(HTable.java:1769)
at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1724)
at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1704)
at org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1301)
... 47 more
Caused by: java.net.SocketTimeoutException: callTimeout=60000, callDuration=68451: row 'SYSTEM.CATALOG,TARGETCUST_DATA,99999999999999' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=host-10-191-36-24,16020,1551146724629, seqNum=0
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:169)
at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65)
... 3 more
Caused by: java.net.UnknownHostException: host-10-191-36-24
at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.<init>(AbstractRpcClient.java:315)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.createBlockingRpcChannel(AbstractRpcClient.java:267)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getClient(ConnectionManager.java:1639)
at org.apache.hadoop.hbase.client.ScannerCallable.prepare(ScannerCallable.java:162)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.prepare(ScannerCallableWithReplicas.java:376)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:134)
... 4 more
2019-02-26 17:35:35 [com.asiainfo.cb2.consumer.ReceiveSMSID]-[ERROR]:33 - sms consumer begin
2019-02-26 17:35:35 [com.asiainfo.cb2.consumer.ReceiveSMSID]-[ERROR]:38 - sms consumer pushType3
分析:
开始的时候,思路还是纠结在,超时的错误java.net.SocketTimeoutException: callTimeout=60000,想着有没有办法能够增加客户端的超时时间,然后还看了datanode的日志,认为是datanode在写数据导磁盘的时候性能问题,新增加hadoop节点,节点间在进行平衡,导致影响了性能,但是,对于以上的分析都没有更好的方法来解决问题。
结果:
等到后续,在仔细看错误,发现,Caused by: java.net.UnknownHostException: host-10-191-36-24,找不到主机的异常,才突然明白,应用程序首先连接到zk,然后zk告知region在哪个regionserver上,然后,应用程序再连接到hbase的regionserver上读写数据。
解决:
所以,在应用程序的/etc/hosts上配置新增的hbase regionserver节点的hosts解析,再次观察应用程序,该问题解决
文档创建时间:2019年2月27日10:34:38
最新文章
- 查看mac中磁盘空间占用情况
- BUG级别定义标准
- 抢凳子日数据sql
- Sublog: 支持Markdown和语法高亮的跨平台博客客户端
- ZOJ 2975 思维
- JAX-WS开发WebService程序
- ASP.NET中定制自己的委托和事件参数类
- Hibernate+jxl+excel导入数据库
- MySQL STRAIGHT_JOIN
- go JSON
- Delphi中三种方法获取Windows任务栏的高度
- css基础知识之属性选择器
- 顺序表的原理与python中的list类型
- Spring的IOC注解开发入门1
- Linux Capability探索实验
- odoo 订餐系统之消息提醒
- [加密]ESP32 -Secure Boot 安全方案
- win10笔记本实现双屏显示的自如切换
- [BZOJ3560]DZY Loves Math V(欧拉函数)
- Native Apps、Web Apps
热门文章
- 元素 ";context:component-scan"; 的前缀 ";context"; 未绑定。
- webView加载url,加载指定字符串
- python编码转换
- Spring IOC基础使用
- 【贪心】经营与开发 @upc_exam_5500
- js -- 绑定的click addEventListener 事件只触发一次
- MAC 开启与关闭SIP
- Java Lambda 表达式 对 Map 对象排序
- Ubuntu安装最新版nodejs
- mysql函数之SUBSTRING_INDEX(str,";/";,-1)