通过tarball形式安装HBASE Cluster（CDH5.0.2）——Hadoop NameNode HA 切换引起的Hbase错误，以及Hbase如何基于NameNode的HA进行配置

配置HBASE的时候一开始按照cdh网站上的说明，hbase.rootdir的值设置使用的是基于Hadoop Namenode HA的nameservice

<property>

      <name>hbase.rootdir</name>

      <value>hdfs://hbasecluster/hbase</value>

</property>

配置后，使用start-hbase.sh启动，只有位于bakup-masters文件中的HMaster能启动，hbase配置的位于Hadoop DateNode上的RegionServer都启动失败，所报错误为连接hdfs:/hbasecluster:8020失败（），当时网上搜索未果，只好改成单机模式：

<property>

      <name>hbase.rootdir</name>

      <value>hdfs://nn1:8020/hbase</value>

</property>

此时，hbase可以启动成功，hbase shell测试，远程java程序测试皆通过。逐渐地我就忘记这个配置遗留问题了，结果昨天心血来潮又测试了一下Namenode的HA切换，这个倒是成功了没有问题。可是在测试java程序访问hbase发现失败了，到服务器上看主节点zk1上的HMaster进程没有了，zk2，zk3上的HMaster进程虽然有也不能链接，于zk1上重新启动hbase集群

stop-hbase.sh

start-hbase.sh

jps

随后用jps查看，发现HMaster进程很快消失，查看log发现，报如下错误：

2014-07-24 08:51:47,023 FATAL [master:zk1:60000] master.HMaster: Unhandled exception. Starting shutdown.

org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby

	at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)

	at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1565)

	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1183)

	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3492)

	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:764)

	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:764)

	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)

	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)

	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)

	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1986)

	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1982)

	at java.security.AccessController.doPrivileged(Native Method)

	at javax.security.auth.Subject.doAs(Subject.java:415)

	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)

	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1980)

	at org.apache.hadoop.ipc.Client.call(Client.java:1409)

	at org.apache.hadoop.ipc.Client.call(Client.java:1362)

	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)

	at com.sun.proxy.$Proxy12.getFileInfo(Unknown Source)

	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

	at java.lang.reflect.Method.invoke(Method.java:606)

	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)

	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)

	at com.sun.proxy.$Proxy12.getFileInfo(Unknown Source)

	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:699)

	at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1757)

	at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1124)

	at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1120)

	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)

	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1120)

	at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1398)

	at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:438)

	at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:146)

	at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:127)

	at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:789)

	at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:606)

	at java.lang.Thread.run(Thread.java:745)

2014-07-24 08:51:47,025 INFO  [master:zk1:60000] master.HMaster: Aborting

2014-07-24 08:51:47,026 DEBUG [master:zk1:60000] master.HMaster: Stopping service threads

2014-07-24 08:51:47,026 INFO  [master:zk1:60000] ipc.RpcServer: Stopping server on 60000

2014-07-24 08:51:47,026 INFO  [RpcServer.handler=0,port=60000] ipc.RpcServer: RpcServer.handler=0,port=60000: exiting

2014-07-24 08:51:47,027 INFO  [RpcServer.handler=1,port=60000] ipc.RpcServer: RpcServer.handler=1,port=60000: exiting

2014-07-24 08:51:47,027 INFO  [RpcServer.handler=3,port=60000] ipc.RpcServer: RpcServer.handler=3,port=60000: exiting

2014-07-24 08:51:47,027 INFO  [RpcServer.handler=2,port=60000] ipc.RpcServer: RpcServer.handler=2,port=60000: exiting

2014-07-24 08:51:47,027 INFO  [RpcServer.handler=4,port=60000] ipc.RpcServer: RpcServer.handler=4,port=60000: exiting

2014-07-24 08:51:47,027 INFO  [RpcServer.handler=5,port=60000] ipc.RpcServer: RpcServer.handler=5,port=60000: exiting

2014-07-24 08:51:47,027 INFO  [RpcServer.handler=6,port=60000] ipc.RpcServer: RpcServer.handler=6,port=60000: exiting

2014-07-24 08:51:47,027 INFO  [RpcServer.handler=7,port=60000] ipc.RpcServer: RpcServer.handler=7,port=60000: exiting

2014-07-24 08:51:47,027 INFO  [RpcServer.handler=8,port=60000] ipc.RpcServer: RpcServer.handler=8,port=60000: exiting

2014-07-24 08:51:47,028 INFO  [RpcServer.handler=9,port=60000] ipc.RpcServer: RpcServer.handler=9,port=60000: exiting

2014-07-24 08:51:47,028 INFO  [RpcServer.handler=10,port=60000] ipc.RpcServer: RpcServer.handler=10,port=60000: exiting

2014-07-24 08:51:47,028 INFO  [RpcServer.handler=11,port=60000] ipc.RpcServer: RpcServer.handler=11,port=60000: exiting

2014-07-24 08:51:47,028 INFO  [RpcServer.handler=12,port=60000] ipc.RpcServer: RpcServer.handler=12,port=60000: exiting

2014-07-24 08:51:47,028 INFO  [RpcServer.handler=13,port=60000] ipc.RpcServer: RpcServer.handler=13,port=60000: exiting

2014-07-24 08:51:47,028 INFO  [RpcServer.handler=14,port=60000] ipc.RpcServer: RpcServer.handler=14,port=60000: exiting

2014-07-24 08:51:47,028 INFO  [RpcServer.handler=15,port=60000] ipc.RpcServer: RpcServer.handler=15,port=60000: exiting

2014-07-24 08:51:47,028 INFO  [RpcServer.handler=16,port=60000] ipc.RpcServer: RpcServer.handler=16,port=60000: exiting

2014-07-24 08:51:47,029 INFO  [RpcServer.handler=17,port=60000] ipc.RpcServer: RpcServer.handler=17,port=60000: exiting

2014-07-24 08:51:47,029 INFO  [RpcServer.handler=19,port=60000] ipc.RpcServer: RpcServer.handler=19,port=60000: exiting

2014-07-24 08:51:47,029 INFO  [RpcServer.handler=18,port=60000] ipc.RpcServer: RpcServer.handler=18,port=60000: exiting

2014-07-24 08:51:47,029 INFO  [RpcServer.handler=20,port=60000] ipc.RpcServer: RpcServer.handler=20,port=60000: exiting

2014-07-24 08:51:47,029 INFO  [RpcServer.handler=21,port=60000] ipc.RpcServer: RpcServer.handler=21,port=60000: exiting

2014-07-24 08:51:47,029 INFO  [RpcServer.handler=22,port=60000] ipc.RpcServer: RpcServer.handler=22,port=60000: exiting

2014-07-24 08:51:47,029 INFO  [RpcServer.handler=24,port=60000] ipc.RpcServer: RpcServer.handler=24,port=60000: exiting

2014-07-24 08:51:47,029 INFO  [RpcServer.handler=23,port=60000] ipc.RpcServer: RpcServer.handler=23,port=60000: exiting

2014-07-24 08:51:47,030 INFO  [RpcServer.handler=26,port=60000] ipc.RpcServer: RpcServer.handler=26,port=60000: exiting

2014-07-24 08:51:47,030 INFO  [RpcServer.handler=27,port=60000] ipc.RpcServer: RpcServer.handler=27,port=60000: exiting

2014-07-24 08:51:47,030 INFO  [RpcServer.handler=25,port=60000] ipc.RpcServer: RpcServer.handler=25,port=60000: exiting

2014-07-24 08:51:47,030 INFO  [RpcServer.handler=28,port=60000] ipc.RpcServer: RpcServer.handler=28,port=60000: exiting

2014-07-24 08:51:47,030 INFO  [RpcServer.handler=29,port=60000] ipc.RpcServer: RpcServer.handler=29,port=60000: exiting

2014-07-24 08:51:47,036 INFO  [Replication.RpcServer.handler=0,port=60000] ipc.RpcServer: Replication.RpcServer.handler=0,port=60000: exiting

2014-07-24 08:51:47,036 INFO  [Replication.RpcServer.handler=1,port=60000] ipc.RpcServer: Replication.RpcServer.handler=1,port=60000: exiting

2014-07-24 08:51:47,037 INFO  [Replication.RpcServer.handler=2,port=60000] ipc.RpcServer: Replication.RpcServer.handler=2,port=60000: exiting

2014-07-24 08:51:47,038 INFO  [RpcServer.listener,port=60000] ipc.RpcServer: RpcServer.listener,port=60000: stopping

2014-07-24 08:51:47,039 INFO  [master:zk1:60000] master.HMaster: Stopping infoServer

2014-07-24 08:51:47,043 INFO  [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopped

2014-07-24 08:51:47,043 INFO  [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopping

2014-07-24 08:51:47,055 INFO  [master:zk1:60000] mortbay.log: Stopped SelectChannelConnector@0.0.0.0:60010

2014-07-24 08:51:47,190 INFO  [master:zk1:60000] zookeeper.ZooKeeper: Session: 0x164756c65ba5008e closed

2014-07-24 08:51:47,191 INFO  [master:zk1:60000] master.HMaster: HMaster main thread exiting

2014-07-24 08:51:47,191 ERROR [main] master.HMasterCommandLine: Master exiting

java.lang.RuntimeException: HMaster Aborted

	at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:192)

	at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:134)

	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)

	at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)

	at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2789)

2014-07-24 08:51:47,191 INFO  [main-EventThread] zookeeper.ClientCnxn: EventThread shut down

仔细看，发现是namenode报的错，检查nn1状态，发现此时nn1处于standby状态，nn2已经处于active状态，突然想起昨天的测试。这激起了我解决这个问题的决心，经过一翻goooooooooooooooogle，终于在hbase官网上的hbase文档中发现了如下几行内容：

2.2.2.2.3. HDFS Client Configuration

Of note, if you have made HDFS client configuration on your Hadoop cluster -- i.e. configuration you want HDFS clients to use as opposed to server-side configurations -- HBase will not see this configuration unless you do one of the following:

　　　　　　Add a pointer to your HADOOP_CONF_DIR to the HBASE_CLASSPATH environment variable in hbase-env.sh.

　　　　　　Add a copy of hdfs-site.xml (or hadoop-site.xml) or, better, symlinks, under ${HBASE_HOME}/conf, or

　　　　　　if only a small set of HDFS client configurations, add them to hbase-site.xml.

An example of such an HDFS client configuration is dfs.replication. If for example, you want to run with a replication factor of 5, hbase will create files with the default of 3 unless you do the above to make the configuration available to HBase.

我才用的是方法二，在所有的master节点和regionserver节点上都建立了hdfs-site.xml的符号链接(scp 分发符号连接不行，都变成真实文件了)，hbase.rootdir配置为nameservice方式，再次测试hbase集群启动，成功!

然后测试hadoop namenode切换，hbase没问题了，至此困扰我多日的问题终于解决了。

最新文章

  1. spring-初始化完成后运行指定内容
  2. ubuntu包管理
  3. WPF控件经验小结：（1） ToolBar去掉右边箭头（扩展图标）
  4. plain framework 1 参考手册 入门指引之 简明教程
  5. ubuntu下各个软件完全卸载
  6. Lintcode: Update Bits
  7. Unable to write inside TEMP environment path
  8. ASP实现https和http之间转化
  9. CC2530芯片介绍
  10. 在树莓派1B上编译安装lnmp服务器
  11. iPhone开发 数据持久化总结（终结篇）—5种数据持久化方法对比
  12. 关于js闭包是否真的会造成内存泄漏(转载)
  13. view类的XML属性
  14. 每天学点Java小知识【1】
  15. 10亿美元融资腾讯跟头，Grail要用基因测序做癌症早期筛查
  16. webpack教程(三)——热刷新
  17. 理论铺垫：阻塞IO、非阻塞IO、IO多路复用/事件驱动IO(单线程高并发原理)、异步IO
  18. HDU6278 Just h-index
  19. Python（四） —— 函数
  20. matlab与python读取tiff文件

热门文章

  1. Android之Activity切换
  2. 分页用到的子查询sql语句
  3. 3. DNN神经网络的正则化
  4. 随机颜色-js
  5. iOS同步后couldn&#39;t load project
  6. JAVA实用工具--javamail
  7. Bootstrap Popover 隐藏的Javasript方法
  8. android开发(27) 看看我的手机里都有什么传感器
  9. python出现UnicodeEncodeError有可能产生的另一个原因
  10. setTimeout()和setInterval()的区别