Spark-HBase集成错误之 java.lang.NoClassDefFoundError: org/htrace/Trace
2024-08-23 12:35:53
在进行Spark与HBase 集成的过程中遇到以下问题:
java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionStateBuilder':
at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$instantiateSessionState(SparkSession.scala:)
at org.apache.spark.sql.SparkSession$$anonfun$sessionState$.apply(SparkSession.scala:)
at org.apache.spark.sql.SparkSession$$anonfun$sessionState$.apply(SparkSession.scala:)
at scala.Option.getOrElse(Option.scala:)
at org.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:)
at org.apache.spark.sql.SparkSession.sessionState(SparkSession.scala:)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$getOrCreate$.apply(SparkSession.scala:)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$getOrCreate$.apply(SparkSession.scala:)
at scala.collection.mutable.HashMap$$anonfun$foreach$.apply(HashMap.scala:)
at scala.collection.mutable.HashMap$$anonfun$foreach$.apply(HashMap.scala:)
at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:)
at org.apache.spark.repl.Main$.createSparkSession(Main.scala:)
... elided
Caused by: java.lang.ClassNotFoundException: java.lang.NoClassDefFoundError: org/htrace/Trace when creating Hive client using classpath:...... Please make sure that jars for your version of hive and hadoop are included in the paths passed to spark.sql.hive.metastore.jars.
at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:)
at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:)
at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:)
at org.apache.spark.sql.hive.HiveExternalCatalog.client$lzycompute(HiveExternalCatalog.scala:)
at org.apache.spark.sql.hive.HiveExternalCatalog.client(HiveExternalCatalog.scala:)
at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$.apply$mcZ$sp(HiveExternalCatalog.scala:)
at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$.apply(HiveExternalCatalog.scala:)
at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$.apply(HiveExternalCatalog.scala:)
at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:)
at org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:)
at org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:)
at org.apache.spark.sql.internal.SharedState.externalCatalog(SharedState.scala:)
at org.apache.spark.sql.hive.HiveSessionStateBuilder.externalCatalog(HiveSessionStateBuilder.scala:)
at org.apache.spark.sql.hive.HiveSessionStateBuilder.catalog$lzycompute(HiveSessionStateBuilder.scala:)
at org.apache.spark.sql.hive.HiveSessionStateBuilder.catalog(HiveSessionStateBuilder.scala:)
at org.apache.spark.sql.hive.HiveSessionStateBuilder.catalog(HiveSessionStateBuilder.scala:)
at org.apache.spark.sql.internal.BaseSessionStateBuilder.build(BaseSessionStateBuilder.scala:)
at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$instantiateSessionState(SparkSession.scala:)
... more
Caused by: java.lang.reflect.InvocationTargetException: java.lang.NoClassDefFoundError: org/htrace/Trace
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:)
at java.lang.reflect.Constructor.newInstance(Constructor.java:)
at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:)
... more
Caused by: java.lang.NoClassDefFoundError: org/htrace/Trace
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:)
at com.sun.proxy.$Proxy17.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:)
at com.sun.proxy.$Proxy18.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:)
at org.apache.hadoop.hdfs.DistributedFileSystem$.doCall(DistributedFileSystem.java:)
at org.apache.hadoop.hdfs.DistributedFileSystem$.doCall(DistributedFileSystem.java:)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:)
at org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:)
at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:)
at org.apache.spark.sql.hive.client.HiveClientImpl.<init>(HiveClientImpl.scala:)
... more
Caused by: java.lang.ClassNotFoundException: org.htrace.Trace
at java.net.URLClassLoader.findClass(URLClassLoader.java:)
at java.lang.ClassLoader.loadClass(ClassLoader.java:)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:)
at java.lang.ClassLoader.loadClass(ClassLoader.java:)
... more
<console>:: error: not found: value spark
import spark.implicits._
^
<console>:: error: not found: value spark
import spark.sql
^
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.2.
/_/ Using Scala version 2.11. (Java HotSpot(TM) -Bit Server VM, Java 1.8.0_60)
Type in expressions to have them evaluated.
Type :help for more information. scala>
我事先已经将所需要的jar支持包全部拷贝到了各节点下的spark路径下的jar目录,但是在启动bin/spark-shell时总是报以上错误。在网上也看来很多解决方法,比如将缺失的jar包 :hbase/lib 下 hbase*.jar,metrics*.jar.htrace*.jar 等拷贝到spark/jar目录,但这不正是我的启动准备工作吗,等于没说!
从故障根源分析,造成此错误的原因在于Spark启动的时候无法正常从我们的jar目录加载到它所需要的jar包(此处表现为/htrace-core-3.0.4.jar包)。所以,直接将包添加到jar目录是无法解决此问题的,而我们需要做的是给它指明CLASSPATH路径,在它不能找到所需要的jar包时就可以从此路径寻找。
解决办法:
在spark/conf目录下的spark-env.sh配置文件添加以下配置内容
export HBASE_CLASSPATH='hbase classpath'
export HADOOP_CLASSPATH='hadoop classpath'
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HBASE_CLASSPATH
以上就是博主为大家介绍的这一板块的主要内容,这都是博主自己的学习过程,希望能给大家带来一定的指导作用,有用的还望大家点个支持,如果对你没用也望包涵,有错误烦请指出。如有期待可关注博主以第一时间获取更新哦,谢谢!同时也欢迎转载,但必须在博文明显位置标注原文地址,解释权归博主所有!
最新文章
- 微软Microsoft开源历程与相关资源
- [Shell]条件判断与流程控制:if, case, for, while, until
- 接口、抽象类、方法复写、类Equals方法重写
- ZOJ 2750 Idiomatic Phrases Game(Dijkstra)
- css margin collapse
- UIActivityIndicatorView活动控制器的大小改变
- SQL Server 查看数据库空间分配情况的 2 种方法
- Django form模块使用心得
- jQuery对于动态生成的元素绑定无效的问题~~
- PA教材提纲 TAW10-1
- c复杂函数指针
- 动手动脑(lesson 3)
- drupal的node.html.twig说明
- Eclipse无法编译,提示错误“找不到或者无法加载主类”解决方法
- python——ADSL拨号程序
- Django:全文检索功能可参考博客
- Angular使用总结 --- 模型驱动表单
- day10 python学习 函数的嵌套命名空间作用域 三元运算 位置参数 默认参数 动态参数
- angular可自定义的对话框,弹窗指令
- Js添加、读取、删除cookie,判断cookie是否有效,指定domain域下主路径path下设置cookie,设置expires过期时间