Sqoop-问题
2024-08-28 09:44:10
1.权限问题
// :: INFO mapreduce.Job: Job job_1555064328415_0328 failed with state FAILED due to: Job setup failed : org.apache.hadoop.security.AccessControlException: Permission denied: user=yarn, access=WRITE, inode="/user/yarn":hdfs:supergroup:drwxr-xr-x
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:)
at org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:)
oozie调用sqoop时报错,解决:改所有者
sudo -u hdfs hadoop fs -chown -R yarn /user/yarn
2.少MySQL驱动包
// :: INFO hive.metastore: Closed a connection to metastore, current connections:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.0.-.cdh6.0.0.p0./jars/slf4j-log4j12-1.7..jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.0.-.cdh6.0.0.p0./jars/log4j-slf4j-impl-2.8..jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
// :: INFO sqoop.Sqoop: Running Sqoop version: 1.4.-cdh6.0.0
// :: WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
// :: INFO conf.HiveConf: Found configuration file file:/etc/hive/conf.cloudera.hive/hive-site.xml
// :: INFO hive.metastore: Trying to connect to metastore with URI thrift://hadoop1:9083
// :: INFO hive.metastore: Opened a connection to metastore, current connections:
// :: INFO hive.metastore: Connected to metastore.
// :: INFO hive.metastore: Trying to connect to metastore with URI thrift://hadoop1:9083
// :: INFO hive.metastore: Opened a connection to metastore, current connections:
// :: INFO hive.metastore: Connected to metastore.
// :: WARN tool.BaseSqoopTool: Output field/record delimiter options are not useful in HCatalog jobs for most of the output types except text based formats is text. It is better to use --hive-import in those cases. For non text formats,
// :: INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
// :: INFO tool.CodeGenTool: Beginning code generation
// :: ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: Could not load db driver class: com.mysql.jdbc.Driver
java.lang.RuntimeException: Could not load db driver class: com.mysql.jdbc.Driver
at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:)
at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:)
at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:)
at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:)
at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:)
at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:)
at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:)
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:)
at org.apache.sqoop.Sqoop.run(Sqoop.java:)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:)
at org.apache.sqoop.Sqoop.main(Sqoop.java:)
// :: INFO hive.metastore: Closed a connection to metastore, current connections:
sqoop时,应该是有的节点没有放mysql驱动连接jar,将mysql驱动包放入/usr/share/java/下(mysql-connector-java.jar)
最新文章
- Meta标签中的viewport属性及含义
- ARM指令教程
- 金山云—ubuntu-web服务器完整部署笔记
- DSP(1) -- 离散时间信号的序列类型
- 脉冲计数功能在ESM335x-Linux主板上的实现
- Windows Server 2012及以上安装IIS的步骤
- servlet session 相关
- c# 并行运算
- 机器学习中的范数规则化之(一)L0、L1与L2范数(转)
- iptables规则表
- 关于开源框架GPUImage 的简单说明
- OC语言-02面向对象的三大特性
- 一些.Net面试题 (BS 方向)
- [Unity3D]Unity3D游戏开发之ACT游戏三连击效果实现综述
- Godaddy Drupal Update
- BST 二叉搜索树
- java 类与对象
- DB Query Analyzer has been downloaded more than 100,000 times
- MYSQL之IFNULL
- uniDAC的安装和使用
热门文章
- Spark源码分析 – SparkEnv
- mysql5.7.22在centos7.5下的安装
- ArcGIS Data Store 初体验
- 给Django后台富文本编辑器添加上传文件的功能
- python定义函数时默认参数注意事项
- 浅谈pipreqs组件(自动生成需要导入的模块信息)
- rabbitMQ基本概念
- 17.出现fatal signal(SIGSEGV),code 1,fault addr 0x0 in tid 29931的问题
- How to store scaling parameters for later use
- MySQL协议分析(1)