CDH6.2的spark访问oss
CDH6配置oss后:
spark的配置
/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars
/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hive/lib
/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/spark/jars #只是这个目录就可以
/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop/lib
这几个目录可能都要放jar包
这个目录也可以要放:
Caused by: java.lang.ClassNotFoundException: org.jdom.input.JDOMParseException
cd /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/spark/jars/
ls |grep ali
rm -f aliyun-sdk-oss-2.8.3.jar
rm -f hadoop-aliyun.jar
ls |grep ali
cp /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/aliyun-sdk-oss-2.8.3.jar /etc/hive/auxlib/
cp /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/hadoop-aliyun-3.0.0-cdh6.2.0.jar /etc/hive/auxlib/
cp /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/aliyun-sdk-oss-2.8.3.jar /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hive/lib/
cp /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/hadoop-aliyun-3.0.0-cdh6.2.0.jar /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hive/lib/
cp /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/aliyun-sdk-oss-2.8.3.jar /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/spark/jars/
cp /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/hadoop-aliyun-3.0.0-cdh6.2.0.jar /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/spark/jars/
cp /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/aliyun-sdk-oss-2.8.3.jar /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop/lib/
cp /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/hadoop-aliyun-3.0.0-cdh6.2.0.jar /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop/lib/
关于阿里云OSS报错-java.lang.ClassNotFoundException: org.jdom.input.JDOMParseException:
https://tech.antfin.com/docs/2/97907
cp /data/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/jdom-1.1.jar /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/spark/jars/
最新文章
- ASP.NET提取多层嵌套json数据的方法
- WebViewJavascriptBridge详细使用（转载）
- Android DiskLruCache 硬盘缓存
- 4. 在Inspector面板中显示类中变量+ 拓展编辑器
- Filebeat中文指南
- Canvas rontate(旋转) 使用误区
- [SignalR]初步认识以及安装
- 求和问题(DFS)
- Java对象比较器对泛型List进行排序-Demo
- Beauty Contest 凸包+旋转卡壳法
- docker之使用System.Drawing生成图片缺少Gdiplus.dll错误
- 第44章：MongoDB-集群--Sharding(分片)--分片的片键选择
- Docker入门3------手动编辑自定义镜像
- Android : 跟我学Binder --- (3) C程序示例
- python(leetcode)-136只出现一次的数字
- springboot 项目pom.xml文件基本配置
- 转载：HTuple数据和VC数据的相互赋值
- BTC和BCH 区别和联系？
- About Saliency Object Detection
- java 多线程 28 ： 多线程组件之 Semaphore 信号量