HDFS java API TROUBLESHOOTING
官方文档:
https://hadoop.apache.org/docs/r2.9.2/hadoop-project-dist/hadoop-common/SingleCluster.html
配置免密登录,用于 nameNode 与 dataNode 通信
ssh-keygen -t rsa
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
验证ssh,不需要输入密码即可登录。登录后执行 exit 退出。
ssh localhost
exist
etc/hadoop/core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://192.168.3.127:8020</value>
</property>
</configuration>
etc/hadoop/hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value></value>
</property> <property>
<name>dfs.name.dir</name>
<value>file:/home/hdfs/name</value>
<description>namenode上存储hdfs名字空间元数据 </description>
</property> <property>
<name>dfs.data.dir</name>
<value>file:/home/hdfs/data</value>
<description>datanode上数据块的物理存储位置</description>
</property>
</configuration>
开放端口
firewall-cmd --add-port=/tcp --permanent
firewall-cmd --add-port=/tcp --permanent
firewall-cmd --add-port=/tcp --permanent
firewall-cmd --reload
1. java.lang.IllegalArgumentException: URI has an authority component
在执行 `bin/hdfs namenode -format` 的时候报错。
检查 hdfs-site.xml 配置是否正确
<property>
<name>dfs.name.dir</name>
<value>file:/home/hdfs/name</value>
<description>namenode上存储hdfs名字空间元数据 </description>
</property> <property>
<name>dfs.data.dir</name>
<value>file:/home/hdfs/data</value>
<description>datanode上数据块的物理存储位置</description>
</property>
2. java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset.
解压 hadoop-2.9.2.tar.gz 到 D:\app\
System.setProperty("hadoop.home.dir", "D:\\app\\hadoop-2.9.2");
3. java.io.FileNotFoundException: Could not locate Hadoop executable: D:\app\hadoop-2.9.2\bin\winutils.exe
下载 winutils.exe 放到 {HADOOP_HOME}\bin\ 下
4. Permission denied: user=xxx, access=WRITE, inode="/":root:supergroup:drwxr-xr-x
/**
* 解决无权限访问,设置远程hadoop的linux用户名称
*/
private static final String USER = "root"; fileSystem = FileSystem.get(new URI(HDFS_PATH), configuration, USER);
5. java.net.ConnectException: Connection timed out: no further information 与 org.apache.hadoop.ipc.RemoteException: File /hello-hadoop.md could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and 1 node(s) are excluded in this operation.
# 开放 dataNode端口
firewall-cmd --add-port=/tcp --permanent
firewall-cmd --reload
6. No FileSystem for scheme "hdfs"
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>${org.apache.hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>${org.apache.hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${org.apache.hadoop.version}</version>
</dependency>
有问题欢迎留言交流。
技术交流群:282575808
--------------------------------------
声明: 原创文章,未经允许,禁止转载!
--------------------------------------
最新文章
- 网友分享 调用dll的语音朗读 不能变速,不好
- Delphi 调用C# DLL(包含委托)
- [ZigBee] 7、ZigBee之UART剖析(ONLY串口发送)
- jQuery的XX如何实现?——3.data与cache机制
- 4、android xml中drawableTop(drawableBoottom、drawableLeft、drawableRight)在java代码中的动态配置
- 第二百九十七天 how can I 坚持
- Oracle UTL_HTTP(收集汇总有用资料)
- SQL进阶随笔--case用法(一)
- Serialization
- MHA(上)
- Centos 7 安装Docker-ce记录
- ES标准
- Python并发复习4- concurrent.futures模块(线程池和进程池)
- Java爬取B站弹幕 —— Python云图Wordcloud生成弹幕词云
- diff 命令详解
- 10.2-uC/OS-III内部任务管理(任务状态)
- WebView&;HTML5-----使用WebView播放HTML5视频文件
- webdriver 获取页面response
- java文件的读写程序代码
- 五十四 网络编程 TCP编程