Hadoop集群模式安装出现的若干问题

一、域名解析问题

域名解析暂时失败问题
　　vim /etc/sysconfig/network
　　查看主机名
　　vim etc/hosts
配置IP地址与主机名 192.168.60.132 centos　　#改正就OK啦

二、启动问题

Starting namenodes on [localhost]

ERROR: Attempting to operate on hdfs namenode as root

ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation.

Starting datanodes

ERROR: Attempting to operate on hdfs datanode as root

ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation.

Starting secondary namenodes [bogon]

ERROR: Attempting to operate on hdfs secondarynamenode as root

ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation.

处理1

$ vim sbin/start-dfs.sh

$ vim sbin/stop-dfs.sh

两处增加以下内容

HDFS_DATANODE_USER=root

HDFS_DATANODE_SECURE_USER=hdfs

HDFS_NAMENODE_USER=root

HDFS_SECONDARYNAMENODE_USER=root

处理2

$ vim sbin/start-yarn.sh

$ vim sbin/stop-yarn.sh

两处增加以下内容

YARN_RESOURCEMANAGER_USER=root

HADOOP_SECURE_DN_USER=yarn

YARN_NODEMANAGER_USER=root

三、端口问题

    localhost: ssh: connect to host localhost port 22: Cannot assign requested address

    cd /etc/ssh

    vim sshd_config

    添加 Port 22

四、Failed to get D-Bus connection: Operation not permitted

解决方法：docker run --privileged -ti -e "container=docker" -v /sys/fs/cgroup:/sys/fs/cgroup hadoop-master /usr/sbin/init

五、服务报错

sshd re-exec requires execution with an absolute path

在开启SSHD服务时报错.

sshd re-exec requires execution with an absolute path

用绝对路径启动,也报错如下:

Could not load host key: /etc/ssh/ssh_host_key

Could not load host key: /etc/ssh/ssh_host_rsa_key

Could not load host key: /etc/ssh/ssh_host_dsa_key

Disabling protocol version 1. Could not load host key

Disabling protocol version 2. Could not load host key

sshd: no hostkeys available — exiting

解决过程:

#ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key

#ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key

#/usr/sbin/sshd

执行后报错：

Could not load host key: /etc/ssh/ssh_host_ecdsa_key

Could not load host key: /etc/ssh/ssh_host_ed25519_key

解决过程：

#ssh-keygen -t dsa -f /etc/ssh/ssh_host_ecdsa_key

#ssh-keygen -t rsa -f /etc/ssh/ssh_host_ed25519_key

#/usr/sbin/sshd

六、报错如下

    WARNING: HADOOP_SECURE_DN_USER has been replaced by HDFS_DATANODE_SECURE_USER. Using value of HADOOP_SECURE_DN_USER.

    Starting namenodes on [master]

    master: /usr/hadoop/hadoop-3.2.0/libexec/hadoop-functions.sh: line 982: ssh: command not found

    Starting datanodes

    Last login: Mon Jan 28 08:32:32 UTC 2019 on pts/0

    localhost: /usr/hadoop/hadoop-3.2.0/libexec/hadoop-functions.sh: line 982: ssh: command not found

    Starting secondary namenodes [b982e2adc393]

    Last login: Mon Jan 28 08:32:33 UTC 2019 on pts/0

    b982e2adc393: /usr/hadoop/hadoop-3.2.0/libexec/hadoop-functions.sh: line 982: ssh: command not found

    Starting resourcemanager

    Last login: Mon Jan 28 08:32:35 UTC 2019 on pts/0

    Starting nodemanagers

    Last login: Mon Jan 28 08:32:42 UTC 2019 on pts/0

    localhost: /usr/hadoop/hadoop-3.2.0/libexec/hadoop-functions.sh: line 982: ssh: command not foun

解决：

    在

     $ vim sbin/start-dfs.sh

     $ vim sbin/stop-dfs.sh

    将HADOOP_SECURE_DN_USER=hdfs替换为HADOOP_DATANODE_SECURE_DN_USER=hdfs

    centos默认安装有ssh服务，没有客户端。

    查看ssh安装

    # rpm -qa | grep openssh

    openssh-5.3p1-123.el6_9.x86_64

    openssh-server-5.3p1-123.el6_9.x86_64

    没有安装openssh-clients

    yum安装ssh客户端

    yum -y install openssh-clients

七、docker: Error response from daemon: cgroups: cannot find cgroup mount destination: unknown.

没有找到具体的解决方法，重启后可以访问。

八、permission denied (publickey,gssapi-keyex,gssapi-with-mic,password)

vim /etc/ssh/sshd_config

# $OpenBSD: sshd_config,v 1.80 2008/07/02 02:24:18 djm Exp $

        # This is the sshd server system-wide configuration file.  See

        # sshd_config(5) for more information.

        # This sshd was compiled with PATH=/usr/local/bin:/bin:/usr/bin

        # The strategy used for options in the default sshd_config shipped with

        # OpenSSH is to specify options with their default value where

        # possible, but leave them commented.  Uncommented options change a

        # default value.

        #Port 22

        #AddressFamily any

        #ListenAddress 0.0.0.0

        #ListenAddress ::

        # Disable legacy (protocol version 1) support in the server for new

        # installations. In future the default will change to require explicit

        # activation of protocol 1

        Protocol 2

        # HostKey for protocol version 1

-----------
就是改一下信息
RSAAuthentication yes
PubkeyAuthentication yes
PasswordAuthentication no

改完后一定要重启ssh服务 service sshd restart
systemctl restart sshd.service

九、WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

系统预装的glibc库是2.12版本，而hadoop期望是2.14版本，所以打印警告信息。
现在有两个办法，重新编译glibc.2.14版本，安装后专门给hadoop使用，这个有点危险。
第二个办法直接在log4j日志中去除告警信息。在//usr/local/hadoop-2.5.2/etc/hadoop/log4j.properties文件中添加
log4j.logger.org.apache.hadoop.util.NativeCodeLoader=ERROR

十、MapReduce操作出现错误: 找不到或无法加载主类org.apache.hadoop.mapreduce.v2.app.MRAppMaster 问题解决方法

在命令行下输入 Hadoop classpath
编辑yarn-site.xml
添加一下内容

<configuration>

<property>

<name>yarn.application.classpath</name>

<value>输入刚才返回的Hadoop classpath路径</value>

</property>

</configuration>

巴特西

Hadoop集群模式安装出现的若干问题

最新文章

热门文章