CentOS 7安装Hadoop集群
准备三台虚拟机,ip分别为192.168.220.10(master)、192.168.220.11(slave1)、192.168.220.12(slave2)
准备好jdk-6u45-linux-x64.bin和hadoop-1.2.1-bin.tar.gz,放在/usr/local/src/目录下
安装JDK(每台虚拟机都安装)
1.进入到/usr/local/src/目录,执行./jdk-6u45-linux-x64.bin
2.修改~/.bashrc,在文件末尾增加三行
export JAVA_HOME=/usr/local/src/jdk1.6.0_45
export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib
export PATH=$PATH:$JAVA_HOME/bin
3.使环境变量生效,执行source ~/.bashrc
安装Hadoop
在192.168.220.10机器上安装hadoop
1.进入到/usr/local/src/目录,解压hadoop-1.2.1-bin.tar.gz,执行tar -zxf hadoop-1.2.1-bin.tar.gz
2.修改配置文件
masters文件
master
slaves文件
slave1
slave2
core-site.xml文件
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/src/hadoop-1.2.1/tmp</value>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://192.168.220.10:9000</value>
</property>
</configuration>
mapred-site.xml文件
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>http://192.168.220.10:9001</value>
</property>
</configuration>
hdfs-site.xml文件
<configuration>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
</configuration>
hadoop-env.sh文件,在后面添加一行
export JAVA_HOME=/usr/local/src/jdk1.6.0_45
3.将/usr/local/src/hadoop-1.2.1目录拷贝到192.168.220.11、192.168.220.12机器上
配置hostname
配置192.168.220.10的主机名为master
1.执行hostname master
2.修改/etc/hostname文件
master
修改192.168.220.11的主机名为slave1,修改192.168.220.12的主机名为slave2
配置host文件
三台机器的/etc/hosts文件末尾添加以下代码
192.168.220.10 master
192.168.220.11 slave1
192.168.220.12 slave2
关闭防火墙和SELinux
分别在每台机器上执行
systemctl stop firewalld.service
setenforce
配置SSH
1.在192.168.220.10上执行ssh-keygen,在~目录下新增.ssh目录,目录中的文件为id_rsa,id_rsa.pub
2.将id_rsa.pub拷贝为authorized_keys
cp id_rsa.pub authorized_keys
3.在192.168.220.11和192.168.220.12上分别执行ssh-keygen
4.将192.168.220.11和192.168.220.12上id_rsa.pub的内容分别拷贝到192.168.220.10的authorized_keys文件中,如下:
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9mGRhFOdcoHw9GUnKQmqThNKpsyah93Dtq/d8RICGWIHDRJ3GXd0sEcb743ejwbuCMmtlhheXcU0FuyA6Cm0jvMyvDfaPKArtxl6KT7Z93uC0VDCXDRomueux81HAIVjc7ZqlXwVeYs1LITxEeJykKlFOXvK7JexWhWGdMMADwxbFMbaNsZ9EwRxcFLFtNg65FQ+u8CIV9KR3D02kemwLCsP+xiRcgs+wirQPm5JM+2cJoLsVQBz3Hk335IsEhc1Xb9Cralo8Tt8gh/ho8K/1pVjvyW1b0LkP9HGNdwVYD9wkWdEJRkryLXBEXpjk4xu+riF+N4rOzJD root@master
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn79fdfR/NjzPVD3NPj1vBBfQdVOrv7jeb4UJCOsd7xioPRiz8gOQnOmhu5C+GchbyGA+tg5pXwnNJTOO2wn32U4lOPndW0okN/wqyN4vgq/taJi7JgY/8rneBiGaIIdNIy/pAGlMwb53Qn766adetMhsxYMD2l4uxmbVVjzCRb8QP5EsAYTmmFOODzJsPm70uF3j1Q8zGavYg0wFSYR/yECQns4DBSuBJNxdGY6PskBXqurahwi5yaR3vWV1Ix4wtB6BYuQomEnGdzOSfrBMZ/yc5tXo0xmEfY7wFkize6z9Pm2E3oDoMR18YkwT1Cz6fHikVILA9cldtL root@slave1
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCydYCvASzCZggks4hMqOcYSGLO2eAvocWezNOMwspTfpJ105Jumb/vf5h6cRZeckq56IvhSV6t6mytk4pZoZjjZPSmWvCwLtMRMPShNbA3BYtj5V3WRKV8ZcMrNdD//U7iHHoJm57vI/m+XO42YSYjPw7JDkb8Ij9b6zgI3fyvbSSYeXb451PlyJLHdxIzRMAaZDSbAML9e7EO8VJB9Wf9bXpow4+VeP33it3kgMNUlHQtyqduSwYGxVVtGsUTJkxnuRsbWeeA1/pp8MNFKUgBTMALTVHByglgZqwGcbblJxsG832PIZNRECIFqorm6odftjnT4DR7/0yR root@slave2
5.将192.168.220.10的authorized_keys文件拷贝到192.168.220.11、192.168.220.12机器上
配置完成后三台机器可以互相访问不用密码
好了,到这里Hadoop集群就配置完了,让我们来使用下吧
在192.168.220.10上格式化namenode
进入到/usr/local/src/hadoop-1.2.1/bin目录,执行
./hadoop namenode -format
出现
// :: INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = master/192.168.220.10
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 1.2.1
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013
STARTUP_MSG: java = 1.6.0_45
************************************************************/
// :: INFO util.GSet: Computing capacity for map BlocksMap
// :: INFO util.GSet: VM type = -bit
// :: INFO util.GSet: 2.0% max memory =
// :: INFO util.GSet: capacity = ^ = entries
// :: INFO util.GSet: recommended=, actual=
// :: INFO namenode.FSNamesystem: fsOwner=root
// :: INFO namenode.FSNamesystem: supergroup=supergroup
// :: INFO namenode.FSNamesystem: isPermissionEnabled=true
// :: INFO namenode.FSNamesystem: dfs.block.invalidate.limit=
// :: INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval= min(s), accessTokenLifetime= min(s)
// :: INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length =
// :: INFO namenode.NameNode: Caching file names occuring more than times
// :: INFO common.Storage: Image file /usr/local/src/hadoop-1.2./tmp/dfs/name/current/fsimage of size bytes saved in seconds.
// :: INFO namenode.FSEditLog: closing edit log: position=, editlog=/usr/local/src/hadoop-1.2./tmp/dfs/name/current/edits
// :: INFO namenode.FSEditLog: close success: truncate to , editlog=/usr/local/src/hadoop-1.2./tmp/dfs/name/current/edits
// :: INFO common.Storage: Storage directory /usr/local/src/hadoop-1.2./tmp/dfs/name has been successfully formatted.
// :: INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at master/192.168.220.10
************************************************************/
用jps查看进程
[root@master bin]# jps
JobTracker
NameNode
SecondaryNameNode
Jps
在192.168.220.11查看进程
DataNode
Jps
TaskTracker
在192.168.220.12查看进程
DataNode
TaskTracker
Jps
测试下:
执行./hadoop fs -ls /
drwxr-xr-x - root supergroup -- : /usr
上传文件,执行./hadoop fs -put /root/w.txt /
查看文件执行./hadoop fs -cat /w.txt,显示
ddd
成功
注:
问题:上传文件时出现以下错误
[root@master bin]# ./hadoop fs -put /root/a.txt /
// :: WARN hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /a.txt could only be replicated to nodes, instead of
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:)
at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:)
at org.apache.hadoop.ipc.Server$Handler$.run(Server.java:)
at org.apache.hadoop.ipc.Server$Handler$.run(Server.java:)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:) at org.apache.hadoop.ipc.Client.call(Client.java:)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:)
at com.sun.proxy.$Proxy1.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:)
at com.sun.proxy.$Proxy1.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$(DFSClient.java:)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:) // :: WARN hdfs.DFSClient: Error Recovery for null bad datanode[] nodes == null
// :: WARN hdfs.DFSClient: Could not get block locations. Source file "/a.txt" - Aborting...
put: java.io.IOException: File /a.txt could only be replicated to nodes, instead of
// :: ERROR hdfs.DFSClient: Failed to close file /a.txt
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /a.txt could only be replicated to nodes, instead of
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:)
at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:)
at org.apache.hadoop.ipc.Server$Handler$.run(Server.java:)
at org.apache.hadoop.ipc.Server$Handler$.run(Server.java:)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:) at org.apache.hadoop.ipc.Client.call(Client.java:)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:)
at com.sun.proxy.$Proxy1.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)
at java.lang.reflect.Method.invoke(Method.java:)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:)
at com.sun.proxy.$Proxy1.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$(DFSClient.java:)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:)
解决方法:关闭防火墙
最新文章
- RDIFramework.NET -.NET快速信息化系统开发整合框架 【开发实例 EasyUI】之产品管理(WebForm版)
- spark streaming中使用checkpoint
- RMAN备份演练初级篇
- 判断字符串中是否有SQL攻击代码
- 加密app.config
- Dubbo xml配置 和注解配置 写法
- npm install 本地安装与全局安装
- C++应用程序在Windows下的编译、链接(一)概述
- ecostore搜索注意事项
- 针对iPhone的pt、Android的dp、HTML的css像素与dpr、设计尺寸和物理像素的浅分析
- video+ audio
- 基于vue-cli配置移动端自适应
- DB2批量插入性能对比
- oracle 根据一个表生成另一个新表和一个现有表给一个新的表赋值
- keepalived 安装篇-个人实践-编译安装
- git 创建本地仓库、远程仓库,上传项目
- webpack浅析~
- linux系统Centos环境下搭建SVN服务器及权限配置
- spring boot 多数据源配置与使用
- Android系统启动流程(四)Launcher启动过程与系统启动流程