基于HBase Hadoop 分布式集群环境下的MapReduce程序开发

　　HBase分布式集群环境搭建成功后，连续4、5天实验客户端Map/Reduce程序开发，这方面的代码网上多得是，写个测试代码非常容易，可是真正运行起来可说是历经挫折。下面就是我最终调通并让程序在集群上运行起来的一些经验教训。

　　一、首先说一下我的环境：

　　1，集群的环境配置请见这篇博文。

　　2，开发客户机环境：操作系统是CentOS6.5，JDK版本是1.7.0-60，开发工具是Eclipse（原始安装是从google的ADT网站下载的ADT专用开发环境，后来加装了Java企业开发的工具，启动Flash还是ADT的：））。

　　3，etc/hosts文件添加了集群的所有主机。

　　二、必要准备工作：

　　1，java类库准备：从集群中scp复制了Hadoop/share下所有文件，hbase/lib目录下的所有文件到开发机的/home/username/lib目录下备用(其中username是开发用户名称)，打开eclipse，选择菜单window->Preferences,

在左侧树形菜单中导航至Java--Build Path--User Libraries，我添加了3个User Library，“hadoop-common”，“hadoop-yarn”，“hbase”

然后分别添加如下（这是User Libraries的导出文件，可以在上面的界面中导入）的jar包。

<?xml version="1.0" encoding="UTF-8" standalone="no"?>

<eclipse-userlibraries version="2">

    <library name="hadoop-common" systemlibrary="false">

        <archive path="/home/gzg/javadev/lib/hadoop/mapreduce/hadoop-mapreduce-client-app-2.3.0-cdh5.0.2.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/hadoop-auth-2.3.0-cdh5.0.2.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/hadoop-annotations-2.3.0-cdh5.0.2.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/hadoop-common-2.3.0-cdh5.0.2.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/commons-logging-1.1.3.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/commons-compress-1.4.1.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/guava-11.0.2.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/commons-collections-3.2.1.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/slf4j-api-1.7.5.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/slf4j-log4j12-1.7.5.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/zookeeper-3.4.5-cdh5.0.2.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/commons-configuration-1.6.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/commons-math3-3.1.1.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/activation-1.1.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/asm-3.2.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/avro-1.7.5-cdh5.0.2.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/commons-beanutils-1.7.0.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/commons-beanutils-core-1.8.0.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/commons-cli-1.2.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/commons-codec-1.4.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/commons-digester-1.8.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/commons-el-1.0.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/commons-httpclient-3.1.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/commons-io-2.4.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/commons-lang-2.6.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/commons-net-3.1.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/httpclient-4.2.5.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/httpcore-4.2.5.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/jackson-core-asl-1.8.8.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/jackson-jaxrs-1.8.8.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/jackson-xc-1.8.8.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/jasper-compiler-5.5.23.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/jasper-runtime-5.5.23.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/java-xmlbuilder-0.4.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/jaxb-api-2.2.2.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/jaxb-impl-2.2.3-1.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/jersey-core-1.9.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/jersey-json-1.9.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/jersey-server-1.9.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/jets3t-0.9.0.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/jettison-1.1.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/jetty-6.1.26.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/jetty-util-6.1.26.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/jsch-0.1.42.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/jsp-api-2.1.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/jsr305-1.3.9.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/junit-4.8.2.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/log4j-1.2.17.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/mockito-all-1.8.5.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/paranamer-2.3.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/protobuf-java-2.5.0.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/servlet-api-2.5.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/snappy-java-1.0.4.1.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/stax-api-1.0-2.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/xmlenc-0.52.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/common/lib/xz-1.0.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/mapreduce/lib/netty-3.6.2.Final.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/hdfs/hadoop-hdfs-2.3.0-cdh5.0.2.jar"/>

    </library>

    <library name="hadoop-yarn" systemlibrary="false">

        <archive path="/home/gzg/javadev/lib/hadoop/mapreduce2/hadoop-mapreduce-client-app-2.3.0-cdh5.0.2.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/mapreduce2/hadoop-mapreduce-client-core-2.3.0-cdh5.0.2.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/mapreduce2/hadoop-mapreduce-client-jobclient-2.3.0-cdh5.0.2.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/mapreduce2/hadoop-mapreduce-client-common-2.3.0-cdh5.0.2.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/mapreduce2/hadoop-mapreduce-client-shuffle-2.3.0-cdh5.0.2.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/yarn/hadoop-yarn-api-2.3.0-cdh5.0.2.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/yarn/hadoop-yarn-client-2.3.0-cdh5.0.2.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/yarn/hadoop-yarn-common-2.3.0-cdh5.0.2.jar"/>

        <archive path="/home/gzg/javadev/lib/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.3.0-cdh5.0.2.jar"/>

    </library>

    <library name="hbase" systemlibrary="false">

        <archive path="/home/gzg/javadev/lib/hbase/hbase-client-0.96.1.1-cdh5.0.2.jar"/>

        <archive path="/home/gzg/javadev/lib/hbase/hbase-common-0.96.1.1-cdh5.0.2.jar"/>

        <archive path="/home/gzg/javadev/lib/hbase/hbase-hadoop2-compat-0.96.1.1-cdh5.0.2.jar"/>

        <archive path="/home/gzg/javadev/lib/hbase/hbase-hadoop-compat-0.96.1.1-cdh5.0.2.jar"/>

        <archive path="/home/gzg/javadev/lib/hbase/hbase-it-0.96.1.1-cdh5.0.2.jar"/>

        <archive path="/home/gzg/javadev/lib/hbase/hbase-prefix-tree-0.96.1.1-cdh5.0.2.jar"/>

        <archive path="/home/gzg/javadev/lib/hbase/hbase-protocol-0.96.1.1-cdh5.0.2.jar"/>

        <archive path="/home/gzg/javadev/lib/hbase/hbase-testing-util-0.96.1.1-cdh5.0.2.jar"/>

        <archive path="/home/gzg/javadev/lib/hbase/hbase-thrift-0.96.1.1-cdh5.0.2.jar"/>

        <archive path="/home/gzg/javadev/lib/hbase/protobuf-java-2.5.0.jar"/>

        <archive path="/home/gzg/javadev/lib/hbase/hbase-server-0.96.1.1-cdh5.0.2.jar"/>

        <archive path="/home/gzg/javadev/lib/hbase/hbase-shell-0.96.1.1-cdh5.0.2.jar"/>

        <archive path="/home/gzg/javadev/lib/hbase/htrace-core-2.01.jar"/>

    </library>

</eclipse-userlibraries>

注：这个很有可能并不是运行Hbase Map/Reduce任务需要的最精简的类库，恕我没有逐一测试。

　　2，新建一个Java/Application，我的名字就叫hbase1，然后在项目project视图中点击右键，选择“Build Path”--“Add Libraries...”--选择“User Library”，然后选中上面添加的三个User Library。

　　3，复制集群的三个配置文件到项目的src目录下，这三个配置文件是：

core-site.xml  hdfs-site.xml  yarn-site.xml

　　三、编写代码，这里基本按照我所遇挫折过程逐步完善代码（缺少类库之类的就略掉了）：

　　 1，遭遇空指针错误--reverseDNS

主文件内容：

/**

 * @author gzg

 *

 */

public class Test1  {

	/**

	 * @param args

	 * @throws IOException

	 * @throws ZooKeeperConnectionException

	 * @throws MasterNotRunningException

	 */

	Configuration conf=null;

	public static void main(String[] args) {

		try {

			new TestHBaseMapred().run(false);

		} catch(yh.utils.system.CommandError cep){

			cep.printStackTrace();

		} catch (ClassNotFoundException e) {

			// TODO Auto-generated catch block

			e.printStackTrace();

		} catch (IOException e) {

			// TODO Auto-generated catch block

			e.printStackTrace();

		} catch (InterruptedException e) {

			// TODO Auto-generated catch block

			e.printStackTrace();

		}

	}

}

　TestHBaseMapRed.java内容：

public class TestHBaseMapred {

	public void run(boolean toFile) throws ClassNotFoundException, IOException, InterruptedException{

		org.apache.hadoop.conf.Configuration conf = new org.apache.hadoop.conf.Configuration();
　　　　　　　　　//设置zookeeper的地址s

		conf.set("hbase.zookeeper.quorum", "zk1,zk2,zk3");
　　　　　　　　　//此设置好像没用

		conf.set("hadoop.job.user", "hadoop");

		org.apache.hadoop.mapreduce.Job job=null;

		String  tmpTable = "_mrtest1";

		yh.utils.ElapsedTimeMeasure etm=yh.utils.ElapsedTimeMeasure.createAndStart();

		try {

			job = Job.getInstance(conf);

			job.setJobName("count meter");

			if(toFile==false){

				HBaseUtil.dropTable(conf, tmpTable);

				HBaseUtil.createTable(conf, tmpTable, "c");

				job.setOutputFormatClass(org.apache.hadoop.hbase.mapreduce.TableOutputFormat.class);

				job.getConfiguration().set(org.apache.hadoop.hbase.mapreduce.TableOutputFormat.OUTPUT_TABLE, tmpTable);

				//

				job.setReducerClass(TblReduce.class);

			}else{

				job.setOutputFormatClass(org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.class);

				Path op = new Path("result");

				org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.setOutputPath(job, op);

				//

				job.setReducerClass(TxtReduce.class);

			}			

			Scan scan = new Scan();

			scan.addFamily(Bytes.toBytes("c"));

			org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initTableMapperJob("dzdl_t1",scan,TblMap.class ,Text.class,LongWritable.class,job);

			job.waitForCompletion(true);

		} catch (IOException e) {

			// TODO Auto-generated catch block

			e.printStackTrace();

		}

		System.out.println("time elapsed :"+etm.stop().toString());

	}

	public static class TblMap extends org.apache.hadoop.hbase.mapreduce.TableMapper<Text, LongWritable>{

		@Override

		protected void map(ImmutableBytesWritable key,Result result,org.apache.hadoop.mapreduce.Mapper.Context  ctx) throws IOException, InterruptedException{

			Cell c=result.getColumnLatestCell(Bytes.toBytes("c"), Bytes.toBytes("col3"));

			String col= Bytes.toString(CellUtil.cloneQualifier(c));

			Text kv = new Text();

			kv.set("count_result");

			LongWritable lv = new LongWritable();

			lv.set(1);

			ctx.write(kv, lv);

			//org.apache.log4j.Logger.getRootLogger().info(kv+" : 1");

		}

	}

	public static class  TblReduce extends org.apache.hadoop.hbase.mapreduce.TableReducer<Text,LongWritable,ImmutableBytesWritable>{

		@Override

		protected void reduce(Text key,Iterable<LongWritable> values,Context ctx) throws IOException, InterruptedException{

			long t =0;

			for(LongWritable v:values){

				t+=v.get();

			}

			byte[] ok=Bytes.toBytes("count_result");

			Put p=new Put(ok);

			p.add(Bytes.toBytes("c"),key.copyBytes(),Bytes.toBytes(t));

		p.add(Bytes.toBytes("c"),Bytes.toBytes("14112896"),Bytes.toBytes(14112896));

			ctx.write(new ImmutableBytesWritable(ok),p);

		}

	}

	public static class TxtReduce extends org.apache.hadoop.mapreduce.lib.reduce.LongSumReducer<Text>{

		@Override

		public void reduce(Text key,Iterable<LongWritable> values,Context ctx) throws IOException, InterruptedException{

			long t =0;

			for(LongWritable v:values){

				t+=v.get();

			}

			//Put p=new Put(key.copyBytes());

			//p.add(Bytes.toBytes("c"),key.copyBytes(),Bytes.toBytes(t));

			//ctx.write(new ImmutableBytesWritable(key.copyBytes()),p);

			LongWritable l = new LongWritable();

			l.set(t);

			ctx.write(key,l);

		}

	}

}

此时运行，会报如下错误：

Exception in thread "main" java.lang.NullPointerException

	at org.apache.hadoop.net.DNS.reverseDns(DNS.java:92)

	at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.reverseDNS(TableInputFormatBase.java:218)

	at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.getSplits(TableInputFormatBase.java:184)

	at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:493)

	at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:510)

	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:394)

	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1295)

	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1292)

	at java.security.AccessController.doPrivileged(Native Method)

	at javax.security.auth.Subject.doAs(Subject.java:415)

	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)

	at org.apache.hadoop.mapreduce.Job.submit(Job.java:1292)

	at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1313)

	at hbase1.TestHBaseMapred.run(TestHBaseMapred.java:82)

	at hbase1.Test1.main(Test1.java:69)

看错误，不知道为什么需要调用DNS服务，hosts文件好像没有作用，搜索很久，终于在一个老外的问题里找到答案，conf添加如下设置解决。

conf.set("hbase.nameserver.address","nn1,nn2");

添加后，运行仍会报下面的错无，但是好像并不影响程序继续运行：

2014-08-21 14:05:04,158 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn1/192.168.5.111 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2: Name or service not known]; remaining name '111.5.168.192.in-addr.arpa'

2014-08-21 14:05:04,170 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn6/192.168.5.116 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '116.5.168.192.in-addr.arpa'

2014-08-21 14:05:04,171 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn3/192.168.5.113 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '113.5.168.192.in-addr.arpa'

2014-08-21 14:05:04,172 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn4/192.168.5.114 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '114.5.168.192.in-addr.arpa'

2014-08-21 14:05:04,173 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn4/192.168.5.114 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '114.5.168.192.in-addr.arpa'

2014-08-21 14:05:04,173 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn1/192.168.5.111 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '111.5.168.192.in-addr.arpa'

2014-08-21 14:05:04,174 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn6/192.168.5.116 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '116.5.168.192.in-addr.arpa'

2014-08-21 14:05:04,174 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn4/192.168.5.114 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '114.5.168.192.in-addr.arpa'

2014-08-21 14:05:04,175 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn5/192.168.5.115 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '115.5.168.192.in-addr.arpa'

2014-08-21 14:05:04,175 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn6/192.168.5.116 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '116.5.168.192.in-addr.arpa'

2014-08-21 14:05:04,199 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn5/192.168.5.115 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '115.5.168.192.in-addr.arpa'

2014-08-21 14:05:04,200 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn2/192.168.5.112 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '112.5.168.192.in-addr.arpa'

2014-08-21 14:05:04,201 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn5/192.168.5.115 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '115.5.168.192.in-addr.arpa'

2014-08-21 14:05:04,201 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn2/192.168.5.112 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '112.5.168.192.in-addr.arpa'

2014-08-21 14:05:04,202 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn3/192.168.5.113 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '113.5.168.192.in-addr.arpa'

2014-08-21 14:05:04,539 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - number of splits:15

　　 2，本地运行的问题。解决了上面的问题，程序终于可以运行了，可是我在rm的web界面中却看不到我提交的任务，再次求助bing，终于找到症结所在，需要添加设置如下：

                //此项必须设置，否则程序只是在本地客户端上运行，通过rm的浏览器管理界面看不到！

		conf.set("mapreduce.framework.name", "yarn");

　　 3，添加了上面语句后，新的问题又来了，运行程序出现新的错误：

2014-08-21 14:15:08,887 INFO [org.apache.hadoop.mapreduce.Job] - The url to track the job: http://rm2:23188/proxy/application_1408378794442_0014/

2014-08-21 14:15:08,888 INFO [org.apache.hadoop.mapreduce.Job] - Running job: job_1408378794442_0014

2014-08-21 14:15:21,466 INFO [org.apache.hadoop.mapreduce.Job] - Job job_1408378794442_0014 running in uber mode : false

2014-08-21 14:15:21,469 INFO [org.apache.hadoop.mapreduce.Job] -  map 0% reduce 0%

2014-08-21 14:15:34,988 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000013_0, Status : FAILED

2014-08-21 14:15:35,023 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000011_0, Status : FAILED

Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)

	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)

	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)

	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)

	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)

	at java.security.AccessController.doPrivileged(Native Method)

	at javax.security.auth.Subject.doAs(Subject.java:415)

	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)

	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)

	... 8 more

Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)

	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)

	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)

	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)

	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)

	at java.security.AccessController.doPrivileged(Native Method)

	at javax.security.auth.Subject.doAs(Subject.java:415)

	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)

	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)

	... 8 more

2014-08-21 14:15:39,485 INFO [org.apache.hadoop.mapreduce.Job] -  map 7% reduce 0%

2014-08-21 14:15:39,490 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000003_0, Status : FAILED

Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)

	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)

	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)

	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)

	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)

	at java.security.AccessController.doPrivileged(Native Method)

	at javax.security.auth.Subject.doAs(Subject.java:415)

	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)

	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)

	... 8 more

2014-08-21 14:15:39,497 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000006_0, Status : FAILED

Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)

	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)

	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)

	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)

	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)

	at java.security.AccessController.doPrivileged(Native Method)

	at javax.security.auth.Subject.doAs(Subject.java:415)

	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)

	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)

	... 8 more

2014-08-21 14:15:39,505 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000001_0, Status : FAILED

Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)

	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)

	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)

	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)

	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)

	at java.security.AccessController.doPrivileged(Native Method)

	at javax.security.auth.Subject.doAs(Subject.java:415)

	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)

	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)

	... 8 more

2014-08-21 14:15:40,663 INFO [org.apache.hadoop.mapreduce.Job] -  map 0% reduce 0%

2014-08-21 14:15:40,736 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000004_0, Status : FAILED

Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)

	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)

	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)

	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)

	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)

	at java.security.AccessController.doPrivileged(Native Method)

	at javax.security.auth.Subject.doAs(Subject.java:415)

	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)

	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)

	... 8 more

2014-08-21 14:15:42,783 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000013_1, Status : FAILED

Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)

	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)

	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)

	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)

	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)

	at java.security.AccessController.doPrivileged(Native Method)

	at javax.security.auth.Subject.doAs(Subject.java:415)

	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)

	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)

	... 8 more

2014-08-21 14:15:44,921 INFO [org.apache.hadoop.mapreduce.Job] -  map 7% reduce 0%

2014-08-21 14:15:44,925 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000005_0, Status : FAILED

Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)

	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)

	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)

	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)

	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)

	at java.security.AccessController.doPrivileged(Native Method)

	at javax.security.auth.Subject.doAs(Subject.java:415)

	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)

	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)

	... 8 more

Container killed by the ApplicationMaster.

Container killed on request. Exit code is 143

Container exited with a non-zero exit code 143

2014-08-21 14:15:46,019 INFO [org.apache.hadoop.mapreduce.Job] -  map 0% reduce 0%

2014-08-21 14:15:46,066 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000000_0, Status : FAILED

Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)

	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)

	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)

	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)

	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)

	at java.security.AccessController.doPrivileged(Native Method)

	at javax.security.auth.Subject.doAs(Subject.java:415)

	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)

	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)

	... 8 more

2014-08-21 14:15:46,070 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000002_0, Status : FAILED

Container killed by the ApplicationMaster.

Container killed on request. Exit code is 143

Container exited with a non-zero exit code 143

Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)

	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)

	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)

	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)

	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)

	at java.security.AccessController.doPrivileged(Native Method)

	at javax.security.auth.Subject.doAs(Subject.java:415)

	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)

	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)

	... 8 more

2014-08-21 14:15:46,073 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000009_0, Status : FAILED

Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)

	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)

	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)

	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)

	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)

	at java.security.AccessController.doPrivileged(Native Method)

	at javax.security.auth.Subject.doAs(Subject.java:415)

	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)

	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)

	... 8 more

2014-08-21 14:15:46,076 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000014_0, Status : FAILED

Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)

	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)

	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)

	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)

	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)

	at java.security.AccessController.doPrivileged(Native Method)

	at javax.security.auth.Subject.doAs(Subject.java:415)

	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)

	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)

	... 8 more

2014-08-21 14:15:47,086 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000007_0, Status : FAILED

Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)

	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)

	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)

	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)

	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)

	at java.security.AccessController.doPrivileged(Native Method)

	at javax.security.auth.Subject.doAs(Subject.java:415)

	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)

	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)

	... 8 more

2014-08-21 14:15:59,146 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000006_1, Status : FAILED

Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)

	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)

	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)

	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)

	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)

	at java.security.AccessController.doPrivileged(Native Method)

	at javax.security.auth.Subject.doAs(Subject.java:415)

	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)

	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)

	... 8 more

2014-08-21 14:16:00,237 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000003_1, Status : FAILED

Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)

	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)

	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)

	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)

	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)

	at java.security.AccessController.doPrivileged(Native Method)

	at javax.security.auth.Subject.doAs(Subject.java:415)

	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)

	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)

	... 8 more

2014-08-21 14:16:01,312 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000013_2, Status : FAILED

Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)

	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)

	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)

	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)

	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)

	at java.security.AccessController.doPrivileged(Native Method)

	at javax.security.auth.Subject.doAs(Subject.java:415)

	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)

	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)

	... 8 more

2014-08-21 14:16:01,329 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000004_1, Status : FAILED

Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)

	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)

	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)

	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)

	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)

	at java.security.AccessController.doPrivileged(Native Method)

	at javax.security.auth.Subject.doAs(Subject.java:415)

	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)

	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)

	... 8 more

2014-08-21 14:16:03,396 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000008_0, Status : FAILED

Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)

	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)

	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)

	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)

	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)

	at java.security.AccessController.doPrivileged(Native Method)

	at javax.security.auth.Subject.doAs(Subject.java:415)

	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)

	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)

	... 8 more

2014-08-21 14:16:03,401 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000000_1, Status : FAILED

Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)

	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)

	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)

	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)

	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)

	at java.security.AccessController.doPrivileged(Native Method)

	at javax.security.auth.Subject.doAs(Subject.java:415)

	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)

	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)

	... 8 more

2014-08-21 14:16:04,488 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000009_1, Status : FAILED

Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)

	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)

	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)

	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)

	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)

	at java.security.AccessController.doPrivileged(Native Method)

	at javax.security.auth.Subject.doAs(Subject.java:415)

	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)

	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)

	... 8 more

2014-08-21 14:16:04,493 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000001_1, Status : FAILED

Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)

	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)

	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)

	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)

	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)

	at java.security.AccessController.doPrivileged(Native Method)

	at javax.security.auth.Subject.doAs(Subject.java:415)

	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)

	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)

	... 8 more

2014-08-21 14:16:05,530 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000010_0, Status : FAILED

Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)

	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)

	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)

	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)

	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)

	at java.security.AccessController.doPrivileged(Native Method)

	at javax.security.auth.Subject.doAs(Subject.java:415)

	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)

	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)

	... 8 more

2014-08-21 14:16:06,546 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000012_0, Status : FAILED

Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)

	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)

	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)

	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)

	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)

	at java.security.AccessController.doPrivileged(Native Method)

	at javax.security.auth.Subject.doAs(Subject.java:415)

	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)

	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)

	... 8 more

2014-08-21 14:16:09,672 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000006_2, Status : FAILED

Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)

	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)

	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)

	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)

	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)

	at java.security.AccessController.doPrivileged(Native Method)

	at javax.security.auth.Subject.doAs(Subject.java:415)

	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)

	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)

	... 8 more

2014-08-21 14:16:10,935 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000014_1, Status : FAILED

Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)

	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)

	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)

	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)

	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)

	at java.security.AccessController.doPrivileged(Native Method)

	at javax.security.auth.Subject.doAs(Subject.java:415)

	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)

	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)

	... 8 more

2014-08-21 14:16:19,239 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000012_1, Status : FAILED

Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)

	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)

	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)

	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)

	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)

	at java.security.AccessController.doPrivileged(Native Method)

	at javax.security.auth.Subject.doAs(Subject.java:415)

	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)

	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)

	... 8 more

2014-08-21 14:16:19,243 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000003_2, Status : FAILED

Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)

	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)

	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)

	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)

	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)

	at java.security.AccessController.doPrivileged(Native Method)

	at javax.security.auth.Subject.doAs(Subject.java:415)

	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)

	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)

	... 8 more

2014-08-21 14:16:19,246 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000000_2, Status : FAILED

Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)

	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)

	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)

	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)

	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)

	at java.security.AccessController.doPrivileged(Native Method)

	at javax.security.auth.Subject.doAs(Subject.java:415)

	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)

	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)

	... 8 more

2014-08-21 14:16:20,307 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000004_2, Status : FAILED

Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)

	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)

	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)

	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)

	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)

	at java.security.AccessController.doPrivileged(Native Method)

	at javax.security.auth.Subject.doAs(Subject.java:415)

	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)

	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)

	... 8 more

2014-08-21 14:16:21,326 INFO [org.apache.hadoop.mapreduce.Job] - Task Id : attempt_1408378794442_0014_m_000011_1, Status : FAILED

Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)

	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)

	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)

	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)

	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)

	at java.security.AccessController.doPrivileged(Native Method)

	at javax.security.auth.Subject.doAs(Subject.java:415)

	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)

	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

Caused by: java.lang.ClassNotFoundException: Class hbase1.TestHBaseMapred$TblMap not found

	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)

	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)

	... 8 more

2014-08-21 14:16:23,352 INFO [org.apache.hadoop.mapreduce.Job] -  map 100% reduce 100%

2014-08-21 14:16:24,378 INFO [org.apache.hadoop.mapreduce.Job] - Job job_1408378794442_0014 failed with state FAILED due to: Task failed task_1408378794442_0014_m_000013

Job failed as tasks failed. failedMaps:1 failedReduces:0

2014-08-21 14:16:24,584 INFO [org.apache.hadoop.mapreduce.Job] - Counters: 13

	Job Counters

		Failed map tasks=31

		Killed map tasks=13

		Launched map tasks=44

		Other local map tasks=29

		Data-local map tasks=15

		Total time spent by all maps in occupied slots (ms)=1665094

		Total time spent by all reduces in occupied slots (ms)=0

		Total time spent by all map tasks (ms)=832547

		Total vcore-seconds taken by all map tasks=832547

		Total megabyte-seconds taken by all map tasks=1278792192

	Map-Reduce Framework

		CPU time spent (ms)=0

		Physical memory (bytes) snapshot=0

		Virtual memory (bytes) snapshot=0

*** measure elapsed time,stoped at 2014-08-21 14:16:24:588

time elapsed :0days 0hours 1minutes 29seconds 879milliseconds[89.879seconds]

试着设置 job.setJarByClass(Test1.class); 无效。此问题困扰了我两天，难道只能上传到集群集群上运行吗？这样也太不方便了，N久后，终于我有幸找到了小鱼儿的博客的这篇文章MapReduce提交作业常见问题，终于是找到了解决方法。我采用的是文中所提第三种方法，在程序开始运行前自己首先打包自己，就是调用的是写好的shell文件，这里也贴出来，以下两个文件在和src同级的dist目录中

[gzg@centos dist]$ cat manifest.mf

Manifest-Version: 1.0

Class-Path: . $HADOOP_HOME/lib hbase1_lib/hadoop-mapreduce-client-app-2.3.0-cdh5.0.2.ja

 r hbase1_lib/hadoop-auth-2.3.0-cdh5.0.2.jar hbase1_lib/hadoop-annotat

 ions-2.3.0-cdh5.0.2.jar hbase1_lib/hadoop-common-2.3.0-cdh5.0.2.jar h

 base1_lib/commons-logging-1.1.3.jar hbase1_lib/commons-compress-1.4.1

 .jar hbase1_lib/guava-11.0.2.jar hbase1_lib/commons-collections-3.2.1

 .jar hbase1_lib/slf4j-api-1.7.5.jar hbase1_lib/slf4j-log4j12-1.7.5.ja

 r hbase1_lib/zookeeper-3.4.5-cdh5.0.2.jar hbase1_lib/commons-configur

 ation-1.6.jar hbase1_lib/commons-math3-3.1.1.jar hbase1_lib/activatio

 n-1.1.jar hbase1_lib/asm-3.2.jar hbase1_lib/avro-1.7.5-cdh5.0.2.jar h

 base1_lib/commons-beanutils-1.7.0.jar hbase1_lib/commons-beanutils-co

 re-1.8.0.jar hbase1_lib/commons-cli-1.2.jar hbase1_lib/commons-codec-

 1.4.jar hbase1_lib/commons-digester-1.8.jar hbase1_lib/commons-el-1.0

 .jar hbase1_lib/commons-httpclient-3.1.jar hbase1_lib/commons-io-2.4.

 jar hbase1_lib/commons-lang-2.6.jar hbase1_lib/commons-net-3.1.jar hb

 ase1_lib/httpclient-4.2.5.jar hbase1_lib/httpcore-4.2.5.jar hbase1_li

 b/jackson-core-asl-1.8.8.jar hbase1_lib/jackson-jaxrs-1.8.8.jar hbase

 1_lib/jackson-mapper-asl-1.8.8.jar hbase1_lib/jackson-xc-1.8.8.jar hb

 ase1_lib/jasper-compiler-5.5.23.jar hbase1_lib/jasper-runtime-5.5.23.

 jar hbase1_lib/java-xmlbuilder-0.4.jar hbase1_lib/jaxb-api-2.2.2.jar

 hbase1_lib/jaxb-impl-2.2.3-1.jar hbase1_lib/jersey-core-1.9.jar hbase

 1_lib/jersey-json-1.9.jar hbase1_lib/jersey-server-1.9.jar hbase1_lib

 /jets3t-0.9.0.jar hbase1_lib/jettison-1.1.jar hbase1_lib/jetty-6.1.26

 .jar hbase1_lib/jetty-util-6.1.26.jar hbase1_lib/jsch-0.1.42.jar hbas

 e1_lib/jsp-api-2.1.jar hbase1_lib/jsr305-1.3.9.jar hbase1_lib/junit-4

 .8.2.jar hbase1_lib/log4j-1.2.17.jar hbase1_lib/mockito-all-1.8.5.jar

  hbase1_lib/paranamer-2.3.jar hbase1_lib/protobuf-java-2.5.0.jar hbas

 e1_lib/servlet-api-2.5.jar hbase1_lib/snappy-java-1.0.4.1.jar hbase1_

 lib/stax-api-1.0-2.jar hbase1_lib/xmlenc-0.52.jar hbase1_lib/xz-1.0.j

 ar hbase1_lib/netty-3.6.2.Final.jar hbase1_lib/hadoop-hdfs-2.3.0-cdh5

 .0.2.jar hbase1_lib/hadoop-mapreduce-client-app-2.3.0-cdh5.0.2_2.jar

 hbase1_lib/hadoop-mapreduce-client-core-2.3.0-cdh5.0.2.jar hbase1_lib

 /hadoop-mapreduce-client-jobclient-2.3.0-cdh5.0.2.jar hbase1_lib/hado

 op-mapreduce-client-common-2.3.0-cdh5.0.2.jar hbase1_lib/hadoop-mapre

 duce-client-shuffle-2.3.0-cdh5.0.2.jar hbase1_lib/hadoop-yarn-api-2.3

 .0-cdh5.0.2.jar hbase1_lib/hadoop-yarn-client-2.3.0-cdh5.0.2.jar hbas

 e1_lib/hadoop-yarn-common-2.3.0-cdh5.0.2.jar hbase1_lib/hadoop-yarn-a

 pplications-distributedshell-2.3.0-cdh5.0.2.jar hbase1_lib/hbase-clie

 nt-0.96.1.1-cdh5.0.2.jar hbase1_lib/hbase-common-0.96.1.1-cdh5.0.2.ja

 r hbase1_lib/hbase-hadoop2-compat-0.96.1.1-cdh5.0.2.jar hbase1_lib/hb

 ase-hadoop-compat-0.96.1.1-cdh5.0.2.jar hbase1_lib/hbase-it-0.96.1.1-

 cdh5.0.2.jar hbase1_lib/hbase-prefix-tree-0.96.1.1-cdh5.0.2.jar hbase

 1_lib/hbase-protocol-0.96.1.1-cdh5.0.2.jar hbase1_lib/hbase-shell-0.9

 6.1.1-cdh5.0.2.jar hbase1_lib/hbase-testing-util-0.96.1.1-cdh5.0.2.ja

 r hbase1_lib/hbase-thrift-0.96.1.1-cdh5.0.2.jar hbase1_lib/protobuf-j

 ava-2.5.0_2.jar hbase1_lib/htrace-core-2.01.jar hbase1_lib/hbase-serv

 er-0.96.1.1-cdh5.0.2.jar

Main-Class: hbase1.Test1

[gzg@centos dist]$ cat makejar

cd "$(dirname $0)"

rm -f hbase1.jar

jar cvfm hbase1.jar manifest.mf  -C "../bin/" .

同时，修改了TestHBaseMapred.run方法签名，传递jar包路径，新的run方法如下：

public void run(String appJarFile,boolean toFile) throws ClassNotFoundException, IOException, InterruptedException{

		//org.apache.hadoop.conf.Configuration conf = yh.utils.hbase.HBaseUtil.getConfiguration("zk1,zk2,zk3");

		org.apache.hadoop.conf.Configuration conf = new org.apache.hadoop.conf.Configuration();

		conf.set("hbase.zookeeper.quorum", "zk1,zk2,zk3");

		conf.set("hadoop.job.user", "hadoop");

		conf.set("mapreduce.framework.name", "yarn");

		conf.set("hbase.nameserver.address","nn1,nn2");

		org.apache.hadoop.mapreduce.Job job=null;

		String  tmpTable = "_mrtest1";

		yh.utils.ElapsedTimeMeasure etm=yh.utils.ElapsedTimeMeasure.createAndStart();

		try {

			job = Job.getInstance(conf);

			job.setJobName("count meter");

			//job.setJarByClass(Test1.class);

			job.setJar(appJarFile);

			if(toFile==false){

				HBaseUtil.dropTable(conf, tmpTable);

				HBaseUtil.createTable(conf, tmpTable, "c");

				job.setOutputFormatClass(org.apache.hadoop.hbase.mapreduce.TableOutputFormat.class);

				job.getConfiguration().set(org.apache.hadoop.hbase.mapreduce.TableOutputFormat.OUTPUT_TABLE, tmpTable);

				//

				job.setReducerClass(TblReduce.class);

			}else{

				job.setOutputFormatClass(org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.class);

				Path op = new Path("result");

				org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.setOutputPath(job, op);

				//

				job.setReducerClass(TxtReduce.class);

			}			

			Scan scan = new Scan();

			scan.addFamily(Bytes.toBytes("c"));

			org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initTableMapperJob("dzdl_t1",scan,TblMap.class ,Text.class,LongWritable.class,job);

			job.waitForCompletion(true);

		} catch (IOException e) {

			// TODO Auto-generated catch block

			e.printStackTrace();

		}

		System.out.println("time elapsed :"+etm.stop().toString());

	}

　　 4，在选择写入文件的Reduce方法是，同样遇到权限问题，我采用的解决方法是给所有用户授予了读写权限，这里注意/tmp也要授权，运行时用到

[hadoop@zk1 hbase]$ hdfs dfs -chmod 777 /user

[hadoop@zk1 hbase]$ hdfs dfs -chmod 777 /tmp

　　另外，最后还有一点注意，设置log4j.properties文件中的级别为INFO，不要设置为WARN，否则看不到很多信息，不利于发现错误原因。

# Configure logging for testing: optionally with log file

#log4j.rootLogger=WARN, stdout

log4j.rootLogger=INFO, stdout

# log4j.rootLogger=WARN, stdout, logfile

log4j.appender.stdout=org.apache.log4j.ConsoleAppender

log4j.appender.stdout.layout=org.apache.log4j.PatternLayout

log4j.appender.stdout.layout.ConversionPattern=%d %p [%c] - %m%n

log4j.appender.logfile=org.apache.log4j.FileAppender

log4j.appender.logfile.File=target/spring.log

log4j.appender.logfile.layout=org.apache.log4j.PatternLayout

log4j.appender.logfile.layout.ConversionPattern=%d %p [%c] - %m%n

　　四、At last，终于大功告成，看看map/reduce的运行过程

*** measure elapsed time,started at 2014-08-21 14:48:35:880

2014-08-21 14:48:36,601 WARN [org.apache.hadoop.util.NativeCodeLoader] - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

2014-08-21 14:48:36,682 INFO [org.apache.hadoop.conf.Configuration.deprecation] - hadoop.native.lib is deprecated. Instead, use io.native.lib.available

2014-08-21 14:48:42,227 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:zookeeper.version=3.4.5-cdh5.0.2--1, built on 06/09/2014 16:09 GMT

2014-08-21 14:48:42,227 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:host.name=211.137.170.246

2014-08-21 14:48:42,227 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.version=1.7.0_60

2014-08-21 14:48:42,228 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.vendor=Oracle Corporation

2014-08-21 14:48:42,228 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.home=/usr/java/jdk1.7.0_60/jre

2014-08-21 14:48:42,228 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.class.path=/home/gzg/javadev/lib/hadoop/mapreduce/hadoop-mapreduce-client-app-2.3.0-cdh5.0.2.jar:/home/gzg/javadev/lib/hadoop/common/lib/hadoop-auth-2.3.0-cdh5.0.2.jar:/home/gzg/javadev/lib/hadoop/common/lib/hadoop-annotations-2.3.0-cdh5.0.2.jar:/home/gzg/javadev/lib/hadoop/common/hadoop-common-2.3.0-cdh5.0.2.jar:/home/gzg/javadev/lib/hadoop/common/lib/commons-logging-1.1.3.jar:/home/gzg/javadev/lib/hadoop/common/lib/commons-compress-1.4.1.jar:/home/gzg/javadev/lib/hadoop/common/lib/guava-11.0.2.jar:/home/gzg/javadev/lib/hadoop/common/lib/commons-collections-3.2.1.jar:/home/gzg/javadev/lib/hadoop/common/lib/slf4j-api-1.7.5.jar:/home/gzg/javadev/lib/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/home/gzg/javadev/lib/hadoop/common/lib/zookeeper-3.4.5-cdh5.0.2.jar:/home/gzg/javadev/lib/hadoop/common/lib/commons-configuration-1.6.jar:/home/gzg/javadev/lib/hadoop/common/lib/commons-math3-3.1.1.jar:/home/gzg/javadev/lib/hadoop/common/lib/activation-1.1.jar:/home/gzg/javadev/lib/hadoop/common/lib/asm-3.2.jar:/home/gzg/javadev/lib/hadoop/common/lib/avro-1.7.5-cdh5.0.2.jar:/home/gzg/javadev/lib/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/gzg/javadev/lib/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/gzg/javadev/lib/hadoop/common/lib/commons-cli-1.2.jar:/home/gzg/javadev/lib/hadoop/common/lib/commons-codec-1.4.jar:/home/gzg/javadev/lib/hadoop/common/lib/commons-digester-1.8.jar:/home/gzg/javadev/lib/hadoop/common/lib/commons-el-1.0.jar:/home/gzg/javadev/lib/hadoop/common/lib/commons-httpclient-3.1.jar:/home/gzg/javadev/lib/hadoop/common/lib/commons-io-2.4.jar:/home/gzg/javadev/lib/hadoop/common/lib/commons-lang-2.6.jar:/home/gzg/javadev/lib/hadoop/common/lib/commons-net-3.1.jar:/home/gzg/javadev/lib/hadoop/common/lib/httpclient-4.2.5.jar:/home/gzg/javadev/lib/hadoop/common/lib/httpcore-4.2.5.jar:/home/gzg/javadev/lib/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/home/gzg/javadev/lib/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/home/gzg/javadev/lib/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/home/gzg/javadev/lib/hadoop/common/lib/jackson-xc-1.8.8.jar:/home/gzg/javadev/lib/hadoop/common/lib/jasper-compiler-5.5.23.jar:/home/gzg/javadev/lib/hadoop/common/lib/jasper-runtime-5.5.23.jar:/home/gzg/javadev/lib/hadoop/common/lib/java-xmlbuilder-0.4.jar:/home/gzg/javadev/lib/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/gzg/javadev/lib/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/gzg/javadev/lib/hadoop/common/lib/jersey-core-1.9.jar:/home/gzg/javadev/lib/hadoop/common/lib/jersey-json-1.9.jar:/home/gzg/javadev/lib/hadoop/common/lib/jersey-server-1.9.jar:/home/gzg/javadev/lib/hadoop/common/lib/jets3t-0.9.0.jar:/home/gzg/javadev/lib/hadoop/common/lib/jettison-1.1.jar:/home/gzg/javadev/lib/hadoop/common/lib/jetty-6.1.26.jar:/home/gzg/javadev/lib/hadoop/common/lib/jetty-util-6.1.26.jar:/home/gzg/javadev/lib/hadoop/common/lib/jsch-0.1.42.jar:/home/gzg/javadev/lib/hadoop/common/lib/jsp-api-2.1.jar:/home/gzg/javadev/lib/hadoop/common/lib/jsr305-1.3.9.jar:/home/gzg/javadev/lib/hadoop/common/lib/junit-4.8.2.jar:/home/gzg/javadev/lib/hadoop/common/lib/log4j-1.2.17.jar:/home/gzg/javadev/lib/hadoop/common/lib/mockito-all-1.8.5.jar:/home/gzg/javadev/lib/hadoop/common/lib/paranamer-2.3.jar:/home/gzg/javadev/lib/hadoop/common/lib/protobuf-java-2.5.0.jar:/home/gzg/javadev/lib/hadoop/common/lib/servlet-api-2.5.jar:/home/gzg/javadev/lib/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/gzg/javadev/lib/hadoop/common/lib/stax-api-1.0-2.jar:/home/gzg/javadev/lib/hadoop/common/lib/xmlenc-0.52.jar:/home/gzg/javadev/lib/hadoop/common/lib/xz-1.0.jar:/home/gzg/javadev/lib/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/home/gzg/javadev/lib/hadoop/hdfs/hadoop-hdfs-2.3.0-cdh5.0.2.jar:/home/gzg/javadev/lib/hadoop/mapreduce2/hadoop-mapreduce-client-app-2.3.0-cdh5.0.2.jar:/home/gzg/javadev/lib/hadoop/mapreduce2/hadoop-mapreduce-client-core-2.3.0-cdh5.0.2.jar:/home/gzg/javadev/lib/hadoop/mapreduce2/hadoop-mapreduce-client-jobclient-2.3.0-cdh5.0.2.jar:/home/gzg/javadev/lib/hadoop/mapreduce2/hadoop-mapreduce-client-common-2.3.0-cdh5.0.2.jar:/home/gzg/javadev/lib/hadoop/mapreduce2/hadoop-mapreduce-client-shuffle-2.3.0-cdh5.0.2.jar:/home/gzg/javadev/lib/hadoop/yarn/hadoop-yarn-api-2.3.0-cdh5.0.2.jar:/home/gzg/javadev/lib/hadoop/yarn/hadoop-yarn-client-2.3.0-cdh5.0.2.jar:/home/gzg/javadev/lib/hadoop/yarn/hadoop-yarn-common-2.3.0-cdh5.0.2.jar:/home/gzg/javadev/lib/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.3.0-cdh5.0.2.jar:/home/gzg/javadev/lib/hbase/hbase-client-0.96.1.1-cdh5.0.2.jar:/home/gzg/javadev/lib/hbase/hbase-common-0.96.1.1-cdh5.0.2.jar:/home/gzg/javadev/lib/hbase/hbase-hadoop2-compat-0.96.1.1-cdh5.0.2.jar:/home/gzg/javadev/lib/hbase/hbase-hadoop-compat-0.96.1.1-cdh5.0.2.jar:/home/gzg/javadev/lib/hbase/hbase-it-0.96.1.1-cdh5.0.2.jar:/home/gzg/javadev/lib/hbase/hbase-prefix-tree-0.96.1.1-cdh5.0.2.jar:/home/gzg/javadev/lib/hbase/hbase-protocol-0.96.1.1-cdh5.0.2.jar:/home/gzg/javadev/lib/hbase/hbase-testing-util-0.96.1.1-cdh5.0.2.jar:/home/gzg/javadev/lib/hbase/hbase-thrift-0.96.1.1-cdh5.0.2.jar:/home/gzg/javadev/lib/hbase/protobuf-java-2.5.0.jar:/home/gzg/javadev/lib/hbase/hbase-server-0.96.1.1-cdh5.0.2.jar:/home/gzg/javadev/lib/hbase/hbase-shell-0.96.1.1-cdh5.0.2.jar:/home/gzg/javadev/lib/hbase/htrace-core-2.01.jar:/home/gzg/javadev/hbase1/bin:/usr/java/jdk1.7.0_60/jre/lib:/usr/lib64

2014-08-21 14:48:42,228 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

2014-08-21 14:48:42,228 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.io.tmpdir=/tmp

2014-08-21 14:48:42,228 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:java.compiler=<NA>

2014-08-21 14:48:42,228 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.name=Linux

2014-08-21 14:48:42,228 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.arch=amd64

2014-08-21 14:48:42,228 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:os.version=2.6.32-431.el6.x86_64

2014-08-21 14:48:42,228 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.name=gzg

2014-08-21 14:48:42,228 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.home=/home/gzg

2014-08-21 14:48:42,228 INFO [org.apache.zookeeper.ZooKeeper] - Client environment:user.dir=/home/gzg/javadev/hbase1

2014-08-21 14:48:42,230 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=zk2:2181,zk1:2181,zk3:2181 sessionTimeout=180000 watcher=hconnection-0x7f120aa6, quorum=zk2:2181,zk1:2181,zk3:2181, baseZNode=/hbase

2014-08-21 14:48:42,254 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=hconnection-0x7f120aa6 connecting to ZooKeeper ensemble=zk2:2181,zk1:2181,zk3:2181

2014-08-21 14:48:42,261 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server zk2/192.168.5.122:2181. Will not attempt to authenticate using SASL (unknown error)

2014-08-21 14:48:42,276 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to zk2/192.168.5.122:2181, initiating session

2014-08-21 14:48:42,335 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server zk2/192.168.5.122:2181, sessionid = 0x1647ed72def6009d, negotiated timeout = 40000

2014-08-21 14:48:43,407 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=zk2:2181,zk1:2181,zk3:2181 sessionTimeout=180000 watcher=catalogtracker-on-hconnection-0x7f120aa6, quorum=zk2:2181,zk1:2181,zk3:2181, baseZNode=/hbase

2014-08-21 14:48:43,408 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=catalogtracker-on-hconnection-0x7f120aa6 connecting to ZooKeeper ensemble=zk2:2181,zk1:2181,zk3:2181

2014-08-21 14:48:43,409 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server zk1/192.168.5.121:2181. Will not attempt to authenticate using SASL (unknown error)

2014-08-21 14:48:43,412 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to zk1/192.168.5.121:2181, initiating session

2014-08-21 14:48:43,421 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server zk1/192.168.5.121:2181, sessionid = 0x15478ef8bf2f016f, negotiated timeout = 40000

2014-08-21 14:48:53,961 INFO [org.apache.zookeeper.ZooKeeper] - Session: 0x15478ef8bf2f016f closed

2014-08-21 14:48:53,961 INFO [org.apache.zookeeper.ClientCnxn] - EventThread shut down

2014-08-21 14:48:53,985 INFO [org.apache.hadoop.hbase.client.HBaseAdmin] - Started disable of _mrtest1

2014-08-21 14:48:54,051 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=zk2:2181,zk1:2181,zk3:2181 sessionTimeout=180000 watcher=catalogtracker-on-hconnection-0x7f120aa6, quorum=zk2:2181,zk1:2181,zk3:2181, baseZNode=/hbase

2014-08-21 14:48:54,053 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server zk3/192.168.5.123:2181. Will not attempt to authenticate using SASL (unknown error)

2014-08-21 14:48:54,057 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=catalogtracker-on-hconnection-0x7f120aa6 connecting to ZooKeeper ensemble=zk2:2181,zk1:2181,zk3:2181

2014-08-21 14:48:54,067 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to zk3/192.168.5.123:2181, initiating session

2014-08-21 14:48:54,072 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server zk3/192.168.5.123:2181, sessionid = 0x174786daf57f01d0, negotiated timeout = 40000

2014-08-21 14:48:54,119 INFO [org.apache.zookeeper.ZooKeeper] - Session: 0x174786daf57f01d0 closed

2014-08-21 14:48:54,119 INFO [org.apache.zookeeper.ClientCnxn] - EventThread shut down

2014-08-21 14:48:54,235 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=zk2:2181,zk1:2181,zk3:2181 sessionTimeout=180000 watcher=catalogtracker-on-hconnection-0x7f120aa6, quorum=zk2:2181,zk1:2181,zk3:2181, baseZNode=/hbase

2014-08-21 14:48:54,236 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=catalogtracker-on-hconnection-0x7f120aa6 connecting to ZooKeeper ensemble=zk2:2181,zk1:2181,zk3:2181

2014-08-21 14:48:54,237 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server zk2/192.168.5.122:2181. Will not attempt to authenticate using SASL (unknown error)

2014-08-21 14:48:54,243 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to zk2/192.168.5.122:2181, initiating session

2014-08-21 14:48:54,281 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server zk2/192.168.5.122:2181, sessionid = 0x1647ed72def6009e, negotiated timeout = 40000

2014-08-21 14:48:54,325 INFO [org.apache.zookeeper.ZooKeeper] - Session: 0x1647ed72def6009e closed

2014-08-21 14:48:54,326 INFO [org.apache.zookeeper.ClientCnxn] - EventThread shut down

2014-08-21 14:48:54,558 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=zk2:2181,zk1:2181,zk3:2181 sessionTimeout=180000 watcher=catalogtracker-on-hconnection-0x7f120aa6, quorum=zk2:2181,zk1:2181,zk3:2181, baseZNode=/hbase

2014-08-21 14:48:54,560 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server zk3/192.168.5.123:2181. Will not attempt to authenticate using SASL (unknown error)

2014-08-21 14:48:54,561 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=catalogtracker-on-hconnection-0x7f120aa6 connecting to ZooKeeper ensemble=zk2:2181,zk1:2181,zk3:2181

2014-08-21 14:48:54,562 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to zk3/192.168.5.123:2181, initiating session

2014-08-21 14:48:54,568 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server zk3/192.168.5.123:2181, sessionid = 0x174786daf57f01d1, negotiated timeout = 40000

2014-08-21 14:48:54,584 INFO [org.apache.zookeeper.ZooKeeper] - Session: 0x174786daf57f01d1 closed

2014-08-21 14:48:54,584 INFO [org.apache.zookeeper.ClientCnxn] - EventThread shut down

2014-08-21 14:48:54,894 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=zk2:2181,zk1:2181,zk3:2181 sessionTimeout=180000 watcher=catalogtracker-on-hconnection-0x7f120aa6, quorum=zk2:2181,zk1:2181,zk3:2181, baseZNode=/hbase

2014-08-21 14:48:54,899 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=catalogtracker-on-hconnection-0x7f120aa6 connecting to ZooKeeper ensemble=zk2:2181,zk1:2181,zk3:2181

2014-08-21 14:48:54,900 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server zk3/192.168.5.123:2181. Will not attempt to authenticate using SASL (unknown error)

2014-08-21 14:48:54,901 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to zk3/192.168.5.123:2181, initiating session

2014-08-21 14:48:54,906 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server zk3/192.168.5.123:2181, sessionid = 0x174786daf57f01d2, negotiated timeout = 40000

2014-08-21 14:48:54,922 INFO [org.apache.zookeeper.ZooKeeper] - Session: 0x174786daf57f01d2 closed

2014-08-21 14:48:54,922 INFO [org.apache.zookeeper.ClientCnxn] - EventThread shut down

2014-08-21 14:48:55,430 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=zk2:2181,zk1:2181,zk3:2181 sessionTimeout=180000 watcher=catalogtracker-on-hconnection-0x7f120aa6, quorum=zk2:2181,zk1:2181,zk3:2181, baseZNode=/hbase

2014-08-21 14:48:55,431 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=catalogtracker-on-hconnection-0x7f120aa6 connecting to ZooKeeper ensemble=zk2:2181,zk1:2181,zk3:2181

2014-08-21 14:48:55,432 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server zk1/192.168.5.121:2181. Will not attempt to authenticate using SASL (unknown error)

2014-08-21 14:48:55,433 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to zk1/192.168.5.121:2181, initiating session

2014-08-21 14:48:55,439 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server zk1/192.168.5.121:2181, sessionid = 0x15478ef8bf2f0170, negotiated timeout = 40000

2014-08-21 14:48:55,456 INFO [org.apache.zookeeper.ZooKeeper] - Session: 0x15478ef8bf2f0170 closed

2014-08-21 14:48:55,457 INFO [org.apache.zookeeper.ClientCnxn] - EventThread shut down

2014-08-21 14:48:55,459 INFO [org.apache.hadoop.hbase.client.HBaseAdmin] - Disabled _mrtest1

2014-08-21 14:48:55,638 INFO [org.apache.hadoop.hbase.client.HBaseAdmin] - Deleted _mrtest1

2014-08-21 14:48:55,638 INFO [org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation] - Closing master protocol: MasterService

2014-08-21 14:48:55,638 INFO [org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation] - Closing zookeeper sessionid=0x1647ed72def6009d

2014-08-21 14:48:55,658 INFO [org.apache.zookeeper.ZooKeeper] - Session: 0x1647ed72def6009d closed

2014-08-21 14:48:55,659 INFO [org.apache.zookeeper.ClientCnxn] - EventThread shut down

2014-08-21 14:48:55,824 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=zk2:2181,zk1:2181,zk3:2181 sessionTimeout=180000 watcher=hconnection-0x30dd1e89, quorum=zk2:2181,zk1:2181,zk3:2181, baseZNode=/hbase

2014-08-21 14:48:55,826 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=hconnection-0x30dd1e89 connecting to ZooKeeper ensemble=zk2:2181,zk1:2181,zk3:2181

2014-08-21 14:48:55,826 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server zk2/192.168.5.122:2181. Will not attempt to authenticate using SASL (unknown error)

2014-08-21 14:48:55,836 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to zk2/192.168.5.122:2181, initiating session

2014-08-21 14:48:55,855 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server zk2/192.168.5.122:2181, sessionid = 0x1647ed72def6009f, negotiated timeout = 40000

2014-08-21 14:48:56,473 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=zk2:2181,zk1:2181,zk3:2181 sessionTimeout=180000 watcher=catalogtracker-on-hconnection-0x30dd1e89, quorum=zk2:2181,zk1:2181,zk3:2181, baseZNode=/hbase

2014-08-21 14:48:56,474 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=catalogtracker-on-hconnection-0x30dd1e89 connecting to ZooKeeper ensemble=zk2:2181,zk1:2181,zk3:2181

2014-08-21 14:48:56,475 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server zk2/192.168.5.122:2181. Will not attempt to authenticate using SASL (unknown error)

2014-08-21 14:48:56,476 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to zk2/192.168.5.122:2181, initiating session

2014-08-21 14:48:56,482 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server zk2/192.168.5.122:2181, sessionid = 0x1647ed72def600a0, negotiated timeout = 40000

2014-08-21 14:48:56,500 INFO [org.apache.zookeeper.ZooKeeper] - Session: 0x1647ed72def600a0 closed

2014-08-21 14:48:56,501 INFO [org.apache.zookeeper.ClientCnxn] - EventThread shut down

2014-08-21 14:48:56,502 INFO [org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation] - Closing master protocol: MasterService

2014-08-21 14:48:56,502 INFO [org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation] - Closing zookeeper sessionid=0x1647ed72def6009f

2014-08-21 14:48:56,506 INFO [org.apache.zookeeper.ZooKeeper] - Session: 0x1647ed72def6009f closed

2014-08-21 14:48:56,507 INFO [org.apache.zookeeper.ClientCnxn] - EventThread shut down

2014-08-21 14:48:56,508 INFO [root] - Table _mrtest1 created!

2014-08-21 14:48:56,909 INFO [org.apache.zookeeper.ZooKeeper] - Initiating client connection, connectString=zk2:2181,zk1:2181,zk3:2181 sessionTimeout=90000 watcher=hconnection-0x6a86ad83, quorum=zk2:2181,zk1:2181,zk3:2181, baseZNode=/hbase

2014-08-21 14:48:56,911 INFO [org.apache.zookeeper.ClientCnxn] - Opening socket connection to server zk3/192.168.5.123:2181. Will not attempt to authenticate using SASL (unknown error)

2014-08-21 14:48:56,912 INFO [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] - Process identifier=hconnection-0x6a86ad83 connecting to ZooKeeper ensemble=zk2:2181,zk1:2181,zk3:2181

2014-08-21 14:48:56,915 INFO [org.apache.zookeeper.ClientCnxn] - Socket connection established to zk3/192.168.5.123:2181, initiating session

2014-08-21 14:48:56,922 INFO [org.apache.zookeeper.ClientCnxn] - Session establishment complete on server zk3/192.168.5.123:2181, sessionid = 0x174786daf57f01d3, negotiated timeout = 40000

2014-08-21 14:48:56,965 INFO [org.apache.hadoop.hbase.mapreduce.TableOutputFormat] - Created table instance for _mrtest1

2014-08-21 14:49:07,111 INFO [org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider] - Failing over to rm2

2014-08-21 14:49:12,459 WARN [org.apache.hadoop.mapreduce.JobSubmitter] - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.

2014-08-21 14:49:25,381 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn1/192.168.5.111 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2: Name or service not known]; remaining name '111.5.168.192.in-addr.arpa'

2014-08-21 14:49:25,385 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn6/192.168.5.116 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '116.5.168.192.in-addr.arpa'

2014-08-21 14:49:25,386 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn3/192.168.5.113 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '113.5.168.192.in-addr.arpa'

2014-08-21 14:49:25,387 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn4/192.168.5.114 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '114.5.168.192.in-addr.arpa'

2014-08-21 14:49:25,387 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn4/192.168.5.114 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '114.5.168.192.in-addr.arpa'

2014-08-21 14:49:25,388 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn1/192.168.5.111 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '111.5.168.192.in-addr.arpa'

2014-08-21 14:49:25,389 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn6/192.168.5.116 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '116.5.168.192.in-addr.arpa'

2014-08-21 14:49:25,389 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn4/192.168.5.114 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '114.5.168.192.in-addr.arpa'

2014-08-21 14:49:25,390 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn5/192.168.5.115 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '115.5.168.192.in-addr.arpa'

2014-08-21 14:49:25,390 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn6/192.168.5.116 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '116.5.168.192.in-addr.arpa'

2014-08-21 14:49:25,413 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn5/192.168.5.115 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '115.5.168.192.in-addr.arpa'

2014-08-21 14:49:25,414 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn2/192.168.5.112 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '112.5.168.192.in-addr.arpa'

2014-08-21 14:49:25,415 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn5/192.168.5.115 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '115.5.168.192.in-addr.arpa'

2014-08-21 14:49:25,415 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn2/192.168.5.112 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '112.5.168.192.in-addr.arpa'

2014-08-21 14:49:25,416 ERROR [org.apache.hadoop.hbase.mapreduce.TableInputFormatBase] - Cannot resolve the host name for dn3/192.168.5.113 because of javax.naming.ConfigurationException: Unknown DNS server: nn1,nn2 [Root exception is java.net.UnknownHostException: nn1,nn2]; remaining name '113.5.168.192.in-addr.arpa'

2014-08-21 14:49:25,595 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - number of splits:15

2014-08-21 14:49:25,925 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - Submitting tokens for job: job_1408378794442_0015

2014-08-21 14:49:26,170 INFO [org.apache.hadoop.yarn.client.api.impl.YarnClientImpl] - Submitted application application_1408378794442_0015

2014-08-21 14:49:26,219 INFO [org.apache.hadoop.mapreduce.Job] - The url to track the job: http://rm2:23188/proxy/application_1408378794442_0015/

2014-08-21 14:49:26,220 INFO [org.apache.hadoop.mapreduce.Job] - Running job: job_1408378794442_0015

2014-08-21 14:49:38,592 INFO [org.apache.hadoop.mapreduce.Job] - Job job_1408378794442_0015 running in uber mode : false

2014-08-21 14:49:38,594 INFO [org.apache.hadoop.mapreduce.Job] -  map 0% reduce 0%

2014-08-21 14:50:40,813 INFO [org.apache.hadoop.mapreduce.Job] -  map 7% reduce 0%

2014-08-21 14:50:47,911 INFO [org.apache.hadoop.mapreduce.Job] -  map 11% reduce 0%

2014-08-21 14:50:49,945 INFO [org.apache.hadoop.mapreduce.Job] -  map 13% reduce 0%

2014-08-21 14:51:03,145 INFO [org.apache.hadoop.mapreduce.Job] -  map 20% reduce 0%

2014-08-21 14:51:04,159 INFO [org.apache.hadoop.mapreduce.Job] -  map 20% reduce 4%

2014-08-21 14:51:10,265 INFO [org.apache.hadoop.mapreduce.Job] -  map 20% reduce 7%

2014-08-21 14:51:28,566 INFO [org.apache.hadoop.mapreduce.Job] -  map 24% reduce 7%

2014-08-21 14:51:30,598 INFO [org.apache.hadoop.mapreduce.Job] -  map 33% reduce 7%

2014-08-21 14:51:31,613 INFO [org.apache.hadoop.mapreduce.Job] -  map 33% reduce 9%

2014-08-21 14:51:34,656 INFO [org.apache.hadoop.mapreduce.Job] -  map 33% reduce 11%

2014-08-21 14:51:42,840 INFO [org.apache.hadoop.mapreduce.Job] -  map 40% reduce 11%

2014-08-21 14:51:46,912 INFO [org.apache.hadoop.mapreduce.Job] -  map 40% reduce 13%

2014-08-21 14:51:55,050 INFO [org.apache.hadoop.mapreduce.Job] -  map 47% reduce 13%

2014-08-21 14:51:56,074 INFO [org.apache.hadoop.mapreduce.Job] -  map 47% reduce 16%

2014-08-21 14:52:03,177 INFO [org.apache.hadoop.mapreduce.Job] -  map 53% reduce 16%

2014-08-21 14:52:08,241 INFO [org.apache.hadoop.mapreduce.Job] -  map 53% reduce 18%

2014-08-21 14:52:12,303 INFO [org.apache.hadoop.mapreduce.Job] -  map 60% reduce 18%

2014-08-21 14:52:14,343 INFO [org.apache.hadoop.mapreduce.Job] -  map 60% reduce 20%

2014-08-21 14:52:26,528 INFO [org.apache.hadoop.mapreduce.Job] -  map 64% reduce 20%

2014-08-21 14:52:27,551 INFO [org.apache.hadoop.mapreduce.Job] -  map 73% reduce 20%

2014-08-21 14:52:29,581 INFO [org.apache.hadoop.mapreduce.Job] -  map 73% reduce 22%

2014-08-21 14:52:32,621 INFO [org.apache.hadoop.mapreduce.Job] -  map 73% reduce 24%

2014-08-21 14:52:44,811 INFO [org.apache.hadoop.mapreduce.Job] -  map 80% reduce 24%

2014-08-21 14:52:47,848 INFO [org.apache.hadoop.mapreduce.Job] -  map 80% reduce 27%

2014-08-21 14:52:56,950 INFO [org.apache.hadoop.mapreduce.Job] -  map 84% reduce 27%

2014-08-21 14:52:57,966 INFO [org.apache.hadoop.mapreduce.Job] -  map 87% reduce 27%

2014-08-21 14:52:59,997 INFO [org.apache.hadoop.mapreduce.Job] -  map 87% reduce 29%

2014-08-21 14:53:42,487 INFO [org.apache.hadoop.mapreduce.Job] -  map 91% reduce 29%

2014-08-21 14:53:43,506 INFO [org.apache.hadoop.mapreduce.Job] -  map 93% reduce 29%

2014-08-21 14:53:45,540 INFO [org.apache.hadoop.mapreduce.Job] -  map 93% reduce 31%

2014-08-21 14:53:47,575 INFO [org.apache.hadoop.mapreduce.Job] -  map 100% reduce 31%

2014-08-21 14:53:48,584 INFO [org.apache.hadoop.mapreduce.Job] -  map 100% reduce 33%

2014-08-21 14:53:51,611 INFO [org.apache.hadoop.mapreduce.Job] -  map 100% reduce 44%

2014-08-21 14:53:54,642 INFO [org.apache.hadoop.mapreduce.Job] -  map 100% reduce 55%

2014-08-21 14:53:57,679 INFO [org.apache.hadoop.mapreduce.Job] -  map 100% reduce 66%

2014-08-21 14:54:00,710 INFO [org.apache.hadoop.mapreduce.Job] -  map 100% reduce 71%

2014-08-21 14:54:03,749 INFO [org.apache.hadoop.mapreduce.Job] -  map 100% reduce 79%

2014-08-21 14:54:06,791 INFO [org.apache.hadoop.mapreduce.Job] -  map 100% reduce 88%

2014-08-21 14:54:09,829 INFO [org.apache.hadoop.mapreduce.Job] -  map 100% reduce 96%

2014-08-21 14:54:11,854 INFO [org.apache.hadoop.mapreduce.Job] -  map 100% reduce 100%

2014-08-21 14:54:11,873 INFO [org.apache.hadoop.mapreduce.Job] - Job job_1408378794442_0015 completed successfully

2014-08-21 14:54:12,082 INFO [org.apache.hadoop.mapreduce.Job] - Counters: 61

	File System Counters

		FILE: Number of bytes read=324596614

		FILE: Number of bytes written=651096206

		FILE: Number of read operations=0

		FILE: Number of large read operations=0

		FILE: Number of write operations=0

		HDFS: Number of bytes read=1105

		HDFS: Number of bytes written=0

		HDFS: Number of read operations=15

		HDFS: Number of large read operations=0

		HDFS: Number of write operations=0

	Job Counters

		Killed map tasks=6

		Launched map tasks=21

		Launched reduce tasks=1

		Data-local map tasks=15

		Rack-local map tasks=6

		Total time spent by all maps in occupied slots (ms)=5299964

		Total time spent by all reduces in occupied slots (ms)=413798

		Total time spent by all map tasks (ms)=2649982

		Total time spent by all reduce tasks (ms)=206899

		Total vcore-seconds taken by all map tasks=2649982

		Total vcore-seconds taken by all reduce tasks=206899

		Total megabyte-seconds taken by all map tasks=4070372352

		Total megabyte-seconds taken by all reduce tasks=423729152

	Map-Reduce Framework

		Map input records=14112896

		Map output records=14112896

		Map output bytes=296370816

		Map output materialized bytes=324596698

		Input split bytes=1105

		Combine input records=0

		Combine output records=0

		Reduce input groups=1

		Reduce shuffle bytes=324596698

		Reduce input records=14112896

		Reduce output records=1

		Spilled Records=28225792

		Shuffled Maps =15

		Failed Shuffles=0

		Merged Map outputs=15

		GC time elapsed (ms)=13168

		CPU time spent (ms)=403830

		Physical memory (bytes) snapshot=6978674688

		Virtual memory (bytes) snapshot=29039456256

		Total committed heap usage (bytes)=4911529984

	HBase Counters

		BYTES_IN_REMOTE_RESULTS=194636060

		BYTES_IN_RESULTS=5490524772

		MILLIS_BETWEEN_NEXTS=1834495

		NOT_SERVING_REGION_EXCEPTION=0

		NUM_SCANNER_RESTARTS=0

		REGIONS_SCANNED=15

		REMOTE_RPC_CALLS=5050

		REMOTE_RPC_RETRIES=0

		RPC_CALLS=141167

		RPC_RETRIES=0

	Shuffle Errors

		BAD_ID=0

		CONNECTION=0

		IO_ERROR=0

		WRONG_LENGTH=0

		WRONG_MAP=0

		WRONG_REDUCE=0

	File Input Format Counters

		Bytes Read=0

	File Output Format Counters

		Bytes Written=0

*** measure elapsed time,stoped at 2014-08-21 14:54:12:089

time elapsed :0days 0hours 5minutes 36seconds 209milliseconds[336.209seconds]

巴特西

基于HBase Hadoop 分布式集群环境下的MapReduce程序开发

最新文章

热门文章