spark集群进入 bin 下面目录./spark-shell 出现Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

spark集群启动的时候可以正常，进入 ./spark-shell 就会出现如下错误

配置文件：spark-env.sh

export JAVA_HOME=/usr/java/jdk1.7.0_51
export SCALA_HOME=/home/hadoop/scala-2.11.6

export SPARK_MASTER_IP=master24
export SPARK_MASTER_PORT=17077
export SPARK_MASTER_WEBUI_PORT=18080

export SPARK_WORKER_CORES=1
export SPARK_WORKER_MEMORY=30g
export SPARK_WORKER_WEBUI_PORT=18081
export SPARK_WORKER_INSTANCES=1INFO SparkEnv: Registering BlockManagerMaster

错误信息如下：
15// :: INFO DiskBlockManager: Created local directory at /tmp/spark-local--6f8e

// :: INFO MemoryStore: MemoryStore started with capacity 294.9 MB.

// :: INFO ConnectionManager: Bound socket to port  with id = ConnectionManagerId(server2,)

// :: INFO BlockManagerMaster: Trying to register BlockManager

// :: INFO BlockManagerInfo: Registering block manager server2: with 294.9 MB RAM

// :: INFO BlockManagerMaster: Registered BlockManager

// :: INFO HttpServer: Starting HTTP Server

// :: INFO HttpBroadcast: Broadcast server started at http://192.168.1.24:41483

// :: INFO HttpFileServer: HTTP File server directory is /tmp/spark-524059df-53c2-4df8-a2a0-c76c878a3d94

// :: INFO HttpServer: Starting HTTP Server

// :: INFO SparkUI: Started SparkUI at http://server12:4040

// :: WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

java.lang.IllegalArgumentException: java.net.UnknownHostException: cluster1

    at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:)

    at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:)

    at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:)

    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:)

    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:)

    at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:)

    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:)

    at org.apache.hadoop.fs.FileSystem.access$(FileSystem.java:)

    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:)

    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:)

at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.net.UnknownHostException: cluster1
... 61 more

Spark context available as sc.

scala>

是什么原因呢？

巴特西

spark集群进入 bin 下面目录./spark-shell 出现Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

最新文章

热门文章