Sqoop Import HDFS

Sqoop import应用场景——密码访问

　注：测试用表为本地数据库中的表

1.明码访问

sqoop list-databases \

 --connect jdbc:mysql://202.193.60.117/dataweb \

 --username root \

 --password

2.交互式密码

sqoop list-databases \

--connect jdbc:mysql://202.193.60.117/dataweb \

 --username root \

 --P

3.文件授权密码

sqoop list-databases \

 --connect jdbc:mysql://202.193.60.117/dataweb \

 --username root \

 --password-file /usr/hadoop/.password

　　在运行之前先要在指定路径下创建.password文件。

[hadoop@centpy ~]$ cd /usr/hadoop/

[hadoop@centpy hadoop]$ ls

flume  hadoop-2.6.  sqoop

[hadoop@centpy hadoop]$ echo -n "20134997" > .password

[hadoop@centpy hadoop]$ ls -a

.  ..  flume  hadoop-2.6.  .password  sqoop

[hadoop@centpy hadoop]$ more .password 

[hadoop@centpy hadoop]$ chmod 400 .password //根据官方文档说明赋予400权限

　　测试运行之后一定会报以下错误：

// :: WARN tool.BaseSqoopTool: Failed to load password file

java.io.IOException: The provided password file /usr/hadoop/.password does not exist!

    at org.apache.sqoop.util.password.FilePasswordLoader.verifyPath(FilePasswordLoader.java:)

    at org.apache.sqoop.util.password.FilePasswordLoader.loadPassword(FilePasswordLoader.java:)

    at org.apache.sqoop.util.CredentialsUtil.fetchPasswordFromLoader(CredentialsUtil.java:)

    at org.apache.sqoop.util.CredentialsUtil.fetchPassword(CredentialsUtil.java:)

    at org.apache.sqoop.tool.BaseSqoopTool.applyCredentialsOptions(BaseSqoopTool.java:)

    at org.apache.sqoop.tool.BaseSqoopTool.applyCommonOptions(BaseSqoopTool.java:)

    at org.apache.sqoop.tool.ListDatabasesTool.applyOptions(ListDatabasesTool.java:)

    at org.apache.sqoop.tool.SqoopTool.parseArguments(SqoopTool.java:)

    at org.apache.sqoop.Sqoop.run(Sqoop.java:)

    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:)

    at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:)

    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:)

    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:)

    at org.apache.sqoop.Sqoop.main(Sqoop.java:)

Error while loading password file: The provided password file /usr/hadoop/.password does not exist!

　　为了解决该错误，我们需要将.password文件放到HDFS上面去，这样就能找到该文件了。

[hadoop@centpy hadoop]$ hdfs dfs -ls /

Found  items

drwxr-xr-x   - Zimo   supergroup           -- : /actor

drwxr-xr-x   - Zimo   supergroup           -- : /counter

drwxr-xr-x   - hadoop supergroup           -- : /flume

drwxr-xr-x   - hadoop hadoop               -- : /hdfsOutput

drwxr-xr-x   - Zimo   supergroup           -- : /join

drwxr-xr-x   - hadoop supergroup           -- : /maven

drwxr-xr-x   - Zimo   supergroup           -- : /mergeSmallFiles

drwxrwxrwx   - hadoop supergroup           -- : /phone

drwxr-xr-x   - hadoop hadoop               -- : /test

drwx------   - hadoop hadoop               -- : /tmp

drwxr-xr-x   - hadoop hadoop               -- : /weather

drwxr-xr-x   - hadoop hadoop               -- : /weibo

[hadoop@centpy hadoop]$ hdfs dfs -mkdir -p /user/hadoop

[hadoop@centpy hadoop]$ hdfs dfs -put .password /user/hadoop

[hadoop@centpy hadoop]$ hdfs dfs -chmod 400 /user/hadoop/.password

　　现在测试运行一下，注意路径改为HDFS上的/user/hadoop。

[hadoop@centpy hadoop-2.6.]$ sqoop list-databases  --connect jdbc:mysql://202.193.60.117/dataweb  --username root  --password-file /user/hadoop/.password

Warning: /usr/hadoop/sqoop/../hbase does not exist! HBase imports will fail.

Please set $HBASE_HOME to the root of your HBase installation.

Warning: /usr/hadoop/sqoop/../hcatalog does not exist! HCatalog jobs will fail.

Please set $HCAT_HOME to the root of your HCatalog installation.

Warning: /usr/hadoop/sqoop/../accumulo does not exist! Accumulo imports will fail.

Please set $ACCUMULO_HOME to the root of your Accumulo installation.

Warning: /usr/hadoop/sqoop/../zookeeper does not exist! Accumulo imports will fail.

Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.

// :: INFO sqoop.Sqoop: Running Sqoop version: 1.4.

// :: INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.

information_schema

dataweb

mysql

performance_schema

test

　　可以看到成功了。

Sqoop import应用场景——导入全表

1.不指定目录

sqoop import \

 --connect jdbc:mysql://202.193.60.117/dataweb \

 --username root \

 --password-file /user/hadoop/.password \
 --table user_info

　运行过程如下

// :: INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:

// :: INFO db.DBInputFormat: Using read commited transaction isolation

// :: INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(`id`), MAX(`id`) FROM `user_info`

// :: INFO mapreduce.JobSubmitter: number of splits:

// :: INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1529567189245_0001

// :: INFO impl.YarnClientImpl: Submitted application application_1529567189245_0001

// :: INFO mapreduce.Job: The url to track the job: http://centpy:8088/proxy/application_1529567189245_0001/

// :: INFO mapreduce.Job: Running job: job_1529567189245_0001

// :: INFO mapreduce.Job: Job job_1529567189245_0001 running in uber mode : false

// :: INFO mapreduce.Job:  map % reduce %

// :: INFO mapreduce.Job:  map % reduce %

// :: INFO mapreduce.Job:  map % reduce %

// :: INFO mapreduce.Job:  map % reduce %

// :: INFO mapreduce.Job: Job job_1529567189245_0001 completed successfully

// :: INFO mapreduce.Job: Counters:

    File System Counters

        FILE: Number of bytes read=

        FILE: Number of bytes written=

        FILE: Number of read operations=

        FILE: Number of large read operations=

        FILE: Number of write operations=

        HDFS: Number of bytes read=

        HDFS: Number of bytes written=

        HDFS: Number of read operations=

        HDFS: Number of large read operations=

        HDFS: Number of write operations=

    Job Counters

        Launched map tasks=

        Other local map tasks=

        Total time spent by all maps in occupied slots (ms)=

        Total time spent by all reduces in occupied slots (ms)=

        Total time spent by all map tasks (ms)=

        Total vcore-seconds taken by all map tasks=

        Total megabyte-seconds taken by all map tasks=

    Map-Reduce Framework

        Map input records=

        Map output records=

        Input split bytes=

        Spilled Records=

        Failed Shuffles=

        Merged Map outputs=

        GC time elapsed (ms)=

        CPU time spent (ms)=

        Physical memory (bytes) snapshot=

        Virtual memory (bytes) snapshot=

        Total committed heap usage (bytes)=

    File Input Format Counters

        Bytes Read=

    File Output Format Counters

        Bytes Written=

// :: INFO mapreduce.ImportJobBase: Transferred  bytes in 54.3141 seconds (0.8101 bytes/sec)

// :: INFO mapreduce.ImportJobBase: Retrieved  records.

　　再查看一下HDFS下的运行结果

[hadoop@centpy hadoop-2.6.]$ hdfs dfs -cat /user/hadoop/user_info/part-m-*

,admin,,

,hello,,

,hahaha,haha,

　　运行结果和数据库内容匹配。

以上就是博主为大家介绍的这一板块的主要内容，这都是博主自己的学习过程，希望能给大家带来一定的指导作用，有用的还望大家点个支持，如果对你没用也望包涵，有错误烦请指出。如有期待可关注博主以第一时间获取更新哦，谢谢！

巴特西

Sqoop Import HDFS

最新文章

热门文章