hadoop的lzo支持

1、下载相关文件：
2、Configure LZO to build a shared library (required) and use a package-specific prefix (optional but recommended):
3、Build and install LZO:
4、安装 hadoop-lzo
5、配置 Hadoop 属性
6、测试
7、LZO 创建索引

环境为伪分布。

1、下载相关文件：

lzo-2.10.tar.gz：https://www.oberhumer.com/opensource/lzo/

hadoop-lzo-master.zip：https://github.com/twitter/hadoop-lzo/archive/master.zip

2、Configure LZO to build a shared library (required) and use a package-specific prefix (optional but recommended):

[root@zgg opt]# tar -zxvf lzo-2.10.tar.gz

....

[root@zgg opt]# cd lzo-2.10

[root@zgg lzo-2.10]# ./configure --enable-shared --prefix /usr/local/lzo-2.10

3、Build and install LZO:

[root@zgg lzo-2.10]# make && sudo make install

如果是集群环境，编译完 lzo 包之后，将 /usr/local/lzo-2.10目录下生成的所有文件打包，并同步到集群其他节点。

4、安装 hadoop-lzo

[root@zgg opt]# unzip hadoop-lzo-master.zip

....

[root@zgg opt]# vi /etc/profile

....

export C_INCLUDE_PATH=/usr/local/lzo-2.10/include

export LIBRARY_PATH=/usr/local/lzo-2.10/lib

....

[root@zgg opt]# source /etc/profile

[root@zgg opt]# cd hadoop-lzo-master

[root@zgg hadoop-lzo-master]# mvn clean package

....

[INFO] ------------------------------------------------------------------------

[INFO] BUILD SUCCESS

[INFO] ------------------------------------------------------------------------

[INFO] Total time:  16:46 min

[INFO] Finished at: 2021-01-10T14:08:16+08:00

[INFO] ------------------------------------------------------------------------

[root@zgg hadoop-lzo-master]# cd target/

[root@zgg target]# ls

antrun   generated-sources                       hadoop-lzo-0.4.21-SNAPSHOT-sources.jar  native

apidocs  hadoop-lzo-0.4.21-SNAPSHOT.jar          javadoc-bundle-options                  test-classes

classes  hadoop-lzo-0.4.21-SNAPSHOT-javadoc.jar  maven-archiver

# 将`hadoop-lzo-0.4.21-SNAPSHOT.jar`复制到 .../common 目录下

[root@zgg hadoop-lzo-master]# cp target/hadoop-lzo-0.4.21-SNAPSHOT.jar /opt/hadoop-3.2.1/share/hadoop/common

如果是集群环境，需要将hadoop-lzo-0.4.21-SNAPSHOT.jar同步到集群其他节点。

5、配置 Hadoop 属性

hadoop-env.sh：

export LD_LIBRARY_PATH=/usr/local/lzo-2.10/lib

core-site.xml

<property>

	<!-- 配置支持 LZO 压缩 -->

	<name>io.compression.codecs</name>

	<value>

		com.hadoop.compression.lzo.LzoCodec,

		com.hadoop.compression.lzo.LzopCodec

	</value>

</property>

<property>

 	<name>io.compression.codec.lzo.class</name>

 	<value>com.hadoop.compression.lzo.LzoCodec</value>

</property>

mapred-site.xml

<property>

	<!-- 启用map任务输出的压缩 -->

    <name>mapreduce.map.output.compress</name>

    <value>true</value>

</property>

<property>

	<!-- map任务输出的压缩类型 -->

    <name>mapred.map.output.compression.codec</name>

    <value>com.hadoop.compression.lzo.LzopCodec</value>

</property>

<property>

	<!-- 启用job输出的压缩 -->

    <name>mapreduce.output.fileoutputformat.compress</name>

    <value>true</value>

</property>

<property>

	<!-- job输出的压缩类型，这里是LzopCodec -->

    <name>mapreduce.output.fileoutputformat.compress.codec</name>

    <value>com.hadoop.compression.lzo.LzopCodec</value>

</property>

<property>

    <name>mapred.child.env</name>

    <value>LD_LIBRARY_PATH=/usr/local/lzo-2.10/lib</value>

</property>

如果是集群环境，需要将这些配置同步到集群其他节点。

6、测试

# 安装lzop

yum install lzop

# 压缩文件

lzop  wc.txt

# 测试wordcount

[root@zgg target]# hadoop jar /opt/hadoop-3.2.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.1.jar wordcount /in/wc.txt.lzo /out/wc

....

【一个分片，未切片】

2021-01-10 15:54:36,249 INFO mapreduce.JobSubmitter: number of splits:1

[root@zgg target]# hadoop fs -ls /out/wc

Found 2 items

-rw-r--r--   1 root supergroup          0 2021-01-10 16:36 /out/wc/_SUCCESS

-rw-r--r--   1 root supergroup         91 2021-01-10 16:36 /out/wc/part-r-00000.lzo

7、LZO 创建索引

LZO 压缩文件的可切片特性依赖于其索引，故我们需要手动为 LZO 压缩文件创建索引。若无索引，则 LZO 文件的切片只有一个。

# 数据文件的目录是hdfs上的目录

# 【com.hadoop.compression.lzo.DistributedLzoIndexer】

[root@zgg target]# hadoop jar /opt/hadoop-lzo-master/target/hadoop-lzo-0.4.21-SNAPSHOT.jar com.hadoop.compression.lzo.LzoIndexer /in/wc.txt.lzo

2021-01-10 16:41:23,817 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library from the embedded binaries

2021-01-10 16:41:23,820 INFO lzo.LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev 5dbdddb8cfb544e58b4e0b9664b9d1b66657faf5]

2021-01-10 16:41:24,573 INFO lzo.LzoIndexer: [INDEX] LZO Indexing file /in/wc.txt.lzo, size 0.00 GB...

2021-01-10 16:41:24,659 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false

2021-01-10 16:41:24,736 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false

2021-01-10 16:41:24,802 INFO lzo.LzoIndexer: Completed LZO Indexing in 0.23 seconds (0.00 MB/s).  Index size is 0.01 KB.

# 查看

[root@zgg data]# hadoop fs -ls /in

Found 2 items

-rw-r--r--   1 root supergroup        124 2021-01-10 16:13 /in/lzo/wc.txt.lzo

-rw-r--r--   1 root supergroup          8 2021-01-10 16:13 /in/lzo/wc.txt.lzo.index

# 测试

# 【输入路径也必须包含索引文件】

[root@zgg target]# hadoop jar /opt/hadoop-3.2.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.1.jar wordcount /in/lzo /out/wc

....

2021-01-10 16:43:41,854 INFO mapreduce.JobSubmitter: number of splits:2

....

[root@zgg target]# hadoop fs -ls /out/wc

Found 2 items

-rw-r--r--   1 root supergroup          0 2021-01-10 16:44 /out/wc/_SUCCESS

-rw-r--r--   1 root supergroup        102 2021-01-10 16:44 /out/wc/part-r-00000.lzo

[root@zgg target]# hadoop fs -text /out/wc/part-r-00000.lzo

2021-01-10 16:44:47,453 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false

2021-01-10 16:44:47,521 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library from the embedded binaries

2021-01-10 16:44:47,557 INFO lzo.LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev 5dbdddb8cfb544e58b4e0b9664b9d1b66657faf5]

2021-01-10 16:44:47,562 INFO compress.CodecPool: Got brand-new decompressor [.lzo]

,       1

flink   170

hadoop  510

hello   340

spark   170

参考地址：

https://www.cnblogs.com/caoshouling/p/14091113.html

https://github.com/twitter/hadoop-lzo

巴特西