Flume（三） —— 断点续传 与 事务
2024-09-02 17:11:19
断点续传
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe / configure the source
a1.sources.r1.type = TAILDIR
a1.sources.r1.filegroups = f1 f2
a1.sources.r1.filegroups.f1 = /usr/local/flume/example/file1.txt
a1.sources.r1.filegroups.f2 = /usr/local/flume/example/file2.txt
a1.sources.r1.positionFile = /usr/local/flume/position/postition.json
# Describe the sink
a1.sinks.k1.type = logger
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
执行
./bin/flume-ng agent -c conf/ -f job/file-flume-point-logger.conf -n a1 -Dflume.root.logger=INFO,console
运行结果
原理
在配置的/usr/local/flume/position/postition.json文件中，会发现，flume将位置信息记录在这个postition中。
[{"inode":33855378,"pos":23,"file":"/usr/local/flume/example/file1.txt"},{"inode":33855380,"pos":36,"file":"/usr/local/flume/example/file2.txt"}]
事务
put事务和take事务
最新文章
- python学习笔记整理——dictView [未整理]
- AccessViolationException: 尝试读取或写入受保护的内存。这通常指示其他内存已损坏。
- 【leetcode❤python】171. Excel Sheet Column Number
- js数组去重的三种常用方法总结
- （原创）Windows8下安装配置WAMP
- 简单的web三层架构系统【第三版】
- C语言深度剖析---const关键字（转载）
- hdu1172猜数字
- ADOConnection数据库连接池
- json解包与json封包
- .NetCore上传多文件的几种示例
- bzoj4514 [Sdoi2016]数字配对
- Java中 try--catch-- finally、throw、throws 的用法
- Spring：获取容器中的Bean
- Python基础-python数据类型（四）
- [转][C#]TopSelf
- apache 允许 访问软链接 （ Apache won't follow symlinks (403 Forbidden) ）
- Java NIO ServerSocketChannel
- python 读取文件时报错： UnicodeDecodeError: 'gbk' codec can't decode byte 0xa4 in position 127: illegal multibyte sequence
- Oracle重建临时表空间