Flume使用

avro agent

配置文件

cd $FLUME_HOME/conf

vim avro.conf

    a1.sources = r1

    a1.sinks = k1

    a1.channels = c1

    a1.sources.r1.type = avro

    a1.sources.r1.channels = c1

    a1.sources.r1.bind = dev01

    a1.sources.r1.port = 4411

    a1.sinks.k1.type = logger

    a1.channels.c1.type = memory

    a1.channels.c1.capacity = 1000

    a1.channels.c1.transactionCapacity = 100

    a1.sources.r1.channels = c1

    a1.sinks.k1.channel = c1

启动agent

用以下脚本开启agent客户端，从以下信息中可以看到当前 agent 的 jmx 监控端口号是4154，且绑定4411端口。

flume-ng agent -c . -f /opt/bigdata/flume/conf/avro.conf -n a1 -Dflume.root.logger=INFO,console

    Info: Sourcing environment configuration script /opt/bigdata/flume/conf/flume-env.sh

    Info: Including Hadoop libraries found via (/opt/bigdata/hadoop/bin/hadoop) for HDFS access

    Info: Including HBASE libraries found via (/opt/bigdata/hbase/bin/hbase) for HBASE access

    Info: Including Hive libraries found via () for Hive access

    + exec /opt/soft/jdk1.8.0_144/bin/java -Xms100m -Xmx200m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=4154 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dflume.root.logger=INFO,console -cp '/opt/bigdata/flume/conf:/opt/bigdata/flume/lib/*:/opt/bigdata/hadoop/etc/hadoop:/opt/bigdata/hadoop/share/hadoop/common/lib/*:/opt/bigdata/hadoop/share/hadoop/common/*:/opt/bigdata/hadoop/share/hadoop/hdfs:/opt/bigdata/hadoop/share/hadoop/hdfs/lib/*:/opt/bigdata/hadoop/share/hadoop/hdfs/*:/opt/bigdata/hadoop/share/hadoop/yarn/lib/*:/opt/bigdata/hadoop/share/hadoop/yarn/*:/opt/bigdata/hadoop/share/hadoop/mapreduce/lib/*:/opt/bigdata/hadoop/share/hadoop/mapreduce/*:/opt/bigdat......

    org.apache.flume.source.AvroSource.start(AvroSource.java:234)] Starting Avro source r1: { bindAddress: dev01, port: 4411 }...

    2018-11-02 11:42:26,026 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:119)] Monitored counter group for type: SOURCE, name: r1: Successfully registered new MBean.

    2018-11-02 11:42:26,026 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)] Component type: SOURCE, name: r1 started

    2018-11-02 11:42:26,027 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.source.AvroSource.start(AvroSource.java:259)] Avro source r1 started.

启动文本传输

新开一个终端进行以下动作。

创建一个文件

echo "hello world" > /home/dev/flume/log.00

开启文本传输

flume-ng avro-client --conf conf -H dev01 -p 4411 -F /home/dev/flume/log.00

可以从agent客户端看到文件的内容

2018-11-02 11:48:05,426 (New I/O server boss #9) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0xd39a83d1, /192.168.20.10:46114 => /192.168.20.10:4411] OPEN

2018-11-02 11:48:05,428 (New I/O worker #1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0xd39a83d1, /192.168.20.10:46114 => /192.168.20.10:4411] BOUND: /192.168.20.10:4411

2018-11-02 11:48:05,428 (New I/O worker #1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0xd39a83d1, /192.168.20.10:46114 => /192.168.20.10:4411] CONNECTED: /192.168.20.10:46114

2018-11-02 11:48:05,705 (New I/O worker #1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0xd39a83d1, /192.168.20.10:46114 :> /192.168.20.10:4411] DISCONNECTED

2018-11-02 11:48:05,705 (New I/O worker #1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0xd39a83d1, /192.168.20.10:46114 :> /192.168.20.10:4411] UNBOUND

2018-11-02 11:48:05,706 (New I/O worker #1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0xd39a83d1, /192.168.20.10:46114 :> /192.168.20.10:4411] CLOSED

2018-11-02 11:48:05,706 (New I/O worker #1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.channelClosed(NettyServer.java:209)] Connection to /192.168.20.10:46114 disconnected.

2018-11-02 11:48:07,804 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 68 65 6C 6C 6F 20 77 6F 72 6C 64                hello world }

巴特西

Flume使用

avro agent

配置文件

启动agent

启动文本传输

最新文章

热门文章