（未解决）flume监控目录，抓取文件内容推送给kafka，报错
flume监控目录，抓取文件内容推送给kafka，报错：
/export/datas/destFile/220104_YT1013_8c5f13f33c299316c6720cc51f94f7a0_2016101912_318.txt
2019-08-06 23:04:31,434 (pool-3-thread-1) [ERROR - org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:280)] FATAL: Spool Directory source r1: { spoolDir: /export/datas/destFile }: Uncaught exception in SpoolDirectorySource thread. Restart or reconfigure Flume to continue processing.
java.nio.charset.MalformedInputException: Input length = 1
at java.nio.charset.CoderResult.throwException(CoderResult.java:281)
at org.apache.flume.serialization.ResettableFileInputStream.readChar(ResettableFileInputStream.java:283)
at org.apache.flume.serialization.LineDeserializer.readLine(LineDeserializer.java:132)
at org.apache.flume.serialization.LineDeserializer.readEvent(LineDeserializer.java:70)
at org.apache.flume.serialization.LineDeserializer.readEvents(LineDeserializer.java:89)
at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readDeserializerEvents(ReliableSpoolingFileEventReader.java:343)
at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:331)
at org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:250)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
正常处理完文件后，应该删除该文件，应该提示这个信息：2019-08-06 23:04:31,412 (pool-3-thread-1) [INFO - org.apache.flume.client.avro.ReliableSpoolingFileEventReader.deleteCurrentFile(ReliableSpoolingFileEventReader.java:492)] Preparing to delete file
问题原因：文件的字符集编码是UTF-8，要手动设置一下字符集编码。
解决方案：a1.sources.r1.inputCharset = UTF-8 //将server.properties配置文件中的 inputCharset 设置为 UTF-8
还是没用，未解决。
貌似修复了程序的bug，这个问题就没再发生了，不过又有新的问题出现了：
2019-08-07 01:41:45,377 (pool-3-thread-1) [ERROR - org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:280)] FATAL: Spool Directory source r1: { spoolDir: /export/datas/destFile }: Uncaught exception in SpoolDirectorySource thread. Restart or reconfigure Flume to continue processing.
java.lang.IllegalStateException: File has been modified since being read: /export/datas/destFile/220605_YT1013_ba701bb6bf74afb2f14a92de320bc023_2016101808_113.txt
at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.retireCurrentFile(ReliableSpoolingFileEventReader.java:406)
at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:326)
at org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:250)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
^C2019-08-07 01:42:11,310 (agent-shutdown-hook) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.stop(LifecycleSupervisor.java:78)] Stopping lifecycle supervisor 10
2019-08-07 01:42:11,320 (agent-shutdown-hook) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:149)] Component type: CHANNEL, name: c1 stopped
最新文章
- UDP server & client
- Ring buffers and queues
- js中的遍历foreach，$.each()，$().each()
- Bootstrap系列 -- 9. 表格
- Hibernate查询出现java.lang.IllegalArgumentException异常解决方法
- POJ 1716
- 解决JSP页面图片缓存问题
- 多线程12-CyclicBarrier、CountDownLatch、Exchanger
- linux下制作共享库.a和 .so
- python单/双下划线使用
- informix 常见系统表
- cnblog测试
- Confluence 6 代理和 HTTPS 详细配置指南链接
- MINIUI后台获取数据
- 为python.exe或者ipython.exe添加环境变量
- [CF1010E]Store[kd-tree]
- Android学习之——如何将GridView内嵌在ScrollView中
- Scrum立会报告+燃尽图 04
- 配置MDM的描述文件
- ul自适应li问题