elk logstash Managing Multiline Events

1.Java程序的日志特征，logstash 正为此准备好了 codec/multiline 插件！

有时候应用程序会抛异常，就存在着如何合并多行信息的问题，我这里做的配置就是如果当前行是以‘空格’，‘字母‘ 和 ‘-’开头的，那么就直接合并到上

[2019-06-26 11:59:42,758[ INFO ThreadPool 任务运行队列 thread:2](cn.com.to1.common.dac.SupportDAC:575) prepared的查询语句是：SELECT COUNT(DISTINCTr.user_id) FROM TB_DEPARTMENT_INFO d, tb_qy_user_department_ref r where r.department_id = d.id and d.org_id = :orgId AND (d.dept_full_name LIKE :deptFullNameLike or d.dept_full_name = :deptFullName)

[2019-06-26 11:59:42,763[ INFO ThreadPool 任务运行队列 thread:8](cn.com.to1.common.dac.SupportDAC:575) prepared的查询语句是：insert into TB_QY_USER_DEPARTMENT_REF(ID,USER_ID,DEPARTMENT_ID,SORT,ORG_ID) values(:id,:userId,:departmentId,:sort,:orgId)

2019-06-26 11:59:42,764[ERROR DefaultQuartzScheduler_Worker-10](cn.com.to1.component.util.EmailUtil:225) EmailUtil sendWarnEmail error

javax.mail.AuthenticationFailedException

        at javax.mail.Service.connect(Service.java:319)

        at javax.mail.Service.connect(Service.java:169)

        at javax.mail.Service.connect(Service.java:118)

        at javax.mail.Transport.send0(Transport.java:188)

        at javax.mail.Transport.send(Transport.java:118)

        at cn.com.do1.component.util.EmailUtil.send(EmailUtil.java:191)

        at cn.com.do1.component.util.EmailUtil.sendWarnEmail(EmailUtil.java:217)

        at cn.com.do1.component.util.ThreadPoolUtils.warn(ThreadPoolUtils.java:278)

        at cn.com.do1.component.util.ThreadPoolUtils.execute(ThreadPoolUtils.java:127)

        at cn.com.do1.component.runtask.thread.RunTaskJob.runTask(RunTaskJob.java:63)

        at cn.com.do1.component.runtask.thread.RunTaskJob.execute(RunTaskJob.java:52)

        at org.quartz.core.JobRunShell.run(JobRunShell.java:202)

        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)

[2019-06-26 11:59:42,764[ INFO ThreadPool 任务运行队列 thread:8](cn.com.do1.common.dac.SupportDAC:575) prepared的查询语句是：insert into TB_QY_USER_INFO(ID,USER_ID,ORG_ID,PERSON_NAME,PINYIN,SEX,UPDATE_TIME,USER_STATUS,HEAD_PIC,CREATE_TIME,IS_CONCERNED,WX_USER_ID,CORP_ID) values(:id,:userId,:orgId,:personName,:pinyin,:sex,:updateTime,:userStatus,:headPic,:createTime,:isConcerned,:wxUserId,:corpId)

2.logstash的配置关键

合并多行数据(Multiline)

demo

[Aug/08/08 14:54:03] hello world

[Aug/08/09 14:54:04] hello logstash

    hello best practice

    hello raochenlin

[Aug/08/10 14:54:05] the end

logstash的关键配置

[root@VM_0_92_centos opt]# cat /usr/local/logstash/config/exception.conf

input {

  file {

    path => "/opt/50910626.log"

    codec => multiline {

      pattern => "^\["

      negate => true

      what => "previous"

    }

    start_position => "beginning"

  }

}

output {

  elasticsearch { hosts => ["10.0.0.92:9200"]

                  index => "linemuti"

  }

  stdout { codec => rubydebug }

}
说明：匹配以“[”开头的行，如果不是，那肯定是属于前一行的。

参考：https://doc.yonyoucloud.com/doc/logstash-best-practice-cn/codec/multiline.html

巴特西

elk logstash Managing Multiline Events

最新文章

热门文章