logstash5.x安装及简单运用

Logstash requires Java 8. Java 9 is not supported.

1、检测是否安装了java环境

[root@node3 ~]# java -version

java version "1.8.0_144"

Java(TM) SE Runtime Environment (build 1.8.0_144-b01)

Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)

2、安装logstash，这里采用rpm安装

　　https://artifacts.elastic.co/downloads/logstash/logstash-5.6.1.rpm

　　yum install logstash

查看生成了哪些文件，查看logstash的执行文件位置：

/etc/logstash/conf.d

/etc/logstash/jvm.options

/etc/logstash/log4j2.properties

/etc/logstash/logstash.yml

/etc/logstash/startup.options

/usr/share/logstash/CHANGELOG.md

/usr/share/logstash/CONTRIBUTORS

/usr/share/logstash/Gemfile

/usr/share/logstash/Gemfile.jruby-1.9.lock

/usr/share/logstash/LICENSE

/usr/share/logstash/NOTICE.TXT

/usr/share/logstash/bin/cpdump

/usr/share/logstash/bin/ingest-convert.sh

/usr/share/logstash/bin/logstash

/usr/share/logstash/bin/logstash-plugin

/usr/share/logstash/bin/logstash-plugin.bat

/usr/share/logstash/bin/logstash.bat

/usr/share/logstash/bin/logstash.lib.sh

/usr/share/logstash/bin/ruby

/usr/share/logstash/bin/setup.bat

/usr/share/logstash/bin/system-install

/usr/share/logstash/data

　配置文件：

1、配置jvm

/etc/logstash/jvm.options

2、logstash的一些配置

/etc/logstash/logstash.yml

3、环境变量一些的配置

/etc/logstash/startup.options

4、日志与log4j2的配置

/etc/logstash/log4j2.properties

开始第一个任务：

[root@node3 conf.d]# /usr/share/logstash/bin/logstash -e 'input { stdin {} } output { stdout {} }'

WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults

Could not find log4j2 configuration at path //usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console

　提示warning，解决办法：

mkdir -p /usr/share/logstash/config/

ln -s /etc/logstash/* /usr/share/logstash/config

chown -R logstash:logstash /usr/share/logstash/config/

bin/logstash -e 'input { stdin { } } output { stdout {} }'

　如果logstash不适用命令行执行，而是作为一个服务：

　　logstash启动：

　　/etc/init.d/logstash start

　　systemctl start logstash.service

开始编写配置文件进行logstash解析：

1、input插件中file插件的使用

[root@node3 conf.d]# cat file.conf

input {

    file {

        path => ["/var/log/messages"]

        start_position => "beginning"

    }

}

output {

    stdout {

        codec => rubydebug

    }

}

[root@node3 conf.d]# /usr/share/logstash/bin/logstash -f file.conf

　2、多个log日志的输入、

[root@node3 conf.d]# cat file_more_choose.conf

input {

    file {

        path => ["/var/log/messages"]

        start_position => "beginning"

    }

    file {

        path => ["/var/log/elasticsearch/my-elastic.log"]

        start_position => "beginning"

    }

}

output {

    stdout {

        codec => rubydebug

    }

}

[root@node3 conf.d]# /usr/share/logstash/bin/logstash -f file_more_choose.conf

　但是发现只打印出elastic的日志，message的日志没有stdout，收集的日志是增量的，之前收集的日志已经存在sincedb中了，所以会默认从之后开始存

Path of the sincedb database file (keeps track of the current position of monitored log files) that will be written to disk. The default will write sincedb files to <path.data>/plugins/inputs/file NOTE: it must be a file path and not a directory path，这是一段sincedb_path的解释

检查配置文件的语法是否正确：

-t, --config.test_and_exit    Check configuration for valid syntax and then exit.

                                   (default: false)

-r, --config.reload.automatic Monitor configuration changes and reload

                                  whenever it is changed.

                                  NOTE: use SIGHUP to manually reload the config

                                   (default: false)

[root@node3 conf.d]# /usr/share/logstash/bin/logstash -f file.conf -t

Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties

Configuration OK

　3、以elasticsearch插件输出：

input {

    file {

        path => ["/var/log/logstash/logstash-plain.log"]

        start_position => "beginning"

        type => "logstash"

    }

}

output {

    elasticsearch {

        hosts => ["192.168.44.134:9200"]

        index => "logstash-log"

        codec => rubydebug

    }

}

4、根据插件type来定义输出插件：

[root@node3 conf.d]# cat type.conf

input {

    file {

       path  => ["/var/log/logstash/logstash-plain.log"]

       start_position => "beginning"

       type => "logstash_2"

    }

    file {

       path => ["/var/log/messages"]

       start_position => "beginning"

       type => "system"

    }

}

output {

    if [type] == "logstash_2" {

        elasticsearch {

            hosts => ["192.168.44.134:9200"]

            index => "logstash_2"

            codec => rubydebug

        }

    }

    if [type] == "system" {

         stdout {

            codec => rubydebug

         }

    }

}

　现在向messages日志中echo一段话：

echo "`date +%F`" >> /var/log/messages

　然后开始执行：

[root@node3 conf.d]# /usr/share/logstash/bin/logstash -f type.conf

Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties

{

      "@version" => "1",

          "host" => "node3",

          "path" => "/var/log/messages",

    "@timestamp" => 2017-09-20T08:19:05.782Z,

       "message" => "2017-09-20",                这是刚刚echo新增的内容

          "type" => "system"

}

　查看es中的索引是否有生成：

巴特西

logstash5.x安装及简单运用

最新文章

热门文章