logstash5安装并实现mariadb数据写入到elasticsearch

java环境这里默认安装了，一般源码安装，这里就不说了

一、安装logstash

安装logstash可以用yum安装，也可以用源码安装：

yum安装：

1.导入GPG:

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

2.创建repo文件，vim /etc/yum.repos.d/logstash.repo

[logstash-.x]

name=Elastic repository for .x packages

baseurl=https://artifacts.elastic.co/packages/5.x/yum

gpgcheck=

gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch

enabled=

autorefresh=

type=rpm-md

3.安装logstash

yum install logstash

二、源码安装（推荐）

1.官网下载需要的稳定版本

2.解压：

unzip logstash-5.5..zip

3.移到安装目录,创建软链接，我这里安装到/opt/app目录

mv logstash-5.5. /opt/app/

ln -s logstash-5.5./ logstash

4.修改配置文件

vim /opt/app/logstash/config/logstash.yml

path.data: /opt/app/logstash/data/

pipeline.workers:

pipeline.output.workers:

pipeline.batch.size:

path.config: /opt/app/logstash/etc  #放配置文件

config.reload.automatic: false      #这里yes的话后面会报错，后面有提到这个问题

config.reload.interval:

path.logs: /data/logs/logstash

vim /opt/app/logstash/config/startup.options 修改部分选项如下：

JAVACMD=/opt/app/jdk/bin/java

LS_HOME=/opt/app/logstash

LS_JAVA_OPTS="/opt/app/jdk"

LS_PIDFILE=/opt/app/logstash/run/logstash.pid

LS_USER=logstash

LS_GROUP=logstash

LS_GC_LOG_FILE=/data/logs/logstash/gc.log

5.创建目录和用户

useradd -g  -u  logstash -s /sbin/nologin

mkdir /opt/app/logstash/run -p

mkdir /data/logs/logstash -p

mkdir /opt/app/logstash/data/ -p

mkdir /opt/app/logstash/etc -p

chown logstash:logstash /opt/app/logstash -R

chown logstash:logstash  /data/logs/logstash

6.创建启动脚本，通过systemd管理：

vim /etc/systemd/system/logstash.service

[Unit]

Description=logstash

[Service]

Type=simple

User=logstash

Group=logstash

EnvironmentFile=-/etc/default/logstash

#ExecStart=/opt/app/logstash/bin/logstash

ExecStart=/opt/app/logstash/bin/logstash --path.settings /opt/app/logstash/config

ExecStop=/bin/kill -s QUIT $MAINPID

Restart=always

WorkingDirectory=/

Nice=

LimitNOFILE=

[Install]

WantedBy=multi-user.target

vim /etc/default/logstash

JAVACMD="/opt/app/jdk/bin/java"

LS_HOME="/opt/app/logstash"

LS_SETTINGS_DIR="/opt/app/logstash/config"

LS_PIDFILE="/opt/app/logstash/run/logstash.pid"

LS_USER="logstash"

LS_GROUP="logstash"

LS_GC_LOG_FILE="/data/logs/logstash/gc.log"

LS_OPEN_FILES=""

LS_NICE=""

SERVICE_NAME="logstash"

SERVICE_DESCRIPTION="logstash"

这样logstash就可以启动了

三、实现logstash写入mariadb数据到elasticsearch

1.安装插件：（这是一个基于jdbc的数据导入插件，可以从各种能对接此驱动的数据库导入数据到ES,还可以自行设定导入执行频率，规则等。

cd /opt/app/logstash

bin/logstash-plugin install logstash-input-jdbc

出现success表明安装成功

tips:

安装插件国外源太慢，解决办法：

（1）先安装gem

yum install gem

(2)替换国内的镜像

gem sources --add https://gems.ruby-china.org/ --remove https://rubygems.org/

（3）验证是否成功

#gem sources -l

***CURRENTSOURCES***

https://gems.ruby-china.org/

（4）修改Gemfile数据源地址

cd /opt/app/logstash

修改Gemfile文件的source的值为：“https://gems.ruby-china.org/”

修改Gemfile.jruby-1.9.lock文件的remote的值为：“https://gems.ruby-china.org/

(5)执行安装插件

bin/logstash-plugin install logstash-input-jdbc

2.安装所需的java驱动包：

下载地址

解压到自定义目录，这个路径会在logstash配置文件中配置使用，我这里是：

/opt/app/logstash/lib/mysql-connector-java-5.1.44

3.生成配置文件，这里有工具可以生成

、、、、、、

tips:

logstash 启动后报错：

Logstash is not able to start since configuration auto reloading was enabled but the configuration contains plugins that don't support it. Quitting... {:pipeline_id=>"main", :plugins=>[LogStash::Inputs::Stdin]}

解决办法：

config.reload.automatic: false #关闭这个选项

巴特西

logstash5安装并实现mariadb数据写入到elasticsearch

最新文章

热门文章