elasticsearch5.2.1使用logstash同步mysql

centos 本人亲测可以首先安装好mysql，elasticsearch 不懂的请参考另一篇文章

安装logstash
官方：https://www.elastic.co/guide/en/logstash/current/installing-logstash.html
1.下载公共密钥
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
2.添加yum源
vim /etc/yum.repos.d/logstash.repo
文件中写入
[logstash-5.x]
name=Elastic repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
保存退出
3.使用yum安装
yum install logstash
4.验证是否安装成功
进入 logstash 安装目录
cd /usr/share/logstash
运行
bin/logstash -e 'input { stdin { } } output { stdout {} }'
等待几秒钟出现
The stdin plugin is now waiting for input:
然后输入
hello world
得到类似的结果
2016-11-24T08:01:55.949Z bogon hello world

安装logstash-input-jdbc插件

1.安装 ruby 和 rubygems（注意：需要 ruby 的版本在 1.8.7 以上）
# yum install -y ruby rubygems
检查 ruby 版本：
# ruby -v
ruby 1.8.7 (2013-06-27 patchlevel 374) [x86_64-linux]
替换国内的镜像
gem sources --remove http://rubygems.org/

gem sources -a http://gems.ruby-china.org/
验证是否成功
gem sources -l
修改Gemfile的数据源地址
whereis logstash # 查看logstash安装的位置，我的在 /usr/share/logstash目录

cd /usr/share/logstash
vim Gemfile
修改 source 的值为： "https://gems.ruby-china.org/"

vim Gemfile.jruby-1.9.lock

# 找到 remote 修改它的值为：https://gems.ruby-china.org/

开始安装

./bin/logstash-plugin install --no-verify logstash-input-jdbc

2.写配置文件开始同步 vi xxx.conf

input {
jdbc {
type => "user"
jdbc_connection_string => "jdbc:mysql://192.168.33.101:3306/test"
jdbc_user => "root"
jdbc_password => "123456"
jdbc_driver_library => "/usr/share/logstash/mysql-connector-java-5.1.41.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_paging_enabled => "true"
jdbc_page_size => "1000"
statement => "select * from user"
schedule => "* * * * *"
}
jdbc {
type => "task"
jdbc_connection_string => "jdbc:mysql://192.168.33.101:3306/test"
jdbc_user => "root"
jdbc_password => "123456"
jdbc_driver_library => "/usr/share/logstash/mysql-connector-java-5.1.41.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_paging_enabled => "true"
jdbc_page_size => "1000"
statement => "select * from task"
schedule => "* * * * *"
}
}
filter{
mutate{
remove_field => [ "@timestamp", "@version"]
}
}
output {
if [type] == "user" {
elasticsearch {
hosts => ["192.168.33.111:9200","192.168.33.112:9200","192.168.33.113:9200"]
index => "testindex"
document_type => "user"
document_id => "%{id}"
user => "elastic"
password => "changeme"
}
}
if [type] == "task" {
elasticsearch {
hosts => ["192.168.33.111:9200","192.168.33.112:9200","192.168.33.113:9200"]
index => "testindex"
document_type => "task"
document_id => "%{id}"
user => "elastic"
password => "changeme"
}
}
}

执行：bin/logstash -f xxx.conf

ok 大功告成！！！

巴特西

elasticsearch5.2.1使用logstash同步mysql

最新文章

热门文章