一、什么是Monstache

Monstache 是Golang语言实现的基于MongoDB的oplog实现实时数据同步及订阅的插件,支持MongoDB与ES之间的数据同步。其中MongoDB需要搭建副本集。

二、安装过程

1、安装Go,并配置环境变量

(1) 下载Go并解压

wget https://dl.google.com/go/go1.14.4.linux-amd64.tar.gz  
tar -C /usr/local -xzf go1.14.4.linux-amd64.tar.gz   (2) 建立go的工作空间,即GOAPTH所指向的目录 创建三个目录 mkdir bin、mkdir src、mkdir pkg (3) 配置环境变量 使用 vim /etc/profile命令打开环境变量配置文件,并将如下内容写入该文件中。其中GOPROXY用来指定阿里云Go模块代理 export GOROOT=/usr/local/go  
export GOPATH=/home/go/  
export PATH=$PATH:$GOROOT/bin:$GOPATH/bin  
export GOPROXY=https://mirrors.aliyun.com/goproxy/   (4) 应用环境变量配置 source /etc/profile  

2、Monstache说明

(1)Monstache-Mongodb-Es 版本关系:

Monstache version

Git branch (used to build plugin)

Docker tag

Description

Elasticsearch

MongoDB

Status

6

rel6

rel6, latest

MongoDB, Inc. go driver

Version 7+

Version 2.6+

Supported

5

rel5

rel5

MongoDB, Inc. go driver

Version 6

Version 2.6+

Supported

4

master

rel4

mgo community go driver

Version 6

Version 3

Deprecated

3

rel3

rel3

mgo community go driver

Versions 2 and 5

Version 3

Deprecated

(2)Monstache配置

参数

说明

mongo-url

MongoDB实例的主节点访问地址。可在实例的基本信息页面获取,获取前需配置MongoDB实例的白名单,即在白名单中添加安装Monstache的ECS实例的内网IP地址。

elasticsearch-urls

Elasticsearch实例的访问地址,格式为http://<Elasticsearch实例的内网地址>:9200。

direct-read-namespaces

指定待同步的集合

change-stream-namespaces

如果要使用MongoDB变更流功能,需要指定此参数。启用此参数后,oplog追踪会被设置为无效

namespace-regex

通过正则表达式指定需要监听的集合。此设置可以用来监控符合正则表达式的集合中数据的变化。

elasticsearch-user

访问Elasticsearch实例的用户名,默认为elastic。

注意 实际业务中不建议使用elastic用户,这样会降低系统安全性。建议使用自建用户,并给予自建用户分配相应的角色和权限。

elasticsearch-password

对应用户的密码。elastic用户的密码在创建实例时指定,如果忘记可进行重置

elasticsearch-max-conns

定义连接Elasticsearch的线程数。默认为4,即使用4个Go线程同时将数据同步到Elasticsearch。

dropped-collections

默认为true,表示当删除MongoDB集合时,会同时删除Elasticsearch中对应的索引。

dropped-databases

默认为true,表示当删除MongoDB数据库时,会同时删除Elasticsearch中对应的索引。

resume

默认为false。设置为true,Monstache会将已成功同步到Elasticsearch的MongoDB操作的时间戳写入monstache.monstache集合中。当Monstache因为意外停止时,可通过该时间戳恢复同步任务,避免数据丢失。如果指定了cluster-name,该参数将自动开启。

resume-strategy

指定恢复策略。仅当resume为true时生效。

verbose

默认为false,表示不启用调试日志。

cluster-name

指定集群名称。指定后,Monstache将进入高可用模式,集群名称相同的进程将进行协调

mapping

指定Elasticsearch索引映射。默认情况下,数据从MongoDB同步到Elasticsearch时,索引会自动映射为数据库名.集合名。如果需要修改索引名称,可通过该参数设置。

3、Monstache安装

(1)进入安装路径

cd /usr/local

(2)使用命令下载或手动上传安装包

1)从git库下载

git clone https://github.com/rwynn/monstache.git,若提示下载失败需先下载git ,使用yum install -y git命令

2)手动上传

资源下载地址: https://github.com/rwynn/monstache

进入安装目录,安装

cd /usr/local/monstache-6.7.11
go install

查看版本

安装完成

4、配置实时同步任务

(1)进入Monstache安装目录,创建并编辑配置文件

#创建monstache的存储目录
cd /usr/local/monstache-6.7.11
mkdir config
mkdir logs

进入config目录,编辑创建配置文件

cd config
vim config.toml
文件内容如下
# connection settings  

# connect to MongoDB using the following URL
mongo-url = "mongodb://192.168.8.79:27017/pns" # connect to the Elasticsearch REST API at the following node URLs
elasticsearch-urls = ["http://192.168.8.153:9200"] # frequently required settings # if you need to seed an index from a collection and not just listen and sync changes events
# you can copy entire collections or views from MongoDB to Elasticsearch
direct-read-namespaces = ["库名.集合名"] # if you want to use MongoDB change streams instead of legacy oplog tailing use change-stream-namespaces
# change streams require at least MongoDB API 3.6+
# if you have MongoDB 4+ you can listen for changes to an entire database or entire deployment
# in this case you usually don't need regexes in your config to filter collections unless you target the deployment.
# to listen to an entire db use only the database name. For a deployment use an empty string.
#change-stream-namespaces = ["mydb.col"] # additional settings # if you don't want to listen for changes to all collections in MongoDB but only a few
# e.g. only listen for inserts, updates, deletes, and drops from mydb.mycollection
# this setting does not initiate a copy, it is only a filter on the change event listener
namespace-regex = '^库名\.集合名$'
# compress requests to Elasticsearch
gzip = true
# generate indexing statistics
#stats = true
# index statistics into Elasticsearch
#index-stats = true
# use the following PEM file for connections to MongoDB
#mongo-pem-file = "/path/to/mongoCert.pem"
# disable PEM validation
#mongo-validate-pem-file = false
# use the following user name for Elasticsearch basic auth
elasticsearch-user = "elastic"
# use the following password for Elasticsearch basic auth
elasticsearch-password = "<your_es_password>"
# use 4 go routines concurrently pushing documents to Elasticsearch
elasticsearch-max-conns = 4
# use the following PEM file to connections to Elasticsearch
#elasticsearch-pem-file = "/path/to/elasticCert.pem"
# validate connections to Elasticsearch
#elastic-validate-pem-file = true
# propogate dropped collections in MongoDB as index deletes in Elasticsearch
dropped-collections = true
# propogate dropped databases in MongoDB as index deletes in Elasticsearch
dropped-databases = true
# do not start processing at the beginning of the MongoDB oplog
# if you set the replay to true you may see version conflict messages
# in the log if you had synced previously. This just means that you are replaying old docs which are already
# in Elasticsearch with a newer version. Elasticsearch is preventing the old docs from overwriting new ones.
#replay = false
# resume processing from a timestamp saved in a previous run
resume = true
# do not validate that progress timestamps have been saved
#resume-write-unsafe = false
# override the name under which resume state is saved
#resume-name = "default"
# use a custom resume strategy (tokens) instead of the default strategy (timestamps)
# tokens work with MongoDB API 3.6+ while timestamps work only with MongoDB API 4.0+
resume-strategy = 0
# exclude documents whose namespace matches the following pattern
#namespace-exclude-regex = '^mydb\.ignorecollection$'
# turn on indexing of GridFS file content
#index-files = true
# turn on search result highlighting of GridFS content
#file-highlighting = true
# index GridFS files inserted into the following collections
#file-namespaces = ["users.fs.files"]
# print detailed information including request traces
verbose = true
# enable clustering mode
#cluster-name = 'es-cn-mp91kzb8m00******'
# do not exit after full-sync, rather continue tailing the oplog
#exit-after-direct-reads = false
[[mapping]]
namespace = "pns.user_device_table"
index = "pns.user_device_table"
#type = "device"

注:1)如果不设置mapping,那么同步到es的索引默认为mongoDB的数据库名.集合名

2)monstache是根据mongoDB的oplog来同步更新的,因此需要开启mongo的副本集才可以

3)当mongo有密码认证时mongoUrl需要改为

mongodb://用户:密码@IP:PORT/数据库名称?authSource=数据库名称&authMechanism=SCRAM-SHA-1,同时这个地址要为mongoDB主节点访问地址

4)direct-read-namespaces表示初始化会全量同步到es,而change-stream-namespaces只会同步有更改变化的

5)namespace-regex不设置时默认同步mongoDB中所有数据的更改变化,可以指定只同步哪个或哪些集合的增删改

(2)配置自启

#切换目录
cd /etc/init.d #编辑命令
vim monstache

脚本内容

#!/bin/bash
#
# chkconfig: 2345 10 90
#
# description: MONSTACHE RUN #程序名
MONSTACHE_NAME="monstache-rel6" #资源位置
MONSTACHE_OPTS=/home/go/bin/monstache
MONSTACHE_CONF=/usr/local/monstache-6.7.11/config/config.toml #日志位置
MONSTACHE_LOGS=/usr/local/monstache-6.7.11/logs/monstache.log #开始方法
start() {
nohup $MONSTACHE_OPTS -f $MONSTACHE_CONF > $MONSTACHE_LOGS 2>&1 &
echo "$MONSTACHE_NAME started success."
} #结束方法
stop() {
echo "stopping $MONSTACHE_NAME ..."
kill -9 `ps -ef|grep $MONSTACHE_OPTS|grep -v grep|grep -v stop|awk '{print $2}'`
} case "$1" in
start)
start
;;
stop)
stop
;;
restart)
stop
start
;;
*)
echo "Userage: $0 {start|stop|restart}"
exit 1
esac
#授权
chmod +x monstache #添加系统服务
chkconfig --add monstache
chkconfig monstache on
chkconfig --list

注:MONSTACHE_OPTS位置是在GO的安装目录下

(3)启停相关

#启停命令
systemctl start|stop|restart monstache #查看进程
ps -ef|grep monstache

注:

  1、使用版本

    mongo 4.4.15

    monstache 6.7.11

    elasticsearch 7.9.2

  2、mongo单机创建副本集

    参考文档:http://www.kaotop.com/it/14307.html

  3、集群下创建副本集

    参考文档:https://blog.csdn.net/qq_42428264/article/details/124227050

参考原文地址:https://blog.csdn.net/weixin_44187730/article/details/117198268

最新文章

  1. 【Google Chrome】 Cross origin requests are only supported for protocol schemes: http, data, chrome, chrome-extension, https, chrome-extension-resource问题解决
  2. Struts1.x 中的 Validate 框架
  3. Java中静态内部类的理解
  4. Xilium.CefGlue CEF Chrome 自动上传文件不弹出对话框 CefDialogHandler
  5. JavaScript 全局对象
  6. 笔记本_thinkpad_e440
  7. apache配置域名指向
  8. shell exit 0 exit 1
  9. linq 日常关键字使用
  10. MVC (M-V-C启动程序调用关系)
  11. 【最大点独立集】【poj1419】【Graph Coloring】
  12. jquery中的页面加载方法load()
  13. Chrome 下动画卡顿问题的另一种可能
  14. C# 使用Parallel并行开发Parallel.For、Parallel.Foreach实例
  15. KVO键值观察的具体实现
  16. SYBASE的select into与insert into使用和区别
  17. 第15章-输入/输出 --- File类
  18. LinkedHashMap源码分析及实现LRU
  19. JAVA 动态代理原理和实现
  20. 【数据表格】datatable+SpringMVC+Spring Data JPA

热门文章

  1. 遗传算法 Genetic Algorithms
  2. 前端Linux部署命令与流程记录
  3. 疯狂吐槽 MAUI 以及 MAUI 入坑知识点
  4. 【学习笔记】珂朵莉树(ODT)
  5. BUG日记---SSM进行多表查询错误-----页面使用&lt;c:foreach&gt;错误
  6. 认知篇:CQRS架构模式的本质
  7. 12月2日内容总结——边框属性,display属性,css盒子模型,浮动、溢出、定位、z-index属性和建议博客页面搭建
  8. React Refs-知识点整理记录
  9. STM32F0库函数初始化系列:PWM输出
  10. 微信小程序使用vant组件样式不生效的问题