KingbaseES R6 集群禁用 root ssh 后需要修改集群为es

案例说明：

在生产环境下，由于安全需要，主机间不允许建立root用户的ssh信任连接，这样导致KingbaseES R6 repmgr集群，通过sys_monitor.sh脚本启动集群时，节点之间不能通过ssh正常访问，导致集群启动失败。本案例借助于es_server和es_client建立用户之间的信任连接，代替ssh访问。

测试数据库版本：

test=# select version();

                                                       version

----------------------------------------------------------------------------------------------------------------------

 KingbaseES V008R006C003B0010 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-46), 64-bit

(1 row)

如下图所示，由于不能建立root用户的信任连接，导致sys_monitor.sh启动无法正常启动：

一、配置es_server启动（所有node）

es_server 配置：

注意：这里的kingbase既不是数据库用户，也不是操作系统用户，而是连接es_server的用户。密码默认是 123456，该密码是通过md5算法加密的。

启动es_server：

[kingbase@node3 bin]$ ./esHAmodel.sh start

[kingbase@node3 bin]$ ps -ef |grep es_server

kingbase 28024     1  0 15:18 pts/2    00:00:00 /home/kingbase/cluster/R6HA/KHA/kingbase/bin/es_server

[kingbase@node3 bin]$ netstat -an |grep 8890

tcp        0      0 0.0.0.0:8890            0.0.0.0:*               LISTEN

测试es_server的连接：

[kingbase@node3 bin]$ ./es_client --help

es-client

Usage:

es-client [OPTION...] -o

Options:

  -U, --username=NAME    username for ES authentication

  -h, --host=HOSTNAME    ES Server host

  -p, --port=PORT        ES Server port number

  -W, --password         password

  -d, --debug            enable debug message (optional)

  -?, --help             print this help

  -o, --option           use user-define cmd: like "ls ."

  [kingbase@node3 bin]$ ./es_client -h 192.168.7.248 -U kingbase -W 123456 -o "hostname"

node1

[kingbase@node3 bin]$ ./es_client -h 192.168.7.249 -U kingbase -W 123456 -o "hostname"

node2

二、配置repmgr.conf支持bmj方式连接

=如下图所示：在sys_monitor.sh脚本中，如果bmj=on，则使用es_server和es_client通讯，所以需修改repmgr.conf启动bmj通讯。=

配置repmgr.conf：（所有node）

[kingbase@node3 bin]$ cat ../etc/repmgr.conf

# 启用bmj

on_bmj=on

node_id=3

node_name=node243

promote_command='/home/kingbase/cluster/R6HA/KHA/kingbase/bin/repmgr  standby promote -f /home/kingbase/cluster/R6HA/KHA/kingbase/etc/repmgr.conf'

follow_command='/home/kingbase/cluster/R6HA/KHA/kingbase/bin/repmgr  standby follow  -f /home/kingbase/cluster/R6HA/KHA/kingbase/etc/repmgr.conf -W --upstream-node-id=%n'

conninfo='host=192.168.7.243 user=esrep dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=2'

log_file='/home/kingbase/cluster/R6HA/KHA/kingbase/hamgr.log'

data_directory='/home/kingbase/cluster/R6HA/KHA/kingbase/data'

sys_bindir='/home/kingbase/cluster/R6HA/KHA/kingbase/bin'

ssh_options='-q -o ConnectTimeout=10 -o StrictHostKeyChecking=no -o ServerAliveInterval=2 -o ServerAliveCountMax=5 -p 22'

reconnect_attempts=2

reconnect_interval=3

failover='automatic'

recovery='automatic'

monitoring_history='no'

trusted_servers='192.168.7.1'

virtual_ip='192.168.7.240/24'

net_device='enp0s3'

ipaddr_path='/sbin'

arping_path='/sbin'

synchronous='quorum'

repmgrd_pid_file='/home/kingbase/cluster/R6HA/KHA/kingbase/hamgrd.pid'

ping_path='/usr/bin'

#priority=0

三、sys_monitor.sh启动集群测试

[kingbase@node3 bin]$ ./sys_monitor.sh restart

2021-03-01 15:25:58 Ready to stop all DB ...

sh: /etc/cron.d/KINGBASECRON: Permission deniedsh: /etc/cron.d/KINGBASECRON: Permission deniedsh: /etc/cron.d/KINGBASECRON: Permission denied2021-03-01 15:25:59 begin to stop repmgrd on "[192.168.7.248]".

2021-03-01 15:25:59 repmgrd on "[192.168.7.248]" stop success.

2021-03-01 15:25:59 begin to stop repmgrd on "[192.168.7.243]".

2021-03-01 15:25:59 repmgrd on "[192.168.7.243]" stop success.

2021-03-01 15:25:59 begin to stop repmgrd on "[192.168.7.249]".

2021-03-01 15:25:59 repmgrd on "[192.168.7.249]" stop success.

2021-03-01 15:25:59 begin to stop DB on "[192.168.7.248]".

waiting for server to shut down.... done

server stopped2021-03-01 15:26:00 DB on "[192.168.7.248]" stop success.

2021-03-01 15:26:00 begin to stop DB on "[192.168.7.249]".

waiting for server to shut down.... done

server stopped2021-03-01 15:26:00 DB on "[192.168.7.249]" stop success.

2021-03-01 15:26:00 begin to stop DB on "[192.168.7.243]".

waiting for server to shut down..... done

server stopped2021-03-01 15:26:01 DB on "[192.168.7.243]" stop success.

2021-03-01 15:26:01 Done.

2021-03-01 15:26:02 Ready to start all DB ...

2021-03-01 15:26:02 begin to start DB on "[192.168.7.243]".

waiting for server to start.... done

server started2021-03-01 15:26:02 execute to start DB on "[192.168.7.243]" success, connect to check it.

2021-03-01 15:26:03 DB on "[192.168.7.243]" start success.

2021-03-01 15:26:03 Try to ping trusted_servers on host 192.168.7.248 ...

2021-03-01 15:26:05 Try to ping trusted_servers on host 192.168.7.243 ...

2021-03-01 15:26:07 Try to ping trusted_servers on host 192.168.7.249 ...

2021-03-01 15:26:09 begin to start DB on "[192.168.7.248]".

waiting for server to start.... done

server started2021-03-01 15:26:10 execute to start DB on "[192.168.7.248]" success, connect to check it.

2021-03-01 15:26:11 DB on "[192.168.7.248]" start success.

2021-03-01 15:26:11 begin to start DB on "[192.168.7.249]".

waiting for server to start.... done

server started2021-03-01 15:26:12 execute to start DB on "[192.168.7.249]" success, connect to check it.

2021-03-01 15:26:13 DB on "[192.168.7.249]" start success.

 ID | Name    | Role    | Status    | Upstream | Location | Priority | Timeline | Connection string

----+---------+---------+-----------+----------+----------+----------+----------+---------------------------------------------------------------------------------------------------------------------------------------------------

 1  | node248 | standby | ! running | node243  | default  | 100      | 23       | host=192.168.7.248 user=esrep dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=2

 2  | node249 | witness | * running | node243  | default  | 0        | 1        | host=192.168.7.249 user=esrep dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=2

 3  | node243 | primary | * running |          | default  | 100      | 23       | host=192.168.7.243 user=esrep dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=2

WARNING: following issues were detected

  - node "node248" (ID: 1) is running but the repmgr node record is inactive

2021-03-01 15:26:13 The primary DB is started.

WARNING: There are no 2 standbys in pg_stat_replication, please check all the standby servers replica from primary

2021-03-01 15:26:37 Success to load virtual ip [192.168.7.240/24] on primary host [192.168.7.243].

2021-03-01 15:26:37 Try to ping vip on host 192.168.7.248 ...

2021-03-01 15:26:39 Try to ping vip on host 192.168.7.243 ...

2021-03-01 15:26:41 Try to ping vip on host 192.168.7.249 ...

2021-03-01 15:26:43 begin to start repmgrd on "[192.168.7.248]".

2021-03-01 15:26:43 repmgrd on "[192.168.7.248]" already started.

2021-03-01 15:26:43 begin to start repmgrd on "[192.168.7.243]".

2021-03-01 15:26:43 repmgrd on "[192.168.7.243]" already started.

2021-03-01 15:26:43 begin to start repmgrd on "[192.168.7.249]".

2021-03-01 15:26:43 repmgrd on "[192.168.7.249]" already started.

 ID | Name    | Role    | Status    | Upstream | repmgrd | PID   | Paused? | Upstream last seen

----+---------+---------+-----------+----------+---------+-------+---------+--------------------

 1  | node248 | standby |   running | node243  | running | 3589  | no      | 0 second(s) ago

 2  | node249 | witness | * running | node243  | running | 23739 | no      | 0 second(s) ago

 3  | node243 | primary | * running |          | running | 30496 | no      | n/a

sh: /etc/cron.d/KINGBASECRON: Permission deniedsh: /etc/logrotate.d/kingbase: Permission deniedchown: changing ownership of ‘/etc/logrotate.d/kingbase’: Operation not permittedchmod: changing permissions of ‘/etc/logrotate.d/kingbase’: Operation not permittedsh: /etc/cron.d/KINGBASECRON: Permission deniedsh: /etc/logrotate.d/kingbase: Permission deniedchown: changing ownership of ‘/etc/logrotate.d/kingbase’: Operation not permittedchmod: changing permissions of ‘/etc/logrotate.d/kingbase’: Operation not permittedsh: /etc/cron.d/KINGBASECRON: Permission deniedsh: /etc/logrotate.d/kingbase: Permission deniedchown: changing ownership of ‘/etc/logrotate.d/kingbase’: Operation not permittedchmod: changing permissions of ‘/etc/logrotate.d/kingbase’: Operation not permitted2021-03-01 15:26:44 Done.

如下图所示：sys_monitor.sh脚本启动访问“/etc/cron.d/KINGBASECRON”和“/etc/lograte.d/kingbase”文件时，出现权限错误：

注：

1）/etc/cron.d/KINGBASECRON，是repmgr集群启动时建立的计划任务，用于启动repmgrd进程。

2）/etc/logrotate.d/kingbase，配置文件用于切割hamgr.log和kbha.log日志

sys_monitor.sh脚本中/etc/cron.d/KINGBASECRON相关配置：

sys_monitor.sh脚本中/etc/logrotate.d/kingbase相关配置：

1）修改/etc/cron.d/KINGBASECRON文件相关权限（如下图所示）（所有node）

2）修改/etc/logrotate.d/kingbase相关权限（所有node）

修改kingbase文件所有者：（所有node）

注释sys_monitor.sh脚本中修改kingbase配置文件所有者和权限的语句：

function init_log_rotate()

{

_host="$1"

_final_target_file="/etc/logrotate.d/kingbase"

eval _rep_log_file=`grep log_file ${rep_conf} | awk -F '=' '{print $2}'`

execute_command ${super_user} $host "\

echo -e '# Generate by sys_monitor.sh at `date`\n\

${kbha_file} {\n\

        weekly\n\

        maxsize 100M\n\

        su ${execute_user} ${execute_user}\n\

        create 0600 ${execute_user} ${execute_user}\n\

        rotate 3\n\

        copytruncate\n\

        dateext\n\

}\n\

${_rep_log_file} {\n\

        weekly\n\

        maxsize 100M\n\

        su ${execute_user} ${execute_user}\n\

        create 0600 ${execute_user} ${execute_user}\n\

        rotate 3\n\

        copytruncate\n\

        dateext\n\

}\n\

' > ${_final_target_file}"

#execute_command ${super_user} $host "chown ${super_user}:${super_user} ${_final_target_file}"

#execute_command ${super_user} $host "chmod 644 ${_final_target_file}"

如下图所示：

四、测试集群启动

[kingbase@node3 bin]$ ./sys_monitor.sh restart

2021-03-01 15:52:08 Ready to stop all DB ...

2021-03-01 15:52:08 begin to stop repmgrd on "[192.168.7.248]".

2021-03-01 15:52:08 repmgrd on "[192.168.7.248]" stop success.

2021-03-01 15:52:08 begin to stop repmgrd on "[192.168.7.243]".

2021-03-01 15:52:08 repmgrd on "[192.168.7.243]" stop success.

2021-03-01 15:52:08 begin to stop repmgrd on "[192.168.7.249]".

2021-03-01 15:52:08 repmgrd on "[192.168.7.249]" stop success.

2021-03-01 15:52:08 begin to stop DB on "[192.168.7.248]".

waiting for server to shut down..... done

server stopped2021-03-01 15:52:09 DB on "[192.168.7.248]" stop success.

2021-03-01 15:52:09 begin to stop DB on "[192.168.7.249]".

waiting for server to shut down.... done

server stopped2021-03-01 15:52:10 DB on "[192.168.7.249]" stop success.

2021-03-01 15:52:10 begin to stop DB on "[192.168.7.243]".

waiting for server to shut down..... done

server stopped2021-03-01 15:52:12 DB on "[192.168.7.243]" stop success.

2021-03-01 15:52:12 Done.

2021-03-01 15:52:12 Ready to start all DB ...

2021-03-01 15:52:12 begin to start DB on "[192.168.7.243]".

waiting for server to start.... done

server started2021-03-01 15:52:12 execute to start DB on "[192.168.7.243]" success, connect to check it.

2021-03-01 15:52:13 DB on "[192.168.7.243]" start success.

2021-03-01 15:52:13 Try to ping trusted_servers on host 192.168.7.248 ...

2021-03-01 15:52:15 Try to ping trusted_servers on host 192.168.7.243 ...

2021-03-01 15:52:17 Try to ping trusted_servers on host 192.168.7.249 ...

2021-03-01 15:52:19 begin to start DB on "[192.168.7.248]".

waiting for server to start.... done

server started2021-03-01 15:52:20 execute to start DB on "[192.168.7.248]" success, connect to check it.

2021-03-01 15:52:21 DB on "[192.168.7.248]" start success.

2021-03-01 15:52:21 begin to start DB on "[192.168.7.249]".

waiting for server to start.... done

server started2021-03-01 15:52:21 execute to start DB on "[192.168.7.249]" success, connect to check it.

2021-03-01 15:52:22 DB on "[192.168.7.249]" start success.

 ID | Name    | Role    | Status    | Upstream | Location | Priority | Timeline | Connection string

----+---------+---------+-----------+----------+----------+----------+----------+---------------------------------------------------------------------------------------------------------------------------------------------------

 1  | node248 | standby |   running | node243  | default  | 100      | 23       | host=192.168.7.248 user=esrep dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=2

 2  | node249 | witness | * running | node243  | default  | 0        | 1        | host=192.168.7.249 user=esrep dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=2

 3  | node243 | primary | * running |          | default  | 100      | 23       | host=192.168.7.243 user=esrep dbname=esrep port=54321 connect_timeout=10 keepalives=1 keepalives_idle=10 keepalives_interval=1 keepalives_count=2

2021-03-01 15:52:22 The primary DB is started.

WARNING: There are no 2 standbys in pg_stat_replication, please check all the standby servers replica from primary

2021-03-01 15:52:46 Success to load virtual ip [192.168.7.240/24] on primary host [192.168.7.243].

2021-03-01 15:52:46 Try to ping vip on host 192.168.7.248 ...

2021-03-01 15:52:48 Try to ping vip on host 192.168.7.243 ...

2021-03-01 15:52:50 Try to ping vip on host 192.168.7.249 ...

2021-03-01 15:52:52 begin to start repmgrd on "[192.168.7.248]".

[2021-03-01 15:54:17] [NOTICE] using provided configuration file "/home/kingbase/cluster/R6HA/KHA/kingbase/bin/../etc/repmgr.conf"

[2021-03-01 15:54:17] [NOTICE] redirecting logging output to "/home/kingbase/cluster/R6HA/KHA/kingbase/hamgr.log"

2021-03-01 15:52:52 repmgrd on "[192.168.7.248]" start success.

2021-03-01 15:52:52 begin to start repmgrd on "[192.168.7.243]".

[2021-03-01 15:52:52] [NOTICE] using provided configuration file "/home/kingbase/cluster/R6HA/KHA/kingbase/bin/../etc/repmgr.conf"

[2021-03-01 15:52:52] [NOTICE] redirecting logging output to "/home/kingbase/cluster/R6HA/KHA/kingbase/hamgr.log"

2021-03-01 15:52:52 repmgrd on "[192.168.7.243]" start success.

2021-03-01 15:52:52 begin to start repmgrd on "[192.168.7.249]".

[2021-03-01 14:50:47] [NOTICE] using provided configuration file "/home/kingbase/cluster/R6HA/KHA/kingbase/bin/../etc/repmgr.conf"

[2021-03-01 14:50:47] [NOTICE] redirecting logging output to "/home/kingbase/cluster/R6HA/KHA/kingbase/hamgr.log"

2021-03-01 15:52:53 repmgrd on "[192.168.7.249]" start success.

 ID | Name    | Role    | Status    | Upstream | repmgrd | PID   | Paused? | Upstream last seen

----+---------+---------+-----------+----------+---------+-------+---------+--------------------

 1  | node248 | standby |   running | node243  | running | 13909 | no      | 0 second(s) ago

 2  | node249 | witness | * running | node243  | running | 28830 | no      | n/a

 3  | node243 | primary | * running |          | running | 6643  | no      | n/a

2021-03-01 15:52:53 Done.

如下图所示：集群启动正常

附件：/etc/logrotate.d/kingbase权限故障处理

如下图所示：sys_monitor.sh脚本启动集群出现以下错误：

解决方案：

[root@node3 ~]# which chmod

/usr/bin/chmod

[root@node3 ~]# which chown

/usr/bin/chown

[root@node3 ~]# ls -lh /usr/bin/chown

-rwxr-xr-x. 1 root root 62K Nov 20  2015 /usr/bin/chown

[root@node3 ~]# ls -lh /usr/bin/chmod

-rwxr-xr-x. 1 root root 58K Nov 20  2015 /usr/bin/chmod

[root@node3 ~]# chmod u+s /usr/bin/chown

[root@node3 ~]# chmod u+s /usr/bin/chmod

[root@node3 ~]# ls -lh /usr/bin/chmod

-rwsr-xr-x. 1 root root 58K Nov 20  2015 /usr/bin/chmod

[root@node3 ~]# ls -lh /usr/bin/chown

-rwsr-xr-x. 1 root root 62K Nov 20  2015 /usr/bin/chown

[root@node3 ~]# ls -lh /etc/logrotate.d/kingbase

-rw-r--r--. 1 kingbase kingbase 492 Mar  1 15:52 /etc/logrotate.d/kingbase

[root@node3 ~]# su - kingbase

Last login: Mon Mar  1 15:51:39 CST 2021 on pts/1

Last failed login: Mon Mar  1 15:58:21 CST 2021 from :0 on :0

There was 1 failed login attempt since the last successful login.

[kingbase@node3 ~]$ chown root.root /etc/logrotate.d/kingbase

[kingbase@node3 ~]$ ls -lh  /etc/logrotate.d/kingbase

-rw-r--r--. 1 root root 492 Mar  1 15:52 /etc/logrotate.d/kingbase

[kingbase@node3 ~]$ chown kingbase.kingbase /etc/logrotate.d/kingbase

[kingbase@node3 ~]$ ls -lh  /etc/logrotate.d/kingbase

-rw-r--r--. 1 kingbase kingbase 492 Mar  1 15:52 /etc/logrotate.d/kingbase

#手工执行“sh /etc/logrotate.d/kingbase”

[kingbase@node3 bin]$ sh /etc/logrotate.d/kingbase

/etc/logrotate.d/kingbase: line 2: /home/kingbase/cluster/R6HA/KHA/kingbase/bin/../kbha.log: Permission denied

/etc/logrotate.d/kingbase: line 3: weekly: command not found

/etc/logrotate.d/kingbase: line 4: maxsize: command not found

[kingbase@node3 kingbase]$ chmod u+x kbha.log

[kingbase@node3 kingbase]$ sh /etc/logrotate.d/kingbase

/etc/logrotate.d/kingbase: line 2: /home/kingbase/cluster/R6HA/KHA/kingbase/bin/../kbha.log: Text file busy

/etc/logrotate.d/kingbase: line 3: weekly: command not found

/etc/logrotate.d/kingbase: line 4: maxsize: command not found

Password:

=通过以上处理，在通过sys_monitor.sh脚本启动集群时，仍然出现“sh /etc/logrotate.d/kingbase"错误，故修改了sys_monitor.sh脚本后，问题解决。=

巴特西

KingbaseES R6 集群禁用 root ssh 后需要修改集群为es_server 案例

最新文章

热门文章