mysql-MHA 故障收集

在manager 主机上开启监控服务，启动不了

[root@manager ~]# managerStart

[]

[root@manager ~]# managerStatus

app1 is stopped(:NOT_RUNNING).

[]+  Exit                   nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /var/log/masterha/app1/manager.log >&1
#说明： 这里我对启动服务的命令做了 别名命令。

#查看日志发现有这么一句话：

Sun Mar  ::  - [error][/usr/local/share/perl5/MHA/ServerManager.pm, ln781] Multi-master configuration is detected, 
but two or more masters are either writable (read-only is not set) or dead! Check configurations for details. Master configurations are as below:

Master 10.0.0.50(10.0.0.50:)

Master 10.0.0.60(10.0.0.60:), replicating from 10.0.0.50(10.0.0.50:)

这句话的大概意思，有两个成为主，而且两个都可写，按照原则同一时间只能有一台主机可以数据写入，不然可能会造成数据不一致的灾难性故障！

在10.0.0.60 上开启mysql设置开启只读

mysql -e 'set global read_only=1'

设置完，还没完依旧开启不了这个监控程序，错误依旧存在

Sun Mar  ::  - [info] Multi-master configuration is detected. Current primary(writable) master is 10.0.0.50(10.0.0.50:)

Sun Mar  ::  - [info] Master configurations are as below:

Master 10.0.0.50(10.0.0.50:)

Master 10.0.0.60(10.0.0.60:), replicating from 10.0.0.50(10.0.0.50:), read-only

Sun Mar  ::  - [error][/usr/local/share/perl5/MHA/ServerManager.pm, ln726] Slave 10.0.0.70(10.0.0.70:) replicates from 10.0.0.60:, but real master is 10.0.0.50(10.0.0.50:)!

Sun Mar  ::  - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln424] Error happened on checking configurations.  at /usr/local/share/perl5/MHA/MasterMonitor.pm line

Sun Mar  ::  - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln523] Error happened on monitoring servers.

Sun Mar  ::  - [info] Got exit code  (Not master dead).

分析了下，为什么会出现两个master呢？因为之前模拟master宕机故障之后，vip飘到60并且60主机被提升为主，70主机本来是50主机的小弟，现在成为了60主机的小弟，这就导致了出现两个master，

为了验证我这样的猜想，我强行设置，70跟随50 混，就change master to 指定主机是50 什么位置信息和binlog文件也是50主机的信息

(￣▽￣)"哈哈，猜中。。。开森了下。。

[root@manager ~]# managerStatus

app1 monitoring program is now on initialization phase(:INITIALIZING_MONITOR). Wait for a while and try checking again.

[root@manager ~]# managerStatus

app1 (pid:) is running(:PING_OK), master:10.0.0.50

Sun Mar  ::  - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.

Sun Mar  ::  - [info] Reading application default configuration from /etc/masterha/app1.cnf..

Sun Mar  ::  - [info] Reading server configuration from /etc/masterha/app1.cnf..

Sun Mar  ::  - [info] MHA::MasterMonitor version 0.56.

Sun Mar  ::  - [info] GTID failover mode =

Sun Mar  ::  - [info] Dead Servers:

Sun Mar  ::  - [info] Alive Servers:

Sun Mar  ::  - [info]   10.0.0.50(10.0.0.50:)

Sun Mar  ::  - [info]   10.0.0.60(10.0.0.60:)

Sun Mar  ::  - [info]   10.0.0.70(10.0.0.70:)

Sun Mar  ::  - [info] Alive Slaves:

Sun Mar  ::  - [info]   10.0.0.60(10.0.0.60:)  Version=5.6.-log (oldest major version between slaves) log-bin:enabled

Sun Mar  ::  - [info]     Replicating from 10.0.0.50(10.0.0.50:)

Sun Mar  ::  - [info]     Primary candidate for the new Master (candidate_master is set)

Sun Mar  ::  - [info]   10.0.0.70(10.0.0.70:)  Version=5.6. (oldest major version between slaves) log-bin:disabled

Sun Mar  ::  - [info]     Replicating from 10.0.0.50(10.0.0.50:)

Sun Mar  ::  - [info] Current Alive Master: 10.0.0.50(10.0.0.50:)

Sun Mar  ::  - [info] Checking slave configurations..

Sun Mar  ::  - [warning]  relay_log_purge= is not set on slave 10.0.0.60(10.0.0.60:).

Sun Mar  ::  - [warning]  relay_log_purge= is not set on slave 10.0.0.70(10.0.0.70:).

Sun Mar  ::  - [warning]  log-bin is not set on slave 10.0.0.70(10.0.0.70:). This host cannot be a master.

Sun Mar  ::  - [info] Checking replication filtering settings..

Sun Mar  ::  - [info]  binlog_do_db= , binlog_ignore_db=

Sun Mar  ::  - [info]  Replication filtering check ok.

Sun Mar  ::  - [info] GTID (with auto-pos) is not supported

Sun Mar  ::  - [info] Starting SSH connection tests..

Sun Mar  ::  - [info] All SSH connection tests passed successfully.

Sun Mar  ::  - [info] Checking MHA Node version..

Sun Mar  ::  - [info]  Version check ok.

Sun Mar  ::  - [info] Checking SSH publickey authentication settings on the current master..

Sun Mar  ::  - [info] HealthCheck: SSH to 10.0.0.50 is reachable.

Sun Mar  ::  - [info] Master MHA Node version is 0.56.

Sun Mar  ::  - [info] Checking recovery script configurations on 10.0.0.50(10.0.0.50:)..

Sun Mar  ::  - [info]   Executing command: save_binary_logs --command=test --start_pos= --binlog_dir=/mysql/data --output_file=/tmp/save_binary_logs_test --manager_version=0.56 --start_file=mysql-bin.

Sun Mar  ::  - [info]   Connecting to root@10.0.0.50(10.0.0.50:)..

  Creating /tmp if not exists..    ok.

  Checking output directory is accessible or not..

   ok.

  Binlog found at /mysql/data, up to mysql-bin.

Sun Mar  ::  - [info] Binlog setting check done.

Sun Mar  ::  - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..

Sun Mar  ::  - [info]   Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=10.0.0.60 --slave_ip=10.0.0.60 --slave_port= --workdir=/tmp --target_version=5.6.-log --manager_version=0.56 --relay_log_info=/mysql/data/relay-log.info  --relay_dir=/mysql/data/  --slave_pass=xxx

Sun Mar  ::  - [info]   Connecting to root@10.0.0.60(10.0.0.60:)..

  Checking slave recovery environment settings..

    Opening /mysql/data/relay-log.info ... ok.

    Relay log found at /mysql/data, up to cadicate-master-relay-bin.

    Temporary relay log file is /mysql/data/cadicate-master-relay-bin.

    Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure.

 done.

    Testing mysqlbinlog output.. done.

    Cleaning up test file(s).. done.

Sun Mar  ::  - [info]   Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=10.0.0.70 --slave_ip=10.0.0.70 --slave_port= --workdir=/tmp --target_version=5.6. --manager_version=0.56 --relay_log_info=/mysql/data/relay-log.info  --relay_dir=/mysql/data/  --slave_pass=xxx

Sun Mar  ::  - [info]   Connecting to root@10.0.0.70(10.0.0.70:)..

  Checking slave recovery environment settings..

    Opening /mysql/data/relay-log.info ... ok.

    Relay log found at /mysql/data, up to slave-relay-bin.

    Temporary relay log file is /mysql/data/slave-relay-bin.

    Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure.

 done.

    Testing mysqlbinlog output.. done.

    Cleaning up test file(s).. done.

Sun Mar  ::  - [info] Slaves settings check done.

Sun Mar  ::  - [info]

10.0.0.50(10.0.0.50:) (current master)

 +--10.0.0.60(10.0.0.60:)

 +--10.0.0.70(10.0.0.70:)

Sun Mar  ::  - [info] Checking master_ip_failover_script status:

Sun Mar  ::  - [info]   /usr/local/bin/master_ip_failover --command=status --ssh_user=root --orig_master_host=10.0.0.50 --orig_master_ip=10.0.0.50 --orig_master_port= 

IN SCRIPT TEST====/etc/init.d/keepalived stop==/etc/init.d/keepalived start===

Checking the Status of the script.. OK

Sun Mar  ::  - [info]  OK.

Sun Mar  ::  - [warning] shutdown_script is not defined.

Sun Mar  ::  - [info] Set master ping interval  seconds.

Sun Mar  ::  - [info] Set secondary check script: /usr/local/bin/masterha_secondary_check -s server03 -s server02

Sun Mar  ::  - [info] Starting ping health check on 10.0.0.50(10.0.0.50:)..

Sun Mar  ::  - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..

分析日志，分析日志，分析日志，重要事情强调3遍！

巴特西

mysql-MHA 故障收集

最新文章

热门文章