MySQL+MHA+keepalive+vip 安装配置

在mysql的复制，当主服务崩溃了，利用mha实现主服务自动切换，并能使其他从服务切换到新的主机。下面是部署步骤

（1）准备三机器：主服务 192.168.8.120，备主 192.168.8.121 ，从服务和管理节点192.168.8.122

（2）修改各台主机名如管理节点192.168.8.122

cat /etc/hosts

[root@centos3 mha]# more /etc/hosts
127.0.0.1 localhost
192.168.8.120 centos1
192.168.8.121 centos2
192.168.8.122 centos3

（3）数据节点

安装mha4mysql-node-0.53.tar.gz mha4mysql-manager-0.53.tar.gz，由于mha4mysql-node 依赖 perl-DBD-MySQL，mha4mysql-manager依赖perl-Config-Tiny perl-Params-Validate perl-Log-Dispatch perl-Parallel-ForkManager 。所以现在这些依赖包。实验使用yum 安装。

对三台mariadb数据节点只需安装mha4mysql-node-0.53.tar.gz ，本文没有写mariadb的安装以及复制。

[root@centos1mha]#rpm -ivh http://dl.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm

[root@centos1mha]#yum -y install perl-DBD-MySQL ncftp

[root@centos1mha]#tar -zxfmha4mysql-node-0.53.tar.gz

[root@centos1mha]# cdmha4mysql-node-0.53

[root@centos1mha]# perl Makefile.PL

[root@centos1mha]#make && make install

(4)管理节点

[root@sh-gs-dbmg0227 ~]# rpm -ivh http://dl.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm

//这个是centos5.x 如果是6.x rpm -ivh http://dl.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm

[root@centos3 mha]# yum -y installperl-DBD-MySQL ncftp

[root@centos3 mha]# tar -zxfmha4mysql-node-0.53.tar.gz

[root@centos3 mha]# cd mha4mysql-node-0.53

[root@centos3 mha]#perl Makefile.PL

[root@centos3 mha]#make && make install

[root@centos3 mha]# yum -y install perl-Config-Tiny perl-Params-Validate perl-Log-Dispatch perl-Parallel-ForkManager perl-Config-IniFiles

[root@centos3 mha]# tar -zxfmha4mysql-manager-0.53.tar.gz

[root@centos3 mha]#perl Makefile.PL

如果在该过程中出现下面错误
Can't locate ExtUtils/MakeMaker.pm in @INC (@INC contains: inc/usr/local/lib64/perl5 /usr/local/share/perl5/usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl/usr/lib64/perl5 /usr/share/perl5 .) at inc/Module/Install/Can.pmline 6.
解决方法：
yum installperl-CPAN

[root@centos3 mha]#make && make install

[root@centos3mha]# mkdir/etc/masterha

[root@centos3mha]# mkdir -p/master/app1
[root@centos3mha]# mkdir -p/scripts

[root@centos3mha]# cp samples/conf/*/etc/masterha/

[root@centos3mha]# cpsamples/scripts/* /scripts

配置管理节点

[root@centos3mha]#more /etc/masterha/masterha_default.cnf
[server default]
user=root
password=123456
ssh_user=root
repl_user=repl
repl_password=qwert
master_binlog_dir= /app/mysql
remote_workdir=/app/mha
secondary_check_script= masterha_secondary_check-s192.168.12.234-s192.168.12.232
ping_interval=1
master_ip_failover_script=/scripts/master_ip_failover
#shutdown_script=/scripts/power_manager
report_script= /scripts/send_report
master_ip_online_change_script=/scripts/master_ip_online_change

[root@centos3mha]# more /etc/masterha/app1.cnf

[server default]
manager_workdir=/app/mha
manager_log=/app/mha/manager.log

[server1]
hostname=192.168.8.120
candidate_master=1

[server2]
hostname=192.168.8.121
candidate_master=1

[server3]
hostname=192.168.8.122
no_master=1

（5）在mysql 添加用户，复制设置

mysql 主节点

grant replication slave on *.* to 'repl'@'192.168.8.%' identifiedby 'qwert';

grant all on *.* to 'root'@'192.168.8.122' identifiedby '123456';

备主节点

grant replication slave on *.* to 'repl'@'192.168.8.%' identifiedby 'qwert';

grant all on *.* to 'root'@'192.168.8122' identifiedby '123456';

set read_only=1
set relay_log_purge=0

从节点

grant all on *.* to 'root'@'192.168.8.122' identifiedby '123456';

set read_only=1
set relay_log_purge=0

（6）配置ssh

[[root@centos3~#ssh-keygen -t rsa

[root@centos3~]# ssh-copy-id -i.ssh/id_rsa.pub root@192.168.8.120

[root@centos3~]#ssh-copy-id -i .ssh/id_rsa.pub root@192.168.8.121

[root@centos1~]# ssh-keygen -t rsa

[root@centos1~]# ssh-copy-id -i.ssh/id_rsa.pub root@192.168.8.121

[root@centos1~]# ssh-copy-id -i.ssh/id_rsa.pub root@192.168.8.122

[root@centos2~]# ssh-keygen -trsa

[root@centos2~]# ssh-copy-id -i.ssh/id_rsa.pub root@192.168.8.120

[root@centos2~]# ssh-copy-id -i .ssh/id_rsa.pub root@192.168.8.122

(7)测试ssh

[root@centos3 mha]# masterha_check_ssh--global_conf=/etc/masterha/masterha_default.cnf --conf=/etc/masterha/app1.cnf
Sat Aug 10 06:15:39 2013 - [info] Reading default configuratoinsfrom /etc/masterha/masterha_default.cnf..
Sat Aug 10 06:15:39 2013 - [info] Reading application defaultconfigurations from /etc/masterha/app1.cnf..
Sat Aug 10 06:15:39 2013 - [info] Reading server configurationsfrom /etc/masterha/app1.cnf..
Sat Aug 10 06:15:39 2013 - [info] Starting SSH connectiontests..
Sat Aug 10 06:15:42 2013 - [debug]
Sat Aug 10 06:15:39 2013 - [debug] Connecting viaSSH from root@192.168.8.120(192.168.8.120:22) toroot@192.168.8.121(192.168.8.121:22)..
Sat Aug 10 06:15:41 2013 -[debug]   ok.
Sat Aug 10 06:15:41 2013 - [debug] Connecting viaSSH from root@192.168.8.120(192.168.8.120:22) toroot@192.168.8.122(192.168.8.122:22)..
Sat Aug 10 06:15:42 2013 -[debug]   ok.
Sat Aug 10 06:15:43 2013 - [debug]
Sat Aug 10 06:15:40 2013 - [debug] Connecting viaSSH from root@192.168.8.121(192.168.8.121:22) toroot@192.168.8.120(192.168.8.120:22)..
Sat Aug 10 06:15:41 2013 -[debug]   ok.
Sat Aug 10 06:15:41 2013 - [debug] Connecting viaSSH from root@192.168.8.121(192.168.8.121:22) toroot@192.168.8.122(192.168.8.122:22)..
Sat Aug 10 06:15:43 2013 -[debug]   ok.
Sat Aug 10 06:15:44 2013 - [debug]
Sat Aug 10 06:15:40 2013 - [debug] Connecting viaSSH from root@192.168.8.122(192.168.8.122:22) toroot@192.168.8.120(192.168.8.120:22)..
Sat Aug 10 06:15:42 2013 -[debug]   ok.
Sat Aug 10 06:15:42 2013 - [debug] Connecting viaSSH from root@192.168.8.122(192.168.8.122:22) toroot@192.168.8.121(192.168.8.121:22)..
Sat Aug 10 06:15:44 2013 -[debug]   ok.
Sat Aug 10 06:15:44 2013 - [info] All SSH connection tests passedsuccessfully.

(8)测试复制

[root@centos3 mha]# masterha_check_repl--global_conf=/etc/masterha/masterha_default.cnf --conf=/etc/masterha/app1.cnf
Sat Aug 10 06:26:13 2013 - [info] Reading default configuratoinsfrom /etc/masterha/masterha_default.cnf..
Sat Aug 10 06:26:13 2013 - [info] Reading application defaultconfigurations from /etc/masterha/app1.cnf..
Sat Aug 10 06:26:13 2013 - [info] Reading server configurationsfrom /etc/masterha/app1.cnf..
Sat Aug 10 06:26:13 2013 - [info] MHA::MasterMonitor version0.53.
Sat Aug 10 06:26:13 2013 - [info] Dead Servers:
Sat Aug 10 06:26:13 2013 - [info] Alive Servers:
Sat Aug 10 06:26:13 2013 -[info]  192.168.8.120(192.168.8.120:3306)
Sat Aug 10 06:26:13 2013 -[info]  192.168.8.121(192.168.8.121:3306)
Sat Aug 10 06:26:13 2013 -[info]  192.168.8.122(192.168.8.122:3306)
Sat Aug 10 06:26:13 2013 - [info] Alive Slaves:
Sat Aug 10 06:26:13 2013 -[info]  192.168.8.121(192.168.8.121:3306) Version=5.5.29-log (oldest major version between slaves)log-bin:enabled
Sat Aug 10 06:26:13 2013 -[info]    Replicating from 192.168.8.120(192.168.8.120:3306)
Sat Aug 10 06:26:13 2013 -[info]    Primary candidate for the new Master (candidate_master isset)
Sat Aug 10 06:26:13 2013 -[info]  192.168.8.122(192.168.8.122:3306) Version=5.5.29-log (oldest major version between slaves)log-bin:enabled
Sat Aug 10 06:26:13 2013 -[info]    Replicating from 192.168.8.120(192.168.8.120:3306)
Sat Aug 10 06:26:13 2013 -[info]    Not candidate for the new Master (no_master is set)
Sat Aug 10 06:26:13 2013 - [info] Current Alive Master:192.168.8.120(192.168.8.120:3306)
Sat Aug 10 06:26:13 2013 - [info] Checking slaveconfigurations..
Sat Aug 10 06:26:13 2013 - [info] Checking replication filteringsettings..
Sat Aug 10 06:26:13 2013 - [info] binlog_do_db= ,binlog_ignore_db=
Sat Aug 10 06:26:13 2013 - [info] Replicationfiltering check ok.
Sat Aug 10 06:26:13 2013 - [info] Starting SSH connectiontests..
Sat Aug 10 06:26:17 2013 - [info] All SSH connection tests passedsuccessfully.
Sat Aug 10 06:26:17 2013 - [info] Checking MHA Node version..
Sat Aug 10 06:26:18 2013 - [info] Version checkok.
Sat Aug 10 06:26:18 2013 - [info] Checking SSH publickeyauthentication settings on the current master..
Sat Aug 10 06:26:18 2013 - [info] HealthCheck: SSH to 192.168.8.120is reachable.
Sat Aug 10 06:26:18 2013 - [info] Master MHA Node version is0.53.
Sat Aug 10 06:26:18 2013 - [info] Checking recovery scriptconfigurations on the current master..
Sat Aug 10 06:26:18 2013 -[info]   Executing command:save_binary_logs --command=test --start_pos=4--binlog_dir=/app/mysql--output_file=/app/mha/save_binary_logs_test --manager_version=0.53--start_file=mysql-bin.000010
Sat Aug 10 06:26:18 2013 -[info]   Connecting toroot@192.168.8.120(192.168.8.120)..
Creating /app/mha if notexists..   ok.
Checking output directory is accessible ornot..
   ok.
Binlog found at /app/mysql, up tomysql-bin.000010
Sat Aug 10 06:26:18 2013 - [info] Master setting check done.
Sat Aug 10 06:26:18 2013 - [info] Checking SSH publickeyauthentication and checking recovery script configurations on allalive slave servers..
Sat Aug 10 06:26:18 2013 -[info]   Executing command :apply_diff_relay_logs --command=test --slave_user=root--slave_host=192.168.8.121 --slave_ip=192.168.8.121--slave_port=3306 --workdir=/app/mha --target_version=5.5.29-log--manager_version=0.53--relay_log_info=/app/mysql/relay-log.info --relay_dir=/app/mysql/ --slave_pass=xxx
Sat Aug 10 06:26:18 2013 -[info]   Connecting toroot@192.168.8.121(192.168.8.121:22)..
Checking slave recovery environmentsettings..
    Opening/app/mysql/relay-log.info ... ok.
    Relay logfound at /app/mysql, up to mysql-relay-bin.000004
    Temporaryrelay log file is /app/mysql/mysql-relay-bin.000004
    Testingmysql connection and privileges.. done.
    Testingmysqlbinlog output.. done.
    Cleaning uptest file(s).. done.
Sat Aug 10 06:26:19 2013 -[info]   Executing command :apply_diff_relay_logs --command=test --slave_user=root--slave_host=192.168.8.122 --slave_ip=192.168.8.122--slave_port=3306 --workdir=/app/mha --target_version=5.5.29-log--manager_version=0.53--relay_log_info=/app/mysql/relay-log.info --relay_dir=/app/mysql/ --slave_pass=xxx
Sat Aug 10 06:26:19 2013 -[info]   Connecting toroot@192.168.8.122(192.168.8.122:22)..
Checking slave recovery environmentsettings..
    Opening/app/mysql/relay-log.info ... ok.
    Relay logfound at /app/mysql, up to mysql-relay-bin.000004
    Temporaryrelay log file is /app/mysql/mysql-relay-bin.000004
    Testingmysql connection and privileges.. done.
    Testingmysqlbinlog output.. done.
    Cleaning uptest file(s).. done.
Sat Aug 10 06:26:20 2013 - [info] Slaves settings check done.
Sat Aug 10 06:26:20 2013 - [info]
192.168.8.120 (current master)
+--192.168.8.121
+--192.168.8.122

Sat Aug 10 06:26:20 2013 - [info] Checking replication health on192.168.8.121..
Sat Aug 10 06:26:20 2013 - [info] ok.
Sat Aug 10 06:26:20 2013 - [info] Checking replication health on192.168.8.122..
Sat Aug 10 06:26:20 2013 - [info] ok.
Sat Aug 10 06:26:20 2013 - [info] Checkingmaster_ip_failover_script status:
Sat Aug 10 06:26:20 2013 -[info]  /scripts/master_ip_failover --command=status --ssh_user=root--orig_master_host=192.168.8.120 --orig_master_ip=192.168.8.120--orig_master_port=3306
Sat Aug 10 06:26:20 2013 - [info] OK.
Sat Aug 10 06:26:20 2013 - [warning] shutdown_script is notdefined.
Sat Aug 10 06:26:20 2013 - [info] Got exit code 0 (Not masterdead).

MySQL Replication Health is OK.

（9）启动management

[root@centos3 mysql]# nohup masterha_manager --global-conf=/etc/masterha/masterha_default.cnf --conf=/etc/masterha/app1.cnf

[root@sh-gs-dbmg0227 /]# masterha_check_status--conf=/etc/masterha/app1.cnf

app1 (pid:7127) is running(0:PING_OK),master:192.168.8.121
（10）在备节点，从节点执行定期删除中继日志

00 00 * * * /usr/local/bin/purge_relay_logs –user=root–password=123456 –disable_relay_log_purge >>/masterha/purge_relay_logs.log 2>&1

（11）测试关闭mysql主服务服务，主机宕机。主服务是否自动切换。

Sat Aug 10 06:29:36 2013 - [info] MHA::MasterMonitor version0.53.
Sat Aug 10 06:29:37 2013 - [info] Dead Servers:
Sat Aug 10 06:29:37 2013 - [info] Alive Servers:
Sat Aug 10 06:29:37 2013 -[info]  192.168.8.120(192.168.8.120:3306)
Sat Aug 10 06:29:37 2013 -[info]  192.168.8.121(192.168.8.121:3306)
Sat Aug 10 06:29:37 2013 -[info]  192.168.8.122(192.168.8.122:3306)
Sat Aug 10 06:29:37 2013 - [info] Alive Slaves:
Sat Aug 10 06:29:37 2013 -[info]  192.168.8.121(192.168.8.121:3306) Version=5.5.29-log (oldest major version between slaves)log-bin:enabled
Sat Aug 10 06:29:37 2013 -[info]    Replicating from 192.168.8.120(192.168.8.120:3306)
Sat Aug 10 06:29:37 2013 -[info]    Primary candidate for the new Master (candidate_master isset)
Sat Aug 10 06:29:37 2013 -[info]  192.168.8.122(192.168.8.122:3306) Version=5.5.29-log (oldest major version between slaves)log-bin:enabled
Sat Aug 10 06:29:37 2013 -[info]    Replicating from 192.168.8.120(192.168.8.120:3306)
Sat Aug 10 06:29:37 2013 -[info]    Not candidate for the new Master (no_master is set)
Sat Aug 10 06:29:37 2013 - [info] Current Alive Master:192.168.8.120(192.168.8.120:3306)
Sat Aug 10 06:29:37 2013 - [info] Checking slaveconfigurations..
Sat Aug 10 06:29:37 2013 - [info] Checking replication filteringsettings..
Sat Aug 10 06:29:37 2013 - [info] binlog_do_db= ,binlog_ignore_db=
Sat Aug 10 06:29:37 2013 - [info] Replicationfiltering check ok.
Sat Aug 10 06:29:37 2013 - [info] Starting SSH connectiontests..
Sat Aug 10 06:29:40 2013 - [info] All SSH connection tests passedsuccessfully.
Sat Aug 10 06:29:40 2013 - [info] Checking MHA Node version..
Sat Aug 10 06:29:41 2013 - [info] Version checkok.
Sat Aug 10 06:29:41 2013 - [info] Checking SSH publickeyauthentication settings on the current master..
Sat Aug 10 06:29:42 2013 - [info] HealthCheck: SSH to 192.168.8.120is reachable.
Sat Aug 10 06:29:42 2013 - [info] Master MHA Node version is0.53.
Sat Aug 10 06:29:42 2013 - [info] Checking recovery scriptconfigurations on the current master..
Sat Aug 10 06:29:42 2013 -[info]   Executing command:save_binary_logs --command=test --start_pos=4--binlog_dir=/app/mysql--output_file=/app/mha/save_binary_logs_test --manager_version=0.53--start_file=mysql-bin.000010
Sat Aug 10 06:29:42 2013 -[info]   Connecting toroot@192.168.8.120(192.168.8.120)..
Creating /app/mha if notexists..   ok.
Checking output directory is accessible ornot..
   ok.
Binlog found at /app/mysql, up tomysql-bin.000010
Sat Aug 10 06:29:42 2013 - [info] Master setting check done.
Sat Aug 10 06:29:42 2013 - [info] Checking SSH publickeyauthentication and checking recovery script configurations on allalive slave servers..
Sat Aug 10 06:29:42 2013 -[info]   Executing command :apply_diff_relay_logs --command=test --slave_user=root--slave_host=192.168.8.121 --slave_ip=192.168.8.121--slave_port=3306 --workdir=/app/mha --target_version=5.5.29-log--manager_version=0.53--relay_log_info=/app/mysql/relay-log.info --relay_dir=/app/mysql/ --slave_pass=xxx
Sat Aug 10 06:29:42 2013 -[info]   Connecting toroot@192.168.8.121(192.168.8.121:22)..
Checking slave recovery environmentsettings..
    Opening/app/mysql/relay-log.info ... ok.
    Relay logfound at /app/mysql, up to mysql-relay-bin.000004
    Temporaryrelay log file is /app/mysql/mysql-relay-bin.000004
    Testingmysql connection and privileges.. done.
    Testingmysqlbinlog output.. done.
    Cleaning uptest file(s).. done.
Sat Aug 10 06:29:43 2013 -[info]   Executing command :apply_diff_relay_logs --command=test --slave_user=root--slave_host=192.168.8.122 --slave_ip=192.168.8.122--slave_port=3306 --workdir=/app/mha --target_version=5.5.29-log--manager_version=0.53--relay_log_info=/app/mysql/relay-log.info --relay_dir=/app/mysql/ --slave_pass=xxx
Sat Aug 10 06:29:43 2013 -[info]   Connecting toroot@192.168.8.122(192.168.8.122:22)..
Checking slave recovery environmentsettings..
    Opening/app/mysql/relay-log.info ... ok.
    Relay logfound at /app/mysql, up to mysql-relay-bin.000004
    Temporaryrelay log file is /app/mysql/mysql-relay-bin.000004
    Testingmysql connection and privileges.. done.
    Testingmysqlbinlog output.. done.
    Cleaning uptest file(s).. done.
Sat Aug 10 06:29:43 2013 - [info] Slaves settings check done.
Sat Aug 10 06:29:43 2013 - [info]
192.168.8.120 (current master)
+--192.168.8.121
+--192.168.8.122

Sat Aug 10 06:29:43 2013 - [info] Checkingmaster_ip_failover_script status:
Sat Aug 10 06:29:43 2013 -[info]  /scripts/master_ip_failover --command=status --ssh_user=root--orig_master_host=192.168.8.120 --orig_master_ip=192.168.8.120--orig_master_port=3306
Sat Aug 10 06:29:44 2013 - [info] OK.
Sat Aug 10 06:29:44 2013 - [warning] shutdown_script is notdefined.
Sat Aug 10 06:29:44 2013 - [info] Set master ping interval 1seconds.
Sat Aug 10 06:29:44 2013 - [info] Set secondary check script:masterha_secondary_check -s 192.168.8.121 -s 192.168.8.122
Sat Aug 10 06:29:44 2013 - [info] Starting ping health check on192.168.8.120(192.168.8.120:3306)..
Sat Aug 10 06:29:44 2013 - [info] Ping(SELECT) succeeded, waitinguntil MySQL doesn't respond..
Sat Aug 10 06:31:27 2013 - [warning] Got error on MySQL selectping: 2006 (MySQL server has gone away)
Sat Aug 10 06:31:27 2013 - [info] Executing seconary network checkscript: masterha_secondary_check -s 192.168.8.121 -s192.168.8.122 --user=root --master_host=192.168.8.120 --master_ip=192.168.8.120 --master_port=3306
Sat Aug 10 06:31:27 2013 - [info] Executing SSH check script:save_binary_logs --command=test --start_pos=4--binlog_dir=/app/mysql--output_file=/app/mha/save_binary_logs_test --manager_version=0.53--binlog_prefix=mysql-bin
Sat Aug 10 06:31:28 2013 - [info] HealthCheck: SSH to 192.168.8.120is reachable.
Sat Aug 10 06:31:28 2013 - [warning] Got error on MySQL connect:2013 (Lost connection to MySQL server at 'reading initialcommunication packet', system error: 111)
Sat Aug 10 06:31:28 2013 - [warning] Connection failed 1time(s)..
Monitoring server 192.168.8.121 is reachable, Master is notreachable from 192.168.8.121. OK.
Monitoring server 192.168.8.122 is reachable, Master is notreachable from 192.168.8.122. OK.
Sat Aug 10 06:31:29 2013 - [info] Master is not reachable from allother monitoring servers. Failover should start.
Sat Aug 10 06:31:29 2013 - [warning] Got error on MySQL connect:2013 (Lost connection to MySQL server at 'reading initialcommunication packet', system error: 111)
Sat Aug 10 06:31:29 2013 - [warning] Connection failed 2time(s)..
Sat Aug 10 06:31:30 2013 - [warning] Got error on MySQL connect:2013 (Lost connection to MySQL server at 'reading initialcommunication packet', system error: 111)
Sat Aug 10 06:31:30 2013 - [warning] Connection failed 3time(s)..
Sat Aug 10 06:31:30 2013 - [warning] Master is not reachable fromhealth checker!
Sat Aug 10 06:31:30 2013 - [warning] Master192.168.8.120(192.168.8.120:3306) is not reachable!
Sat Aug 10 06:31:30 2013 - [warning] SSH is reachable.
Sat Aug 10 06:31:30 2013 - [info] Connecting to a master serverfailed. Reading configuration file/etc/masterha/masterha_default.cnf and /etc/masterha/app1.cnfagain, and trying to connect to all servers to check serverstatus..
Sat Aug 10 06:31:30 2013 - [info] Reading default configuratoinsfrom /etc/masterha/masterha_default.cnf..
Sat Aug 10 06:31:30 2013 - [info] Reading application defaultconfigurations from /etc/masterha/app1.cnf..
Sat Aug 10 06:31:30 2013 - [info] Reading server configurationsfrom /etc/masterha/app1.cnf..
Sat Aug 10 06:31:30 2013 - [info] Dead Servers:
Sat Aug 10 06:31:30 2013 -[info]  192.168.8.120(192.168.8.120:3306)
Sat Aug 10 06:31:30 2013 - [info] Alive Servers:
Sat Aug 10 06:31:30 2013 -[info]  192.168.8.121(192.168.8.121:3306)
Sat Aug 10 06:31:30 2013 -[info]  192.168.8.122(192.168.8.122:3306)
Sat Aug 10 06:31:30 2013 - [info] Alive Slaves:
Sat Aug 10 06:31:30 2013 -[info]  192.168.8.121(192.168.8.121:3306) Version=5.5.29-log (oldest major version between slaves)log-bin:enabled
Sat Aug 10 06:31:30 2013 -[info]    Replicating from 192.168.8.120(192.168.8.120:3306)
Sat Aug 10 06:31:30 2013 -[info]    Primary candidate for the new Master (candidate_master isset)
Sat Aug 10 06:31:30 2013 -[info]  192.168.8.122(192.168.8.122:3306) Version=5.5.29-log (oldest major version between slaves)log-bin:enabled
Sat Aug 10 06:31:30 2013 -[info]    Replicating from 192.168.8.120(192.168.8.120:3306)
Sat Aug 10 06:31:30 2013 -[info]    Not candidate for the new Master (no_master is set)
Sat Aug 10 06:31:30 2013 - [info] Checking slaveconfigurations..
Sat Aug 10 06:31:30 2013 - [info] Checking replication filteringsettings..
Sat Aug 10 06:31:30 2013 - [info] Replicationfiltering check ok.
Sat Aug 10 06:31:30 2013 - [info] Master is down!
Sat Aug 10 06:31:30 2013 - [info] Terminating monitoringscript.
Sat Aug 10 06:31:30 2013 - [info] Got exit code 20 (Masterdead).
Sat Aug 10 06:31:30 2013 - [info] MHA::MasterFailover version0.53.
Sat Aug 10 06:31:30 2013 - [info] Starting master failover.
Sat Aug 10 06:31:30 2013 - [info]
Sat Aug 10 06:31:30 2013 - [info] * Phase 1: Configuration CheckPhase..
Sat Aug 10 06:31:30 2013 - [info]
Sat Aug 10 06:31:30 2013 - [info] Dead Servers:
Sat Aug 10 06:31:30 2013 -[info]  192.168.8.120(192.168.8.120:3306)
Sat Aug 10 06:31:30 2013 - [info] Checking master reachability viamysql(double check)..
Sat Aug 10 06:31:30 2013 - [info] ok.
Sat Aug 10 06:31:30 2013 - [info] Alive Servers:
Sat Aug 10 06:31:30 2013 -[info]  192.168.8.121(192.168.8.121:3306)
Sat Aug 10 06:31:30 2013 -[info]  192.168.8.122(192.168.8.122:3306)
Sat Aug 10 06:31:30 2013 - [info] Alive Slaves:
Sat Aug 10 06:31:30 2013 -[info]  192.168.8.121(192.168.8.121:3306) Version=5.5.29-log (oldest major version between slaves)log-bin:enabled
Sat Aug 10 06:31:30 2013 -[info]    Replicating from 192.168.8.120(192.168.8.120:3306)
Sat Aug 10 06:31:30 2013 -[info]    Primary candidate for the new Master (candidate_master isset)
Sat Aug 10 06:31:30 2013 -[info]  192.168.8.122(192.168.8.122:3306) Version=5.5.29-log (oldest major version between slaves)log-bin:enabled
Sat Aug 10 06:31:30 2013 -[info]    Replicating from 192.168.8.120(192.168.8.120:3306)
Sat Aug 10 06:31:30 2013 -[info]    Not candidate for the new Master (no_master is set)
Sat Aug 10 06:31:30 2013 - [info] ** Phase 1: Configuration CheckPhase completed.
Sat Aug 10 06:31:30 2013 - [info]
Sat Aug 10 06:31:30 2013 - [info] * Phase 2: Dead Master ShutdownPhase..
Sat Aug 10 06:31:30 2013 - [info]
Sat Aug 10 06:31:30 2013 - [info] Forcing shutdown so thatapplications never connect to the current master..
Sat Aug 10 06:31:30 2013 - [info] Executing master IPdeactivatation script:
Sat Aug 10 06:31:30 2013 -[info]  /scripts/master_ip_failover --orig_master_host=192.168.8.120--orig_master_ip=192.168.8.120 --orig_master_port=3306--command=stopssh --ssh_user=root
Sat Aug 10 06:31:31 2013 - [info] done.
Sat Aug 10 06:31:31 2013 - [warning] shutdown_script is not set.Skipping explicit shutting down of the dead master.
Sat Aug 10 06:31:31 2013 - [info] * Phase 2: Dead Master ShutdownPhase completed.
Sat Aug 10 06:31:31 2013 - [info]
Sat Aug 10 06:31:31 2013 - [info] * Phase 3: Master RecoveryPhase..
Sat Aug 10 06:31:31 2013 - [info]
Sat Aug 10 06:31:31 2013 - [info] * Phase 3.1: Getting LatestSlaves Phase..
Sat Aug 10 06:31:31 2013 - [info]
Sat Aug 10 06:31:31 2013 - [info] The latest binary logfile/position on all slaves is mysql-bin.000010:107
Sat Aug 10 06:31:31 2013 - [info] Latest slaves (Slaves thatreceived relay log files to the latest):
Sat Aug 10 06:31:31 2013 -[info]  192.168.8.121(192.168.8.121:3306) Version=5.5.29-log (oldest major version between slaves)log-bin:enabled
Sat Aug 10 06:31:31 2013 -[info]    Replicating from 192.168.8.120(192.168.8.120:3306)
Sat Aug 10 06:31:31 2013 -[info]    Primary candidate for the new Master (candidate_master isset)
Sat Aug 10 06:31:31 2013 -[info]  192.168.8.122(192.168.8.122:3306) Version=5.5.29-log (oldest major version between slaves)log-bin:enabled
Sat Aug 10 06:31:31 2013 -[info]    Replicating from 192.168.8.120(192.168.8.120:3306)
Sat Aug 10 06:31:31 2013 -[info]    Not candidate for the new Master (no_master is set)
Sat Aug 10 06:31:31 2013 - [info] The oldest binary logfile/position on all slaves is mysql-bin.000010:107
Sat Aug 10 06:31:31 2013 - [info] Oldest slaves:
Sat Aug 10 06:31:31 2013 -[info]  192.168.8.121(192.168.8.121:3306) Version=5.5.29-log (oldest major version between slaves)log-bin:enabled
Sat Aug 10 06:31:31 2013 -[info]    Replicating from 192.168.8.120(192.168.8.120:3306)
Sat Aug 10 06:31:31 2013 -[info]    Primary candidate for the new Master (candidate_master isset)
Sat Aug 10 06:31:31 2013 -[info]  192.168.8.122(192.168.8.122:3306) Version=5.5.29-log (oldest major version between slaves)log-bin:enabled
Sat Aug 10 06:31:31 2013 -[info]    Replicating from 192.168.8.120(192.168.8.120:3306)
Sat Aug 10 06:31:31 2013 -[info]    Not candidate for the new Master (no_master is set)
Sat Aug 10 06:31:31 2013 - [info]
Sat Aug 10 06:31:31 2013 - [info] * Phase 3.2: Saving Dead Master'sBinlog Phase..
Sat Aug 10 06:31:31 2013 - [info]
Sat Aug 10 06:31:31 2013 - [info] Fetching dead master's binarylogs..
Sat Aug 10 06:31:31 2013 - [info] Executing command on the deadmaster 192.168.8.120(192.168.8.120:3306): save_binary_logs--command=save --start_file=mysql-bin.000010 --start_pos=107 --binlog_dir=/app/mysql--output_file=/app/mha/saved_master_binlog_from_192.168.8.120_3306_20130810063130.binlog--handle_raw_binlog=1 --disable_log_bin=0--manager_version=0.53
Creating /app/mha if notexists..   ok.
Concat binary/relay logs from mysql-bin.000010pos 107 to mysql-bin.000010 EOF into/app/mha/saved_master_binlog_from_192.168.8.120_3306_20130810063130.binlog..
Dumping binlog format description event, fromposition 0 to 107.. ok.
Dumping effective binlog data from/app/mysql/mysql-bin.000010 position 107 to tail(126).. ok.
Concat succeeded.
Sat Aug 10 06:31:32 2013 - [info] scp fromroot@192.168.8.120:/app/mha/saved_master_binlog_from_192.168.8.120_3306_20130810063130.binlogtolocal:/app/mha/saved_master_binlog_from_192.168.8.120_3306_20130810063130.binlogsucceeded.
Sat Aug 10 06:31:32 2013 - [info] HealthCheck: SSH to 192.168.8.121is reachable.
Sat Aug 10 06:31:33 2013 - [info] HealthCheck: SSH to 192.168.8.122is reachable.
Sat Aug 10 06:31:33 2013 - [info]
Sat Aug 10 06:31:33 2013 - [info] * Phase 3.3: Determining NewMaster Phase..
Sat Aug 10 06:31:33 2013 - [info]
Sat Aug 10 06:31:33 2013 - [info] Finding the latest slave that hasall relay logs for recovering other slaves..
Sat Aug 10 06:31:33 2013 - [info] All slaves received relay logs tothe same position. No need to resync each other.
Sat Aug 10 06:31:33 2013 - [info] Searching new master fromslaves..
Sat Aug 10 06:31:33 2013 - [info] Candidatemasters from the configuration file:
Sat Aug 10 06:31:33 2013 -[info]  192.168.8.121(192.168.8.121:3306) Version=5.5.29-log (oldest major version between slaves)log-bin:enabled
Sat Aug 10 06:31:33 2013 -[info]    Replicating from 192.168.8.120(192.168.8.120:3306)
Sat Aug 10 06:31:33 2013 -[info]    Primary candidate for the new Master (candidate_master isset)
Sat Aug 10 06:31:33 2013 - [info] Non-candidatemasters:
Sat Aug 10 06:31:33 2013 -[info]  192.168.8.122(192.168.8.122:3306) Version=5.5.29-log (oldest major version between slaves)log-bin:enabled
Sat Aug 10 06:31:33 2013 -[info]    Replicating from 192.168.8.120(192.168.8.120:3306)
Sat Aug 10 06:31:33 2013 -[info]    Not candidate for the new Master (no_master is set)
Sat Aug 10 06:31:33 2013 - [info] Searching fromcandidate_master slaves which have received the latest relay logevents..
Sat Aug 10 06:31:33 2013 - [info] New master is192.168.8.121(192.168.8.121:3306)
Sat Aug 10 06:31:33 2013 - [info] Starting master failover..
Sat Aug 10 06:31:33 2013 - [info]
From:
192.168.8.120 (current master)
+--192.168.8.121
+--192.168.8.122

To:
192.168.8.121 (new master)
+--192.168.8.122
Sat Aug 10 06:31:33 2013 - [info]
Sat Aug 10 06:31:33 2013 - [info] * Phase 3.3: New Master Diff LogGeneration Phase..
Sat Aug 10 06:31:33 2013 - [info]
Sat Aug 10 06:31:33 2013 - [info] This server hasall relay logs. No need to generate diff files from the latestslave.
Sat Aug 10 06:31:33 2013 - [info] Sending binlog..
Sat Aug 10 06:31:34 2013 - [info] scp fromlocal:/app/mha/saved_master_binlog_from_192.168.8.120_3306_20130810063130.binlogtoroot@192.168.8.121:/app/mha/saved_master_binlog_from_192.168.8.120_3306_20130810063130.binlogsucceeded.
Sat Aug 10 06:31:34 2013 - [info]
Sat Aug 10 06:31:34 2013 - [info] * Phase 3.4: Master Log ApplyPhase..
Sat Aug 10 06:31:34 2013 - [info]
Sat Aug 10 06:31:34 2013 - [info] *NOTICE: If any error happensfrom this phase, manual recovery is needed.
Sat Aug 10 06:31:34 2013 - [info] Starting recovery on192.168.8.121(192.168.8.121:3306)..
Sat Aug 10 06:31:34 2013 - [info] Generatingdiffs succeeded.
Sat Aug 10 06:31:34 2013 - [info] Waiting until all relay logs areapplied.
Sat Aug 10 06:31:34 2013 - [info] done.
Sat Aug 10 06:31:34 2013 - [info] Getting slave status..
Sat Aug 10 06:31:34 2013 - [info] This slave(192.168.8.121)'sExec_Master_Log_Pos equals toRead_Master_Log_Pos(mysql-bin.000010:107). No need to recover fromExec_Master_Log_Pos.
Sat Aug 10 06:31:34 2013 - [info] Connecting to the target slavehost 192.168.8.121, running recover script..
Sat Aug 10 06:31:34 2013 - [info] Executing command:apply_diff_relay_logs --command=apply --slave_user=root--slave_host=192.168.8.121--slave_ip=192.168.8.121 --slave_port=3306--apply_files=/app/mha/saved_master_binlog_from_192.168.8.120_3306_20130810063130.binlog--workdir=/app/mha --target_version=5.5.29-log--timestamp=20130810063130 --handle_raw_binlog=1--disable_log_bin=0 --manager_version=0.53 --slave_pass=xxx
Sat Aug 10 06:31:35 2013 - [info]
Applying differential binary/relay log files/app/mha/saved_master_binlog_from_192.168.8.120_3306_20130810063130.binlogon 192.168.8.121:3306. This may take long time...
Applying log files succeeded.
Sat Aug 10 06:31:35 2013 - [info] All relay logswere successfully applied.
Sat Aug 10 06:31:35 2013 - [info] Getting new master's binlog nameand position..
Sat Aug 10 06:31:35 2013 - [info] mysql-bin.000008:107
Sat Aug 10 06:31:35 2013 - [info] All otherslaves should start replication from here. Statement should be:CHANGE MASTER TO MASTER_HOST='192.168.8.121', MASTER_PORT=3306,MASTER_LOG_FILE='mysql-bin.000008', MASTER_LOG_POS=107,MASTER_USER='repl', MASTER_PASSWORD='xxx';
Sat Aug 10 06:31:35 2013 - [info] Executing master IP activatescript:
Sat Aug 10 06:31:35 2013 -[info]  /scripts/master_ip_failover --command=start --ssh_user=root--orig_master_host=192.168.8.120 --orig_master_ip=192.168.8.120--orig_master_port=3306 --new_master_host=192.168.8.121--new_master_ip=192.168.8.121--new_master_port=3306
Can't call method "prepare" on an undefined value at/usr/local/share/perl5/MHA/DBHelper.pm line 335.
Sat Aug 10 06:31:35 2013 -[error][/usr/local/share/perl5/MHA/MasterFailover.pm,ln1259] Failed to activate master IP address for192.168.8.121 with return code 10:0
Sat Aug 10 06:31:35 2013 - [warning] Proceeding.
Sat Aug 10 06:31:35 2013 - [info] Setting read_only=0 on192.168.8.121(192.168.8.121:3306)..
Sat Aug 10 06:31:35 2013 - [info] ok.
Sat Aug 10 06:31:35 2013 - [info] ** Finished master recoverysuccessfully.
Sat Aug 10 06:31:35 2013 - [info] * Phase 3: Master Recovery Phasecompleted.
Sat Aug 10 06:31:35 2013 - [info]
Sat Aug 10 06:31:35 2013 - [info] * Phase 4: Slaves RecoveryPhase..
Sat Aug 10 06:31:35 2013 - [info]
Sat Aug 10 06:31:35 2013 - [info] * Phase 4.1: Starting ParallelSlave Diff Log Generation Phase..
Sat Aug 10 06:31:35 2013 - [info]
Sat Aug 10 06:31:35 2013 - [info] -- Slave diff file generation onhost 192.168.8.122(192.168.8.122:3306) started, pid: 11342. Checktmp log /app/mha/192.168.8.122_3306_20130810063130.log if it takestime..
Sat Aug 10 06:31:35 2013 - [info]
Sat Aug 10 06:31:35 2013 - [info] Log messages from 192.168.8.122...
Sat Aug 10 06:31:35 2013 - [info]
Sat Aug 10 06:31:35 2013 - [info] This server hasall relay logs. No need to generate diff files from the latestslave.
Sat Aug 10 06:31:35 2013 - [info] End of log messages from192.168.8.122.
Sat Aug 10 06:31:35 2013 - [info] --192.168.8.122(192.168.8.122:3306) has the latest relay logevents.
Sat Aug 10 06:31:35 2013 - [info] Generating relay diff files fromthe latest slave succeeded.
Sat Aug 10 06:31:35 2013 - [info]
Sat Aug 10 06:31:35 2013 - [info] * Phase 4.2: Starting ParallelSlave Log Apply Phase..
Sat Aug 10 06:31:35 2013 - [info]
Sat Aug 10 06:31:35 2013 - [info] -- Slave recovery on host192.168.8.122(192.168.8.122:3306) started, pid: 11344. Check tmplog /app/mha/192.168.8.122_3306_20130810063130.log if it takestime..
Sat Aug 10 06:31:36 2013 - [info]
Sat Aug 10 06:31:36 2013 - [info] Log messages from 192.168.8.122...
Sat Aug 10 06:31:36 2013 - [info]
Sat Aug 10 06:31:35 2013 - [info] Sending binlog..
Sat Aug 10 06:31:35 2013 - [info] scp fromlocal:/app/mha/saved_master_binlog_from_192.168.8.120_3306_20130810063130.binlogtoroot@192.168.8.122:/app/mha/saved_master_binlog_from_192.168.8.120_3306_20130810063130.binlogsucceeded.
Sat Aug 10 06:31:35 2013 - [info] Starting recovery on192.168.8.122(192.168.8.122:3306)..
Sat Aug 10 06:31:35 2013 - [info] Generatingdiffs succeeded.
Sat Aug 10 06:31:35 2013 - [info] Waiting until all relay logs areapplied.
Sat Aug 10 06:31:35 2013 - [info] done.
Sat Aug 10 06:31:35 2013 - [info] Getting slave status..
Sat Aug 10 06:31:35 2013 - [info] This slave(192.168.8.122)'sExec_Master_Log_Pos equals toRead_Master_Log_Pos(mysql-bin.000010:107). No need to recover fromExec_Master_Log_Pos.
Sat Aug 10 06:31:35 2013 - [info] Connecting to the target slavehost 192.168.8.122, running recover script..
Sat Aug 10 06:31:35 2013 - [info] Executing command:apply_diff_relay_logs --command=apply --slave_user=root--slave_host=192.168.8.122--slave_ip=192.168.8.122 --slave_port=3306--apply_files=/app/mha/saved_master_binlog_from_192.168.8.120_3306_20130810063130.binlog--workdir=/app/mha --target_version=5.5.29-log--timestamp=20130810063130 --handle_raw_binlog=1--disable_log_bin=0 --manager_version=0.53 --slave_pass=xxx
Sat Aug 10 06:31:36 2013 - [info]
Applying differential binary/relay log files/app/mha/saved_master_binlog_from_192.168.8.120_3306_20130810063130.binlogon 192.168.8.122:3306. This may take long time...
Applying log files succeeded.
Sat Aug 10 06:31:36 2013 - [info] All relay logswere successfully applied.
Sat Aug 10 06:31:36 2013 - [info] Resetting slave192.168.8.122(192.168.8.122:3306) and starting replication from thenew master 192.168.8.121(192.168.8.121:3306)..
Sat Aug 10 06:31:36 2013 - [info] Executed CHANGEMASTER.
Sat Aug 10 06:31:36 2013 - [info] Slavestarted.
Sat Aug 10 06:31:36 2013 - [info] End of log messages from192.168.8.122.
Sat Aug 10 06:31:36 2013 - [info] -- Slave recovery on host192.168.8.122(192.168.8.122:3306) succeeded.
Sat Aug 10 06:31:36 2013 - [info] All new slave servers recoveredsuccessfully.
Sat Aug 10 06:31:36 2013 - [info]
Sat Aug 10 06:31:36 2013 - [info] * Phase 5: New master cleanupphease..
Sat Aug 10 06:31:36 2013 - [info]
Sat Aug 10 06:31:36 2013 - [info] Resetting slave info on the newmaster..
Sat Aug 10 06:31:36 2013 - [info] 192.168.8.121:Resetting slave info succeeded.
Sat Aug 10 06:31:36 2013 - [info] Master failover to192.168.8.121(192.168.8.121:3306) completed successfully.
Sat Aug 10 06:31:36 2013 - [info]

----- Failover Report -----

app1: MySQL Master failover 192.168.8.120 to 192.168.8.121succeeded

Master 192.168.8.120 is down!

Check MHA Manager logs at centos3:/app/mha/manager.log fordetails.

Started automated(non-interactive) failover.
Invalidated master IP address on 192.168.8.120.
The latest slave 192.168.8.121(192.168.8.121:3306) has all relaylogs for recovery.
Selected 192.168.8.121 as a new master.
192.168.8.121: OK: Applying all logs succeeded.
Failed to activate master IP address for 192.168.8.121 with returncode 10:0
192.168.8.122: This host has the latest relay log events.
Generating relay diff files from the latest slave succeeded.
192.168.8.122: OK: Applying all logs succeeded. Slave started,replicating from 192.168.8.121.
192.168.8.121: Resetting slave info succeeded.
Master failover to 192.168.8.121(192.168.8.121:3306) completedsuccessfully.
Sat Aug 10 06:31:36 2013 - [info] Sending mail..

以上是mha成功进行主从切换

（11）主从安装keepalived ，实现虚拟ip漂移

[root@centos1 src]# tar -zxf keepalived-1.2.2.tar.gz

[root@centos1 src]# cd keepalived-1.2.2

[root@centos1 keepalived-1.2.2]# ./configure

[root@centos1 keepalived-1.2.2]# make && makeinstall

[root@centos1 keepalived-1.2.2]# mkdir /etc/keepalived

[root@centos1 keepalived-1.2.2]# cp/usr/local/etc/keepalived/keepalived.conf /etc/keepalived/

[root@centos1 keepalived-1.2.2]# cp/usr/local/etc/rc.d/init.d/keepalived /etc/init.d/

[root@centos1 keepalived-1.2.2]# cp/usr/local/etc/sysconfig/keepalived /etc/sysconfig/

(12)主从配置keepalived ，主从的priority设置不同，权重大的为主状态

[root@centos1 keepalived-1.2.2]# more/etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {

   router_id mha
}

vrrp_script check_run {
    script"/root/check_mysql.sh"
    interval1
}
vrrp_instance VI_1 {
    stateBACKUP
    interfaceeth0
   virtual_router_id 51
    priority 100
    advert_int1
   authentication {
       auth_type PASS
       auth_pass 1111
    }

track_script {
        check_run
    }

   virtual_ipaddress {
       192.168.8.123

    }
}

[root@centos1 ~] vi check_mysql.sh

#!/bin/bash
MYSQL=/usr/local/mysql/bin/mysql
MYSQL_HOST=127.0.0.1
MYSQL_USER=root
MYSQL_PASSWORD=
CHECK_TIME=3
#mysql is working MYSQL_OK is 1 , mysql downMYSQL_OK is 0
MYSQL_OK=1
function check_mysql_helth (){
$MYSQL -h $MYSQL_HOST -u $MYSQL_USER -e "show status;">/dev/null 2>&1
if [ $? = 0 ] ;then
    MYSQL_OK=1
else
    MYSQL_OK=0
fi
    return $MYSQL_OK
}
while [ $CHECK_TIME -ne 0 ]
do
    let "CHECK_TIME -= 1"
    check_mysql_helth
if [ $MYSQL_OK = 1 ] ; then
    CHECK_TIME=0
    exit 0
fi

if [ $MYSQL_OK -eq 0 ] && [ $CHECK_TIME-eq 0 ]
then
    pkill keepalived
exit 1
fi
sleep 1
done

此脚本不能检测MySQL 正常运行，不能正常，就关闭keepalived,释放虚拟IP.

在发现主库down,根据输出日志信息，mha做一下操作
1：发现主库不可用，masterha_secondary_check 和save_binary_logs两脚本检测从库是否可达，检测主库binlog 是否可用。并发出failover 信号。
2:检查主库server1，三次不可通后确认无法使用，确定server1 的mysql 不可用，但主机ssh 可达。
3:开始Dead Master Shutdown Phase 阶段，调用master_ip_failover 脚本.
4：开启master_recovry_phase，找到所有slaves 己读到的binlog位置，若生产环境下从库读取不一致，则以最新的为准从dead_master上获取最新的binlog。将dead_master存在但未同步到从上的binlog 保存下来
5：开始Determining New Master Phase段段，寻找备用master，原则为设置candidate_master，拥有最新的binlog。标记为no_master的server永不会设为备主。然后开始failover，mysql 架构由120( 主库) ，121（备主）,122为从库。
6：开始New Master Diff Log Generation Phase阶段，应用差异日志到新主库。确定新主当前的master_log_file,master_log_pos。
7：应用差异日志到所有从库，应用完成后，关闭新主server2 的read_only 选项，对应用可读可写。
8：在所有从库上执行change master。清除新主库的slave 信息，resetslave，完成failover。
9：最后调用send_report 脚本，发送报警短信或是邮件通知dba.

本站仅提供存储服务，所有内容均由用户发布，如发现有害或侵权内容，请点击举报。