- 2 minutes to read
Disaster recover test
- 2 minutes to read
- Have two senhasegura instances
- Both instances in the cluster and working correctly (check Configure a cluster)
- Snapshot from both instances
This test is for two cluster members.
At the bottom of each member, you can check if they have the following tags:
Secondary member (Contingency)
For the test to work, it will be necessary to make an abrupt shutdown in the primary member.
Remember always to create a snapshot before this test.
Since we are doing an abrupt shutdown, this can cause damage to the instance.
Go to the Orbit ➔ Replication ➔ Status.
In the Database cluster table, check if the Cluster size matches the number of members in the cluster, and in the Cluster members table, remember that this test will be 2.
Configure Recovery settings
In the secondary member, go to cluster Orbit ➔ Settings ➔ Recovery.
- Insert the IPs allowed to perform a system recovery. You can use an IP or a subnet mask, e.g., 192.168.1.2, 192.168.1.0/24.
Wildcards are not allowed, use * will not work.
This list will allow the users to see the button "Assume as master" turning the secondary member into a Master and functional instance.
Execute DR Test
Force an abrupt shutdown in your primary member.Caution
Be aware that it must be an abrupt shutdown; otherwise, the cluster will know that one of the members turned it off, and the secondary member will not show the Recovery page.
Once the primary member is down due to unexpected behavior, the secondary member will enter in split-brain, blocking any database change until manual instructions, and show the Recovery page on the web application.
At the web application, click the Assume as master button, click Yes to confirm, setting the secondary member as a new Master.
This process can take a few minutes.Info
If the button does not show, check the Configure Recovery settings to see if there are the correct IPs.
Once the Orbit Web interface is available in the secondary member, check the tag, showing that this instance is now the Master.
To access the other senhasegura modules, enabling the application is necessary. Go to Orbit ➔ Settings ➔ Application toggle the "Enable application" button, and click save at the button. It showing green means that the application is now enabled.
Log off and log in again to now access the other modules.
Now all the senhasegura features are available and functional in the DR secondary member.
Primary Member Recovery
After the steps executed in the step before, perform the following actions:
Turn the primary member on, and wait for the synchronization with the other member cluster database. This can take a few minutes.
The primary member will identify that something went wrong, and now the second member is the master and automatically synchronize the new information between the members.
Once done, the login page should be displayed in the primary web application interface.
Log in to the primary member web application and click Assume as master to return the primary member as Master.
In the secondary member, go to Orbit ➔ Settings ➔ Application toggle the "Enable application" button to disable the application, and click save at the button. Make sure that it is not showing green.
Alternative Primary Member Recovery via SSH
Log in using SSH at port 59022 to the primary member with the mt4adm user.
Execute the command
sudo orbit application status, to check the information Main and primary.
$ sudo orbit application status Application: Active Replication: Active Instance: Cluster Primary: member2 Main: No
- Execute the command
sudo orbit application master.
$ sudo orbit application master Application: Active Replication: Active Instance: Cluster Primary: member1 Main: Yes
This command will set the primary member to Main and redefine it to Primary.