In Part 1 we successfully migrated the data within the cluster to a second datacenter, completing this without downtime to the cluster.
Now we will look at decommissioning the original RS_UK datacenter.
Preparing for Decommission
Firstly we need to ensure all the applications are now pointing directly to the new datacenter GL_EU.
Once we know all applications are writing directly to the new DC, it is worth doing a full repair on each node in the new DC to make sure all the data has migrated correctly from the original DC.
While the repair is happening, the cassandra.yaml needs to be updated to repoint the seeds to nodes within the GL_EU DC.
Altering Keyspace Replication
Once the repair is complete and cassandra.yaml updated, it is time to alter the replication factor of all keyspaces (including the system ones) to only replicate to the GL_EU DC:
ALTER KEYSPACE user_keyspace1 WITH replication = {
'class': 'NetworkTopologyStrategy', 'GL_EU': 3
};
ALTER KEYSPACE user_keyspace2 WITH replication = {
'class': 'NetworkTopologyStrategy', 'GL_EU': 3
};
ALTER KEYSPACE user_keyspace3 WITH replication = {
'class': 'NetworkTopologyStrategy', 'GL_EU': 3
};
We now have the data disconnected, and data written to GL_EU will not replicate back to RS_UK.
Removing the Old Nodes
At this stage the nodes are still connected together across the datacenters, even though data is not flowing, so we need to disconnect and shut down the nodes in RS_UK.
Shut down each node in the RS_UK datacenter in turn, then use the nodetool removenode command to remove the nodes from the cluster. This should be run on one of the nodes within the GL_EU cluster.
The nodetool removenode requires the host ID of the node to be removed, which can be found by running nodetool status:
Datacenter: RS_UK
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Host ID Rack
UN 10.29.30.29 11.66 GB 256 ab479afd-c754-47f7-92fb-47790d734ac9 rack1
UN 10.29.30.33 12.32 GB 256 9aa1c5c5-c6cd-4267-ba68-c6bd8b2ac460 rack2
UN 10.29.30.34 12.16 GB 256 db454258-ac73-4a8a-9c75-226108c66889 rack3
Datacenter: GL_EU
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Host ID Rack
UN 10.131.134.35 13.19 GB 256 114b4a37-7d69-40e5-988b-a4c998e7a02a rack1
UN 10.131.134.39 12.14 GB 256 4173fc2a-e65c-43aa-baa4-a5eefe0ceb60 rack2
UN 10.131.134.42 12.97 GB 256 8b5dde02-1ff1-48cc-9900-6d8f2bb339bf rack3
Now remove each RS_UK node in turn:
nodetool removenode ab479afd-c754-47f7-92fb-47790d734ac9
nodetool removenode 9aa1c5c5-c6cd-4267-ba68-c6bd8b2ac460
nodetool removenode db454258-ac73-4a8a-9c75-226108c66889
Run these commands in turn. Once completed, the cluster will be a single DC cluster again, in the new datacenter.
The cluster has now been successfully migrated to the new datacenter with the old datacenter shut down.
In the third and final part of this series we will look at the problems that may occur.