Moving Cassandra Clusters without Downtime – Part 1

This is a post to go with a recent presentation at the DataStax Accelerate conference, and describes the process used to move clusters from Rackspace to Google Cloud.

Our requirement was to move multiple clusters, one at a time, from the UK to GCP in Belgium without any downtime.

Within this post the original datacenter is called RS_UK and the new datacenter will be GL_EU.

Application Pre-Requisites

Before the data move started we need to ensure the application was following these pre-requisites:

• Local consistency types for reading and writing: LOCAL_ONE or LOCAL_QUORUM
• Application driver needs a DC Aware policy
• Light Weight Transactions (LWT) must use LOCAL_SERIAL

Original Datacenter with 5 nodes

Step 1 – Alter System Keyspaces

Before any new nodes were created in the new datacenter, the first step was to alter the system keyspaces to include the new datacenter:

ALTER KEYSPACE system_auth WITH replication = {
  'class': 'NetworkTopologyStrategy', 'RS_UK': 3, 'GL_EU': 3
};

This should be performed for all the system keyspaces:

• system_auth
• system_schema
• dse_leases
• system_distributed
• dse_perf
• system_traces
• dse_security

Ignore the ones starting dse_ if this is an Apache Cassandra cluster rather than a DataStax version.

Step 2 – Create Nodes in New Datacenter

When preparing for the creation of nodes in the new DC, the following configuration needs to be taken into account:

1. The cluster_name in cassandra.yaml must be the same as the cluster_name in the old DC
2. The seeds should point to the seeds in the old DC
3. Assuming the old nodes were using GossipingPropertyFileSnitch, continue using it and set the dc in cassandra-rackdc.properties to the new DC name

Create the new nodes one at a time, waiting for each one to complete joining the cluster before moving to the next. Each node should join quickly because at this stage only the system keyspaces are being streamed to the new DC.

New GL_EU nodes created

At this point the system keyspaces are replicated to the new DC, but the user keyspaces are not. It is important that the application keeps connecting to the old DC.

System Keyspaces replicated

Step 3 – Alter Replication for User Keyspaces

Now it is time to alter the replication of the user keyspaces, to allow replication of all data to the new DC. Run this statement for each keyspace:

ALTER KEYSPACE user_keyspace1 WITH replication = {
  'class': 'NetworkTopologyStrategy', 'RS_UK': 3, 'GL_EU': 3
};

ALTER KEYSPACE user_keyspace2 WITH replication = {
  'class': 'NetworkTopologyStrategy', 'RS_UK': 3, 'GL_EU': 3
};

Once this is completed, all new inserted data is replicated to GL_EU. However the old data has not yet been streamed across, so it is still not safe to connect to GL_EU.

Step 4 – Rebuild Nodes

On each node in turn, run the following nodetool command:

nodetool rebuild RS_UK

This will take some time for each node to complete, as all the data required for each node is now streamed from the RS_UK datacenter. It is often a good idea to script this part to run on each node in turn.

It is possible that a nodetool rebuild will fail — I will go over some of the reasons we had failures in Part 3. If a node fails you can just re-run the command.

Once all the new nodes have been rebuilt the cluster is now working as a multi-DC cluster, and the applications can connect to either DC, with data flowing automatically between the DCs.

Data flows automatically between the Datacenters

In Part 2 we will look at how to safely decommission the RS_UK datacenter.