As you store data into the Couchbase Server cluster, you may need to alter the number of nodes in the cluster to cope with changes in application load, RAM, disk I/O and networking performance requirements.

Couchbase Server is designed to actively change the number of nodes configured within the cluster to cope with these requirements, all while the cluster is up and running and servicing application requests. The overall process is consists of two stages; the addition or removal of nodes, and rebalancing of the data across the nodes.

The addition and removal processes merely configure a new node or mark a node for removal from the cluster. No actual changes are made to the cluster or data when configuring new nodes or removing existing ones.

During the rebalance operation:

• Using the new Couchbase Server cluster structure, data is moved between the vBuckets on each node from the old structure. This process works by exchanging the data held in vBuckets on each node across the cluster. This has two effects:

  • Removes the data from the machines removed from the cluster. By totally removing the storage of data on these machines, it permits for each removed node to be taken out of the cluster without affecting the cluster operation.

  • Adds data and enables new nodes so that they can serve information to clients. By moving active data to the new nodes, these become responsible for the moved vBuckets and for servicing client requests.

• Rebalancing moves both the data stored in RAM, and the data stored on disk. This applies to each bucket and to each node within the cluster. The time taken for the move depends on the level of cluster activity and the amount of stored information.

• During rebalancing the cluster remains up, and continues to service and handle client requests. Updates and changes to the stored data during the migration process are tracked and will be updated and migrated with the data that existed when the rebalance was requested.

• The current vBucket map, used to identify which nodes in the cluster are responsible for handling client requests, is updated incrementally as each vBucket is moved. The updated vBucket map is communicated to Couchbase client libraries and enabled smart clients (such as Moxi), and permits clients to use the updated structure as the rebalance completes. This ensures that the new structure is used as soon as possible to help spread and even out the load during the rebalance operation.

Because the cluster stays up and active throughout the entire rebalancing process, clients can continue to store and retrieve information and do not need to be aware that a rebalance operation is taking place.

There are four primary reasons to perform a rebalance operation:

• Adding nodes to expand the size of the cluster.

• Removing nodes to reduce the size of the cluster.

• Reacting to a failover situation, where you need to bring the cluster back to a healthy state.

• Temporary removal of one or more nodes to perform a software, operating system, or hardware upgrade.

Regardless of the immediate reason, the purpose of any rebalance is to bring up a cluster to a healthy state where the configured nodes, buckets, and replicas match the current state of the cluster.