Rebalancing nodes
Description
To start a rebalance process, provide the ejectedNodes and knownNodes parameters. These parameters contain the list of nodes that have been marked to be ejected and the list of nodes that are known within the cluster.
Retrieve information about ejected and known nodes by getting the current node configuration. This ensures that the client making the REST API request is aware of the current cluster configuration. Nodes should have been previously added or marked for removal as appropriate.
HTTP method and URI
The information must be supplied via the ejectedNodes and knownNodes parameters as a POST operation to the /controller/rebalance endpoint.
POST /controller/rebalance
Syntax
curl -v -X -u [admin]:[password] POST
http://[localhost]:8091/controller/rebalance
-d ejectedNodes
-d knownNodes
Example
Curl request example:
curl -v -X -u admin:password POST 'http://192.168.0.77:8091/controller/rebalance' \
-d 'ejectedNodes=&knownNodes=ns_1%40192.168.0.77%2Cns_1%40192.168.0.56'
Raw HTTP request example:
POST /controller/rebalance HTTP/1.1
Authorization: Basic QWRtaW5pc3RyYXRvcjpUYW1zaW4=
User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8r zlib/1.2.5
Host: 192.168.0.77:8091
Accept: */*
Content-Length: 63
Content-Type: application/x-www-form-urlencoded
Response codes
The response is 200 (OK) if the operation was successfully submitted.
If the wrong node information has been submitted, JSON with the mismatch error is returned:
{"mismatch":1}
Getting rebalance progress
HTTP method and URI
There are two endpoints for rebalance progress. One is a general request which outputs high-level percentage completion at /pools/default/rebalanceProgress. The second possible endpoint is one corresponds to the detailed rebalance report available in the web console.
GET /pools/default/rebalanceProgress
Syntax
curl -u [admin]:[password]
'[localhost]:8091/pools/default/rebalanceProgress'
Example
Curl request example that returns a JSON structure containing the current progress information:
curl -u admin:password
'192.168.0.77:8091/pools/default/rebalanceProgress'
Raw HTTP request example:
GET /pools/default/rebalanceProgress HTTP/1.1
Authorization: Basic QWRtaW5pc3RyYXRvcjpUYW1zaW4=
User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8r zlib/1.2.5
Host: 192.168.0.77:8091
Accept: */*
Response
The response data packet contains a JSON structure showing the rebalance progress for each node. The progress figure is provided as a percentage (shown as a floating point value between 0 and 1).
{
"status":"running",
"[email protected]":{"progress":0.2734375},
"[email protected]":{"progress":0.09114583333333337}
}
Example: detailed
Curl request example with more details about the rebalance:
curl -u admin:password
'http://192.168.0.77:8091/pools/default/tasks'
Raw HTTP request example:
GET /pools/default/rebalanceProgress HTTP/1.1
Authorization: Basic QWRtaW5pc3RyYXRvcjpUYW1zaW4=
User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8r zlib/1.2.5
Host: 192.168.0.77:8091
Accept: */*
Response: detailed
The response data packet contains a JSON structure showing detailed progress:
{
type: "rebalance",
recommendedRefreshPeriod: 0.25,
status: "running",
progress: 9.049479166666668,
perNode: {
[email protected]: {
progress: 13.4765625
},
[email protected]: {
progress: 4.6223958333333375
}
},
detailedProgress: {
bucket: "default",
bucketNumber: 1,
bucketsCount: 1,
perNode: {
[email protected]: {
ingoing: {
docsTotal: 0,
docsTransferred: 0,
activeVBucketsLeft: 0,
replicaVBucketsLeft: 0
},
outgoing: {
docsTotal: 512,
docsTransferred: 69,
activeVBucketsLeft: 443,
replicaVBucketsLeft: 511
}
},
[email protected]: {
ingoing: {
docsTotal: 512,
docsTransferred: 69,
activeVBucketsLeft: 443,
replicaVBucketsLeft: 0
},
outgoing: {
docsTotal: 0,
docsTransferred: 0,
activeVBucketsLeft: 0,
replicaVBucketsLeft: 443
}
}
}
}
}
This reponse shows percentage complete for each individual node undergoing rebalance. For each specific node, it provides the current number of docs transferred and other items. For details and definitions of these items.
If rebalance fails, the following response error occurs:
[
{
"type": "rebalance",
"status": "notRunning",
"errorMessage": "Rebalance failed. See logs for detailed reason. You can try rebalance again."
}
]
Adjusting rebalance during compaction
Description
If a rebalance is performed while a node is undergoing index compaction, rebalance delays may be experienced. The parameter, rebalanceMovesBeforeCompaction, is used to improve rebalance performance. If this selection is made, index compaction performance is reduced which can result in larger index file size.
This needs to be made as POST request to the /internalSettings endpoint. By default, this setting is 64 which specifies the number of vBuckets which are moved per node until all vBucket movements pauses. After this pause, the system triggers index compaction. Index compaction is not performed while vBuckets are being moved, so if a larger value is specified, it means that the server spends less time compacting the index, which results in larger index files that take up more disk space.
HTTP method and URI
POST /internalSettings rebalanceMovesBeforeCompaction
Syntax
Curl request syntax:
curl -X POST -u admin:password 'http://[localhost]:8091/internalSettings'
-d 'rebalanceMovesBeforeCompaction=[value]'
Example
Curl request example:
curl -X POST -u admin:password 'http://10.5.2.54:8091/internalSettings'
-d 'rebalanceMovesBeforeCompaction=256'