Rollbacks During Replica Set Failover¶

On this page

Collect Rollback Data
Avoid Replica Set Rollbacks
Rollback Limitations

A rollback reverts write operations on a former primary when the member rejoins its replica set after a failover. A rollback is necessary only if the primary had accepted write operations that the secondaries had not successfully replicated before the primary stepped down. When the primary rejoins the set as a secondary, it reverts, or “rolls back,” its write operations to maintain database consistency with the other members.

MongoDB attempts to avoid rollbacks, which should be rare. When a rollback does occur, it is often the result of a network partition. Secondaries that can not keep up with the throughput of operations on the former primary, increase the size and impact of the rollback.

A rollback does not occur if the write operations replicate to another member of the replica set before the primary steps down and if that member remains available and accessible to a majority of the replica set.

Collect Rollback Data¶

When a rollback does occur, administrators must decide whether to apply or ignore the rollback data. MongoDB writes the rollback data to BSON files in the rollback/ folder under the database’s dbPath directory. The names of rollback files have the following form:

<database>.<collection>.<timestamp>.bson

For example:

records.accounts.2011-05-09T18-10-04.0.bson

Administrators must apply rollback data manually after the member completes the rollback and returns to secondary status. Use bsondump to read the contents of the rollback files. Then use mongorestore to apply the changes to the new primary.

Avoid Replica Set Rollbacks¶

For replica sets, the default write concern {w: 1} only provides acknowledgement of write operations on the primary. With the default write concern, data may be rolled back if the primary steps down before the write operations have replicated to any of the secondaries.

To prevent rollbacks of data that have been acknowledged to the client, use w: majority write concern to guarantee that the write operations propagate to a majority of the replica set nodes before returning with acknowledgement to the issuing client.

Note

Regardless of write concern, other clients using "local" (i.e. the default) readConcern can see the result of a write operation before the write operation is acknowledged to the issuing client.
Clients using "local" (i.e. the default) readConcern can read data which may be subsequently rolled back.

Rollback Limitations¶

A mongod instance will not rollback more than 300 megabytes of data. If your system must rollback more than 300 megabytes, you must manually intervene to recover the data. If this is the case, the following line will appear in your mongod log:

[replica set sync] replSet syncThread: 13410 replSet too much data to roll back

In this situation, save the data directly or force the member to perform an initial sync. To force initial sync, sync from a “current” member of the set by deleting the content of the dbPath directory for the member that requires a larger rollback.