FAQ: MongoDB Diagnostics¶

On this page

Where can I find information about a mongod process that stopped running unexpectedly?
Does TCP keepalive time affect MongoDB Deployments?
What tools are available for monitoring MongoDB?
Memory Diagnostics for the MMAPv1 Storage Engine
Memory Diagnostics for the WiredTiger Storage Engine
Sharded Cluster Diagnostics

This document provides answers to common diagnostic questions and issues.

If you don’t find the answer you’re looking for, check the complete list of FAQs or post your question to the MongoDB User Mailing List.

Where can I find information about a `mongod` process that stopped running unexpectedly?¶

If mongod shuts down unexpectedly on a UNIX or UNIX-based platform, and if mongod fails to log a shutdown or error message, then check your system logs for messages pertaining to MongoDB. For example, for logs located in /var/log/messages, use the following commands:

sudo grep mongod /var/log/messages
sudo grep score /var/log/messages

Does TCP `keepalive` time affect MongoDB Deployments?¶

If you experience socket errors between clients and servers or between members of a sharded cluster or replica set that do not have other reasonable causes, check the TCP keepalive value (e.g. on Linux systems store, the tcp_keepalive_time value). A common keepalive period is 7200 seconds (2 hours); however, different distributions and OS X may have different settings.

For MongoDB, you will have better results with shorter keepalive periods, on the order of 120 seconds (two minutes).

If your MongoDB deployment experiences keepalive-related issues, you must alter the keep alive value on all machines hosting MongoDB processes. This includes all machines hosting mongos or mongod servers and all machines hosting client processes that connect to MongoDB.

Note

For non-Linux systems, values greater than or equal to 600 seconds (10 minutes) will be ignored by mongod and mongos. For Linux, values greater than 300 seconds (5 minutes) will be overridden on the mongod and mongos sockets with a maximum of 300 seconds.

On Linux systems:

To view the keep alive setting, you can use one of the following commands:
```
sysctl net.ipv4.tcp_keepalive_time
```
Or:
```
cat /proc/sys/net/ipv4/tcp_keepalive_time
```
The value is measured in seconds.
To change the tcp_keepalive_time value, you can use one of the following command:
```
sudo sysctl -w net.ipv4.tcp_keepalive_time=<value>
```
Or:
```
echo <value> | sudo tee /proc/sys/net/ipv4/tcp_keepalive_time
```
These operations do not persist across system reboots. To persist the setting, add the following line to /etc/sysctl.conf:
```
net.ipv4.tcp_keepalive_time = <value>
```
On Linux, mongod and mongos processes limit the keepalive to a maximum of 300 seconds (5 minutes) on their own sockets by overriding keepalive values greater than 5 minutes.

For OS X systems:

To view the keep alive setting, issue the following command:
```
sysctl net.inet.tcp.keepinit
```
To change the net.inet.tcp.keepinit value, you can use the following command:
```
sysctl -w net.inet.tcp.keepinit=<value>
```
The above method for setting the TCP keepalive is not persistent; you will need to reset the value each time you reboot or restart a system. See your operating system’s documentation for instructions on setting the TCP keepalive value persistently.

For Windows systems:

To view the keep alive setting, issue the following command:
```
reg query HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters /v KeepAliveTime
```
The registry value is not present by default. The system default, used if the value is absent, is 7200000 milliseconds or 0x6ddd00 in hexadecimal.
To change the KeepAliveTime value, use the following command in an Administrator Command Prompt, where <value> is expressed in hexadecimal (e.g. 0x0124c0 is 120000):
```
reg add HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\ /v KeepAliveTime /d <value>
```
Windows users should consider the Windows Server Technet Article on KeepAliveTime for more information on setting keep alive for MongoDB deployments on Windows systems.

You will need to restart mongod and mongos servers for new system-wide keepalive settings to take effect.

What tools are available for monitoring MongoDB?¶

The MongoDB Cloud Manager and Ops Manager, an on-premise solution available in MongoDB Enterprise Advanced include monitoring functionality, which collects data from running MongoDB deployments and provides visualization and alerts based on that data.

For more information, see also the MongoDB Cloud Manager documentation and Ops Manager documentation.

A full list of third-party tools is available as part of the Monitoring for MongoDB documentation.

Memory Diagnostics for the MMAPv1 Storage Engine¶

Do I need to configure swap space?¶

Always configure systems to have swap space. Without swap, your system may not be reliant in some situations with extreme memory constraints, memory leaks, or multiple programs using the same memory. Think of the swap space as something like a steam release valve that allows the system to release extra pressure without affecting the overall functioning of the system.

Nevertheless, systems running MongoDB do not need swap for routine operation. Database files are memory-mapped and should constitute most of your MongoDB memory use. Therefore, it is unlikely that mongod will ever use any swap space in normal operation. The operating system will release memory from the memory mapped files without needing swap and MongoDB can write data to the data files without needing the swap system.

What is a “working set”?¶

The working set is the portion of your data that clients access most often.

Must my working set size fit RAM?¶

Your working set should stay in memory to achieve good performance. Otherwise many random disk IO’s will occur, and unless you are using SSD, this can be quite slow.

One area to watch specifically in managing the size of your working set is index access patterns. If you are inserting into indexes at random locations (as would happen with id’s that are randomly generated by hashes), you will continually be updating the whole index. If instead you are able to create your id’s in approximately ascending order (for example, day concatenated with a random id), all the updates will occur at the right side of the b-tree and the working set size for index pages will be much smaller.

It is fine if databases and thus virtual size are much larger than RAM.

How do I calculate how much RAM I need for my application?¶

The amount of RAM you need depends on several factors, including but not limited to:

The relationship between database storage and working set.
The operating system’s cache strategy for LRU (Least Recently Used)
The impact of journaling
The number or rate of page faults and other MongoDB Cloud Manager gauges to detect when you need more RAM
Each database connection thread will need up to 1 MB of RAM.

MongoDB defers to the operating system when loading data into memory from disk. It simply memory maps all its data files and relies on the operating system to cache data. The OS typically evicts the least-recently-used data from RAM when it runs low on memory. For example if clients access indexes more frequently than documents, then indexes will more likely stay in RAM, but it depends on your particular usage.

To calculate how much RAM you need, you must calculate your working set size, or the portion of your data that clients use most often. This depends on your access patterns, what indexes you have, and the size of your documents. Because MongoDB uses a thread per connection model, each database connection also will need up to 1 MB of RAM, whether active or idle.

If page faults are infrequent, your working set fits in RAM. If fault rates rise higher than that, you risk performance degradation. This is less critical with SSD drives than with spinning disks.

How do I read memory statistics in the UNIX `top` command¶

Because mongod uses memory-mapped files, the memory statistics in top require interpretation in a special way. On a large database, VSIZE (virtual bytes) tends to be the size of the entire database. If the mongod doesn’t have other processes running, RSIZE (resident bytes) is the total memory of the machine, as this counts file system cache contents.

For Linux systems, use the vmstat command to help determine how the system uses memory. On OS X systems use vm_stat.

Memory Diagnostics for the WiredTiger Storage Engine¶

Must my working set size fit RAM?¶

No.

If the cache does not have enough space to load additional data, WiredTiger evicts pages from the cache to free up space.

Note

The storage.wiredTiger.engineConfig.cacheSizeGB only limits the size of the WiredTiger cache, not the total amount of memory used by mongod. The WiredTiger cache is only one component of the RAM used by MongoDB. MongoDB also automatically uses all free memory on the machine via the filesystem cache (data in the filesystem cache is compressed).

In addition, the operating system will use any free RAM to buffer filesystem blocks.

To accommodate the additional consumers of RAM, you may have to decrease WiredTiger cache size.

The default WiredTiger cache size value assumes that there is a single mongod instance per node. If a single node contains multiple instances, then you should decrease the setting to accommodate the other mongod instances.

If you run mongod in a container (e.g. lxc, cgroups, Docker, etc.) that does not have access to all of the RAM available in a system, you must set storage.wiredTiger.engineConfig.cacheSizeGB to a value less than the amount of RAM available in the container. The exact amount depends on the other processes running in the container.

To see statistics on the cache and eviction, use the serverStatus command. The wiredTiger.cache field holds the information on the cache and eviction.

...
"wiredTiger" : {
   ...
   "cache" : {
      "tracked dirty bytes in the cache" : <num>,
      "bytes currently in the cache" : <num>,
      "maximum bytes configured" : <num>,
      "bytes read into cache" :<num>,
      "bytes written from cache" : <num>,
      "pages evicted by application threads" : <num>,
      "checkpoint blocked page eviction" : <num>,
      "unmodified pages evicted" : <num>,
      "page split during eviction deepened the tree" : <num>,
      "modified pages evicted" : <num>,
      "pages selected for eviction unable to be evicted" : <num>,
      "pages evicted because they exceeded the in-memory maximum" : <num>,,
      "pages evicted because they had chains of deleted items" : <num>,
      "failed eviction of pages that exceeded the in-memory maximum" : <num>,
      "hazard pointer blocked page eviction" : <num>,
      "internal pages evicted" : <num>,
      "maximum page size at eviction" : <num>,
      "eviction server candidate queue empty when topping up" : <num>,
      "eviction server candidate queue not empty when topping up" : <num>,
      "eviction server evicting pages" : <num>,
      "eviction server populating queue, but not evicting pages" : <num>,
      "eviction server unable to reach eviction goal" : <num>,
      "pages split during eviction" : <num>,
      "pages walked for eviction" : <num>,
      "eviction worker thread evicting pages" : <num>,
      "in-memory page splits" : <num>,
      "percentage overhead" : <num>,
      "tracked dirty pages in the cache" : <num>,
      "pages currently held in the cache" : <num>,
      "pages read into cache" : <num>,
      "pages written from cache" : <num>,
   },
   ...

For an explanation of some key cache and eviction statistics, such as wiredTiger.cache.bytes currently in the cache and wiredTiger.cache.tracked dirty bytes in the cache, see wiredTiger.cache.

To adjust the size of the WiredTiger cache, see storage.wiredTiger.engineConfig.cacheSizeGB and --wiredTigerCacheSizeGB. Avoid increasing the WiredTiger cache size above its default value.

How do I calculate how much RAM I need for my application?¶

With WiredTiger, MongoDB utilizes both the WiredTiger cache and the filesystem cache.

Changed in version 3.2: Starting in MongoDB 3.2, the WiredTiger cache, by default, will use the larger of either:

60% of RAM minus 1 GB, or
1 GB.

For systems with up to 10 GB of RAM, the new default setting is less than or equal to the 3.0 default setting (For MongoDB 3.0, the WiredTiger cache uses either 1 GB or half of the installed physical RAM, whichever is larger).

For systems with more than 10 GB of RAM, the new default setting is greater than the 3.0 setting.

Via the filesystem cache, MongoDB automatically uses all free memory that is not used by the WiredTiger cache or by other processes. Data in the filesystem cache is compressed.

To adjust the size of the WiredTiger cache, see storage.wiredTiger.engineConfig.cacheSizeGB and --wiredTigerCacheSizeGB. Avoid increasing the WiredTiger cache size above its default value.

Note

The storage.wiredTiger.engineConfig.cacheSizeGB only limits the size of the WiredTiger cache, not the total amount of memory used by mongod. The WiredTiger cache is only one component of the RAM used by MongoDB. MongoDB also automatically uses all free memory on the machine via the filesystem cache (data in the filesystem cache is compressed).

In addition, the operating system will use any free RAM to buffer filesystem blocks.

To accommodate the additional consumers of RAM, you may have to decrease WiredTiger cache size.

The default WiredTiger cache size value assumes that there is a single mongod instance per node. If a single node contains multiple instances, then you should decrease the setting to accommodate the other mongod instances.

If you run mongod in a container (e.g. lxc, cgroups, Docker, etc.) that does not have access to all of the RAM available in a system, you must set storage.wiredTiger.engineConfig.cacheSizeGB to a value less than the amount of RAM available in the container. The exact amount depends on the other processes running in the container.

To view statistics on the cache and eviction rate, see the wiredTiger.cache field returned from the serverStatus command.

Sharded Cluster Diagnostics¶

The two most important factors in maintaining a successful sharded cluster are:

You can prevent most issues encountered with sharding by ensuring that you choose the best possible shard key for your deployment and ensure that you are always adding additional capacity to your cluster well before the current resources become saturated. Continue reading for specific issues you may encounter in a production environment.

In a new sharded cluster, why does all data remains on one shard?¶

Your cluster must have sufficient data for sharding to make sense. Sharding works by migrating chunks between the shards until each shard has roughly the same number of chunks.

The default chunk size is 64 megabytes. MongoDB will not begin migrations until the imbalance of chunks in the cluster exceeds the migration threshold. While the default chunk size is configurable with the chunkSize setting, these behaviors help prevent unnecessary chunk migrations, which can degrade the performance of your cluster as a whole.

If you have just deployed a sharded cluster, make sure that you have enough data to make sharding effective. If you do not have sufficient data to create more than eight 64 megabyte chunks, then all data will remain on one shard. Either lower the chunk size setting, or add more data to the cluster.

As a related problem, the system will split chunks only on inserts or updates, which means that if you configure sharding and do not continue to issue insert and update operations, the database will not create any chunks. You can either wait until your application inserts data or split chunks manually.

Finally, if your shard key has a low cardinality, MongoDB may not be able to create sufficient splits among the data.

Why would one shard receive a disproportion amount of traffic in a sharded cluster?¶

In some situations, a single shard or a subset of the cluster will receive a disproportionate portion of the traffic and workload. In almost all cases this is the result of a shard key that does not effectively allow write scaling.

It’s also possible that you have “hot chunks.” In this case, you may be able to solve the problem by splitting and then migrating parts of these chunks.

In the worst case, you may have to consider re-sharding your data and choosing a different shard key to correct this pattern.

What can prevent a sharded cluster from balancing?¶

If you have just deployed your sharded cluster, you may want to consider the troubleshooting suggestions for a new cluster where data remains on a single shard.

If the cluster was initially balanced, but later developed an uneven distribution of data, consider the following possible causes:

You have deleted or removed a significant amount of data from the cluster. If you have added additional data, it may have a different distribution with regards to its shard key.
Your shard key has low cardinality and MongoDB cannot split the chunks any further.
Your data set is growing faster than the balancer can distribute data around the cluster. This is uncommon and typically is the result of:
- a balancing window that is too short, given the rate of data growth.
- an uneven distribution of write operations that requires more data migration. You may have to choose a different shard key to resolve this issue.
- poor network connectivity between shards, which may lead to chunk migrations that take too long to complete. Investigate your network configuration and interconnections between shards.

Why do chunk migrations affect sharded cluster performance?¶

If migrations impact your cluster or application’s performance, consider the following options, depending on the nature of the impact:

If migrations only interrupt your clusters sporadically, you can limit the balancing window to prevent balancing activity during peak hours. Ensure that there is enough time remaining to keep the data from becoming out of balance again.
If the balancer is always migrating chunks to the detriment of overall cluster performance:
- You may want to attempt decreasing the chunk size to limit the size of the migration.
- Your cluster may be over capacity, and you may want to attempt to add one or two shards to the cluster to distribute load.

It’s also possible that your shard key causes your application to direct all writes to a single shard. This kind of activity pattern can require the balancer to migrate most data soon after writing it. Consider redeploying your cluster with a shard key that provides better write scaling.

← FAQ: Indexes Reference →

Was this page helpful?

Yes No

Thank you for your feedback!

We're sorry! You can Report a Problem to help us improve this page.

FAQ: MongoDB Diagnostics¶

Where can I find information about a mongod process that stopped running unexpectedly?¶

Does TCP keepalive time affect MongoDB Deployments?¶

What tools are available for monitoring MongoDB?¶

Memory Diagnostics for the MMAPv1 Storage Engine¶

Do I need to configure swap space?¶

What is a “working set”?¶

Must my working set size fit RAM?¶

How do I calculate how much RAM I need for my application?¶

How do I read memory statistics in the UNIX top command¶

Memory Diagnostics for the WiredTiger Storage Engine¶

Must my working set size fit RAM?¶

How do I calculate how much RAM I need for my application?¶

Sharded Cluster Diagnostics¶

In a new sharded cluster, why does all data remains on one shard?¶

Why would one shard receive a disproportion amount of traffic in a sharded cluster?¶

What can prevent a sharded cluster from balancing?¶

Why do chunk migrations affect sharded cluster performance?¶

Where can I find information about a `mongod` process that stopped running unexpectedly?¶

Does TCP `keepalive` time affect MongoDB Deployments?¶

How do I read memory statistics in the UNIX `top` command¶