Indexer shows no progress for long periods of time.

Each design document maps to one indexer, so when the indexer runs it updates all views defined in the corresponding design document. Indexing takes resources (CPU, disk IO, memory), therefore Couchbase Server limits the maximum number of indexers that can run in parallel. There are 2 configuration parameters to specify the limit, one for regular (main/active) indexers and other for replica indexers (more on this in a later section). The default for the former is 4 and for the later is 2. They can be queried like this:

> curl -s 'http://Administrator:asdasd@localhost:8091/settings/maxParallelIndexers'
{"globalValue":4,"nodes":{"[email protected]":4}}

maxParallelIndexers is for main indexes and maxParallelReplicaIndexers is for replica indexes. When there are more design documents (indexers) than maxParallelIndexers, some indexers are blocked until there’s a free slot, and the rule is simple as first-come-first-served. These slots are controlled by 2 barriers processes, one for main indexes, and the other for replica indexes. Their current state can be seen from _active_tasks (per node), for example when there’s no indexing happening:

> curl -s 'http://localhost:9500/_active_tasks' | json_xs
[
 {
     "waiting" : 0,
         "started_on" : 1345642656,
         "pid" : "<0.234.0>",
         "type" : "couch_main_index_barrier",
         "running" : 0,
         "limit" : 4,
         "updated_on" : 1345642656
         },
 {
     "waiting" : 0,
         "started_on" : 1345642656,
         "pid" : "<0.235.0>",
         "type" : "couch_replica_index_barrier",
         "running" : 0,
         "limit" : 2,
         "updated_on" : 1345642656
         }
 ]

The waiting fields tells us how many indexers are blocked, waiting for their turn to run. Queries with stale=false have to wait for the indexer to be started (if not already), unblocked and to finish, which can lead to a long time when there are many design documents in the system. Also take into account that the indexer for a particular design document might be running for one node but it might be blocked in another node - when it’s blocked it’s not necessarily blocked in all nodes of the cluster nor when it’s running is necessarily running in all nodes of the cluster - you verify this by querying _active_tasks for each node (this API is not meant for direct user consumption, just for developers and debugging/troubleshooting).

Through _active_tasks (remember, it’s per node, so check it for every node in the cluster), you can see which indexers are running and which are blocked. Here follows an example where we have 5 design documents (indexers) and >maxParallelIndexers is 4:

> curl -s 'http://localhost:9500/_active_tasks' | json_xs
[
   {
      "waiting" : 1,
      "started_on" : 1345644651,
      "pid" :  "<0.234.0>",
      "type" :  "couch_main_index_barrier",
      "running" : 4,
      "limit" : 4,
      "updated_on" : 1345644923
   },
   {
      "waiting" : 0,
      "started_on" : 1345644651,
      "pid" :  "<0.235.0>",
      "type" :  "couch_replica_index_barrier",
      "running" : 0,
      "limit" : 2,
      "updated_on" : 1345644651
   },
   {
      "indexer_type" : "main",
      "started_on" : 1345644923,
      "updated_on" : 1345644923,
      "design_documents" : [
         "_design/test"
      ],
      "pid" :  "<0.4706.0>",
      "signature" : "4995c136d926bdaf94fbe183dbf5d5aa",
      "type" :  "blocked_indexer",
      "set" :  "default"
   },
   {
      "indexer_type" : "main",
      "started_on" : 1345644923,
      "progress" : 0,
      "initial_build" : true,
      "updated_on" : 1345644923,
      "total_changes" : 250000,
      "design_documents" : [
         "_design/test4"
      ],
      "pid" :  "<0.4715.0>",
      "changes_done" : 0,
      "signature" : "15e1f576bc85e3e321e28dc883c90077",
      "type" :  "indexer",
      "set" :  "default"
   },
   {
      "indexer_type" : "main",
      "started_on" : 1345644923,
      "progress" : 0,
      "initial_build" : true,
      "updated_on" : 1345644923,
      "total_changes" : 250000,
      "design_documents" : [
         "_design/test3"
      ],
      "pid" :  "<0.4719.0>",
      "changes_done" : 0,
      "signature" : "018b83ca22e53e14d723ea858ba97168",
      "type" :  "indexer",
      "set" :  "default"
   },
   {
      "indexer_type" : "main",
      "started_on" : 1345644923,
      "progress" : 0,
      "initial_build" : true,
      "updated_on" : 1345644923,
      "total_changes" : 250000,
      "design_documents" : [
         "_design/test2"
      ],
      "pid" :  "<0.4722.0>",
      "changes_done" : 0,
      "signature" : "440b0b3ded9d68abb559d58b9fda3e0a",
      "type" :  "indexer",
      "set" : "default"
   },
   {
      "indexer_type" : "main",
      "started_on" : 1345644923,
      "progress" : 0,
      "initial_build" : true,
      "updated_on" : 1345644923,
      "total_changes" : 250000,
      "design_documents" : [
         "_design/test7"
      ],
      "pid" :  "<0.4725.0>",
      "changes_done" : 0,
      "signature" : "fd2bdf6191e61af6e801e3137e2f1102",
      "type" :  "indexer",
      "set" :  "default"
   }
]

The indexer for design document _design/test is represented by a task with a type field of blocked_indexer, while other indexers have a task with type indexer, meaning they’re running. The task with type couch_main_index_barrier confirms this by telling us there are currently 4 indexers running and 1 waiting for its turn. When an indexer is allowed to execute, its active task with type blocked_indexer is replaced by a new one with type indexer.