Getting TAP information

Couchbase Server uses an internal protocol known as TAP to stream information about data changes between cluster nodes. Couchbase Server uses the TAP protocol during 1) rebalance, 2) replication at other cluster nodes, and 3) persistence of items to disk.

Be aware that this tool is a per-node, per-bucket operation. That means that if you want to perform this operation, you must specify the IP address of a node in the cluster and a named bucket. If you do not provided a named bucket, the server will apply the setting to any default bucket that exists at the specified node. If you want to perform this operation for an entire cluster, you will need to perform the command for every node/bucket combination that exists for that cluster.

The following statistics will be output in response to a cbstats tap request:

Name Description
ep_tap_total_queue Sum of tap queue sizes on the current tap queues
ep_tap_total_fetched Sum of all tap messages sent
ep_tap_bg_max_pending The maximum number of background jobs a tap connection may have
ep_tap_bg_fetched Number of tap disk fetches
ep_tap_bg_fetch_requeued Number of times a tap background fetch task is requeued.
ep_tap_fg_fetched Number of tap memory fetches
ep_tap_deletes Number of tap deletion messages sent
ep_tap_throttled Number of tap messages refused due to throttling.
ep_tap_keepalive How long to keep tap connection state after client disconnect.
ep_tap_count Number of tap connections
ep_tap_bg_num_samples The number of tap background fetch samples included in the average
ep_tap_bg_min_wait The shortest time (µs) for a tap item before it is serviced by the dispatcher
ep_tap_bg_max_wait The longest time (µs) for a tap item before it is serviced by the dispatcher
ep_tap_bg_wait_avg The average wait time (µs) for a tap item before it is serviced by the dispatcher
ep_tap_bg_min_load The shortest time (µs) for a tap item to be loaded from the persistence layer
ep_tap_bg_max_load The longest time (µs) for a tap item to be loaded from the persistence layer
ep_tap_bg_load_avg The average time (µs) for a tap item to be loaded from the persistence layer
ep_tap_noop_interval The number of secs between a no-op is added to an idle connection
ep_tap_backoff_period The number of seconds the tap connection should back off after receiving ETMPFAIL
ep_tap_queue_fill Total enqueued items
ep_tap_queue_drain Total drained items
ep_tap_queue_backoff Total back-off items
ep_tap_queue_backfill Number of backfill remaining
ep_tap_queue_itemondisk Number of items remaining on disk
ep_tap_throttle_threshold Percentage of memory in use before we throttle tap streams
ep_tap_throttle_queue_cap Disk write queue cap to throttle tap streams

You use the cbstats tapagg to get statistics from named tap connections which are logically grouped and aggregated together by prefixes.

For example, if all of your tap connections started with rebalance_ or replication_, you could call cbstats tapagg _ to request stats grouped by the prefix starting with _. This would return a set of statistics for rebalance and a set for replication. The following are possible values returned by cbstats tapagg :

Name Description
[prefix]:count Number of connections matching this prefix
[prefix]:qlen Total length of queues with this prefix
[prefix]:backfill_remaining Number of items needing to be backfilled
[prefix]:backoff Total number of backoff events
[prefix]:drain Total number of items drained
[prefix]:fill Total number of items filled
[prefix]:itemondisk Number of items remaining on disk
[prefix]:total_backlog_size Number of remaining items for replication