Loggregator Guide for Cloud Foundry Operators

Page last updated: December 10, 2015

This topic contains information about the Loggregator System that may be of use to operators of Cloud Foundry deployments.

Scaling Loggregator

Dopplers are components that take in log and metric data from Cloud Foundry system components and store this data in a buffer before periodically forwarding it to the Traffic Controller, which serves up the aggregated data stream through the Firehose WebSocket endpoint. When input from Metron agents exceeds a doppler’s buffer size for a given interval, log and metric data can be lost. There are several ways to minimize this loss.

Increase buffer size

The doppler internal buffer size can be increased by changing the doppler.message\_drain\_buffer\_size property from its default value of 100 in your CloudFoundry BOSH deployment manifest.

Add additional Doppler instances

The number of doppler servers can be increased by modifying the instances property for the doppler\_z1 and doppler\_z2 jobs in your CloudFoundry BOSH deployment manifest.

Scaling Nozzles

Scale nozzles by using the subscription ID, which is specified when the nozzle connects to the Firehose. If you use the same subscription ID on each nozzle instance, the Firehose will evenly distribute events across all instances of the nozzle. For example, if you have two nozzles with the same subscription ID, then half the events will go to one nozzle and half to the other. Similarly, if there were three instances of the nozzle, then each instance would get one-third the traffic.

A stateless nozzle should handle scaling gracefully. If the nozzle buffers or caches the data, the nozzle author must test what happens when the nozzle is scaled up or scaled down.

Slow Nozzle Alerts

The Traffic Controller alerts nozzles if they are consuming events too slowly. If the nozzle falls behind, Loggregator will alert the nozzle in two ways:

• TruncatingBuffer alerts: If the nozzle is consuming messages more slowly than they are being produced, the loggregator system may drop messages. In this case, the loggregator system sends the log message TB: Output channel too full. Dropped (n) messages, where “n” is the number of dropped messages. It also emits a CounterEvent with the name TruncatingBuffer.DroppedMessages. The nozzle receives both messages from the Firehose, alerting the operator to the performance issue.

• PolicyViolation error: The Traffic Controller periodically sends ping control messages over the Firehose WebSocket connection. If a client does not respond to a ping message with a pong message within 30 seconds, the Traffic Controller closes the WebSocket connection with the WebSocket error code “ClosePolicyViolation” (1008). The nozzle should intercept this WebSocket close error, alerting the operator to the performance issue.

An operator can choose to scale her nozzles in response to these alerts, in order to minimize the loss of data.

Managing Syslog Forwarding from Cloud Foundry Components

Syslog data can be forwarded directly from Cloud Foundry components to an external aggregator.

To do this, add the following properties to the properties hash in your CF deployment manifest.

properties:
  syslog_daemon_config:
        address: YOUR-SYSLOG-AGGREGATOR-IP
        port: YOUR-SYSLOG-AGGREGATOR-TCP-PORT
        transport: YOUR-TRANSPORT-PROTOCOL

Replace YOUR-TRANSPORT-PROTOCOL with one of the following transport protocols:

• tcp
• udp
• relp

Customizing Loggregator Components

Each Loggregator component can be customized in many ways by changing its properties in the CF deployment manifest. Some of the most useful are detailed below.

DEA Logging Agent

Doppler

Traffic Controller

Metron Agent

Syslog Drain Binder

See Using Log Management Services.