Application Logging in Cloud Foundry

Page last updated: May 4, 2015

This page assumes you are using cf CLI v6.

Loggregator, the Cloud Foundry component responsible for logging, provides a stream of log output from your application and from Cloud Foundry system components that interact with your app during updates and execution.

By default, Loggregator streams logs to your terminal. If you want to persist more than the limited amount of logging information that Loggregator can buffer, you can drain logs to a third-party log management service. See Third-Party Log Management Services.

Cloud Foundry gathers and stores logs in a best-effort manner. If a client is unable to consume log lines quickly enough, the Loggregator buffer may need to overwrite some lines before the client has consumed them. A syslog drain or a CLI tail can usually keep up with the flow of application logs.

Contents of a Log Line

Every log line contains four fields:

  1. Timestamp
  2. Log type (origin code)
  3. Channel: either STDOUT or STDERR
  4. Message

Loggregator assigns the timestamp when it receives log data. The log data is opaque to Loggregator, which simply puts it in the message field of the log line. Applications or system components sending log data to Loggregator may include their own timestamps, which then appear in the message field.

Origin codes distinguish the different log types. Origin codes from system components have three letters. The application origin code is APP followed by slash and a digit that indicates the application instance.

Many frameworks write to an application log that is separate from STDOUT and STDERR. This is not supported by Loggregator. Your application must write to STDOUT or STDERR for its logs to be included in the Loggregator stream. Check the buildpack your application uses to determine whether it automatically insures that your application correctly writes logs to STDOUT and STDERR only. Some buildpacks do this, and some do not.

Log Types and Their Messages

Different types of logs have different message formats, as shown in the examples below.

API

Users make API calls to request changes in application state. Cloud Controller, the Cloud Foundry component responsible for the API, logs the actions that Cloud Controller takes in response.

For example:

2014-02-13T11:44:52.11-0800 [API]     OUT Updated app with guid e1ca6390-cf78-4fc7-9d86-5b7ed01e9c28 ({"instances"=>2})

STG

The Droplet Execution Agent emits STG logs when staging or restaging an app. These actions implement the desired state requested by the user. Once the droplet has been uploaded, STG messages end and DEA messages begin.

For example:

2014-02-07T10:54:36.80-0800 [STG]     OUT -----> Downloading and installing node

DEA

The Droplet Execution Agent emits DEA logs beginning when it starts or stops the app. These actions implement the desired state requested by the user. The DEA also emits messages when an app crashes.

For example:

2014-02-13T11:44:52.07-0800 [DEA]     OUT Starting app instance (index 1) with guid e1ca6390-cf78-4fc7-9d86-5b7ed01e9c28

RTR

The Router emits RTR logs when it routes HTTP requests to the application. Router messages include the application name followed by a Router timestamp and then selections from the HTTP request.

For example:

2014-02-13T11:42:31.96-0800 [RTR]     OUT nifty-gui.shared-domain.com - [13/02/2014:19:42:31 +0000]
"GET /favicon.ico HTTP/1.1" 404 23 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36" 10.10.2.142:6609 response_time:0.004092262
app_id:e1ca6390-cf78-4fc7-9d86-5b7ed01e9c28

LGR

Loggregator emits LGR to indicate problems with the logging process. Examples include “can’t reach syslog drain url” and “dropped log messages due to high rate.”

APP

Every application emits logs according to choices by the developer. The digit appended to the code indicates app instance: 0 is the first instance, 1 is the second, and so on.

For example:

2014-02-13T11:44:27.71-0800 [App/0]   OUT Express server started

Writing to the Log from your Application

Your application must write logs to STDERR or STDOUT. Both are typically buffered, and you should flush the buffer before delivering the message to Loggregator.

Alternatively, you can write log messages to STDERR or STDOUT synchronously. This approach is mainly used for debugging because it may affect application performance.

Viewing Logs in the Command Line Interface

You view logs in the CLI using the cf logs command. You can tail, dump, or filter log output.

Tailing Logs

cf logs APP_NAME streams Loggregator output to the terminal.

For example:

$ cf logs nifty-gui
    Connected, tailing logs for app nifty-gui in org janclouduser / space jancloudspace as admin...
    2014-02-06T12:00:19.44-0800 [API]
        OUT Updated app with guid c8612fc2-85b1-464c-92f5-d4a1156eacbf
        ({"route"=>"2ef5796b-475a-4615-9c71-75bbe277022e"})
    2014-02-06T12:00:27.54-0800 [DEA]
        OUT Got staging request for app with id
        c8612fc2-85b1-464c-92f5-d4a1156eacbf
    2014-02-06T12:00:28.51-0800 [API]
        OUT Updated app with guid c8612fc2-85b1-464c-92f5-d4a1156eacbf
        ({"state"=>"STARTED"})
    ...

Use Ctrl-C (^C) to exit the real-time stream.

Dumping Logs

cf logs APP_NAME --recent displays all the lines in the Loggregator buffer.

Filtering Logs

To view some subset of log output, use cf logs in conjunction with filtering commands of your choice. In the example below, grep -v excludes all Router logs:

$ cf logs nifty-gui --recent | grep -v RTR
    Connected, dumping recent logs for app nifty-gui in org jancloudspace-org / space development
    as [email protected]...
    2014-02-07T10:54:40.41-0800 [STG]
        OUT -----> Uploading droplet (5.6M)
    2014-02-07T10:54:44.44-0800 [DEA]
        OUT Starting app instance (index 0) with guid
        4d397313-20e0-478a-9a74-307446eb7640
    2014-02-07T10:54:46.31-0800 [App/0]
        OUT Express server started
    2014-02-07T10:57:53.60-0800 [API]
        OUT Updated app with guid 4d397313-20e0-478a-9a74-307446eb7640
        ({"instances"=>2})
    2014-02-07T10:57:53.64-0800 [DEA]
        OUT Starting app instance (index 1) with guid
        4d397313-20e0-478a-9a74-307446eb7640
    2014-02-07T10:57:55.88-0800 [App/1]
        OUT Express server started
    ...