Application and system logs in Stackato are aggregated into streams which can be viewed, tailed, filtered, and and sent via drains to other log aggregators for archiving or analysis. There are three general types of streams:
A message is a single log line or event in a stream.
Each message has a key which identifies which stream it belongs to (see Keys below).
Log streams are handled by three processes which run on all Stackato nodes:
apptail is an additional process which runs only on DEA and Stager nodes. It sends user application logs to logyard, injecting relevant application-specific events from the cloud_events stream.
A "drain" is a receiver for a log stream. Logyard has three kinds:
Drains for system log and cloud event streams can be added by admins with the kato log drain command. For example:
$ kato log drain add --prefix systail.kato mydrain udp://logs.papertrailapp.com:12345
This creates a UDP drain that recieves messages from kato.log (on all nodes in the cluster) and forwards them to Papertrail on port 12345.
The --prefix flag takes a key prefix as it's argument.
To delete the drain:
$ kato log drain delete mydrain
The kato history command uses a built-in drain which forwards to a Redis server on the Primary node.
The 'file' drain type will append to a local file. To overwrite the file instead, add the 'overwrite=1' option:
$ kato log drain add debug file:///s/logs/debug-1.log overwrite=1
Log drains can emit entries in a variety of formats:
For example, to add a drain with just the timestamp, application name and message:
$ kato log drain add -p apptail -f '{{.HumanTime}} - {{.AppName}}: {{.Text}}' \
> all-apps file:///s/logs/apptail-short.log
JSON keys are enclosed in double curly braces and prefixed with a period. The spaces, hyphen, and colon here are functioning as delimiters. The resulting entry might look like this:
2013-01-22T16:01:14-08:00 - myenv: Application 'myenv' is now running on DEA 27da51
Different JSON keys are available in different log streams:
apptail.:
event.:
systail.:
You can see a list of the default drain formats using kato config get:
$ kato config get logyard drainformats
apptail: ! '{{.HumanTime}} {{.Source}}.{{.InstanceIndex}}: {{.Text}}'
event: ! '{{.Type}}@{{.NodeID}}: {{.Desc}} -- via {{.Process}}'
systail: ! '{{.Name}}@{{.NodeID}}: {{.Text}}'
These default log formats are used when the corresponding prefix is used and no format options ("-f") are specified. For example kato drain add -p systail.dea ... would format the drain using the 'systail' drain format.
Custom formats for drains can be saved as a named type in the Logyard configuration. To do this, add the formatting string to a new key in logyard/drainformats. For example, to save the log format used in the 'all-apps' drain example above:
$ kato config set logyard drainformats/simplefmt "{{.HumanTime}} - {{.AppName}}: {{.Text}}"
You can use this named format when setting up new drains. For example, a shorter command for creating the 'all-apps' drain would be:
$ kato log drain add -p apptail -f simplefmt all-apps file:///s/logs/apptail-short.log
A custom "systail" log stream might look like this:
$ kato config set logyard drainformats/systail-papertrail '{{.HumanTime}} - {{.Name}}@{{.NodeID}} -- {{.Text}}'
This could be forwarded to a log analysis service:
$ kato log drain add papertrail udp://logs.papertrailapp.com:45678 -f systail-papertrail
You can also change the default apptail, event, and systail drain formats to modify the output of any drains using these prefixes (e.g. stackato drain, Cloud Events in the Management Console, and kato log tail respectively).
You can add custom drains to Logyard to look for certain events or parse certain log messages (e.g. tracking application push requests or user logins). Examples of custom drains and more advanced usage of Logyard can be found in the Logyard Developer Guide
Drains for application log streams can be added by users with the stackato log drain command. See the Application Logs section for an example.
The stackato client automatically uses 'apptail.*appid*' as the message key prefix. Users can add drains only for the applications they own, or have access to via groups.
By default, each application can have up to two drains. This limit is configurable with kato config (default_account_capacity/drains).
If a drain gets disconnected (e.g. if the log aggregation service goes down), Logyard will retry the connection at the following intervals:
This ensures that once connectivity is restored, the drains will re-establish their connections within (at most) 5 minutes.
Application drains will retry for one day. Temporary drains (e.g. kato tail) will retry for 25 minutes. All other drains will retry indefinitely.
These timeouts can be configured. To see a list of the configured timeouts, use kato config get. For example:
$ kato config get logyard retrylimits
appdrain.: 24h
tmp.: 25m
To set a timout (minimum 21m), use kato config set. For example, to set the timeout limit to 10 hours on all drains named with the prefix "papertrail":
$ kato config set logyard retrylimits/papertrail 10h
These limits will take effect on new drains, deleted/re-created drains, or for all matching drains after kato process restart logyard has been run on all nodes.
For performance reasons, it's necessary to limit the total number of concurrent user application drains running on a Stackato system. This is set by default to 200. Once this limit is reached, users will see the following notificition when trying to add a new drain:
$ stackato drain add mydrain udp://loghost.example.com:12346
Error 22002: No more drains can be added; contact your cluster admin.
To change the limit, set the max_user_drains in the cloud_controller configuration using kato config. For example:
$ kato config set cloud_controller max_user_drains 250
Each message in a log stream is prefixed with a key, identifying what type of message it is or to which log stream it belongs. The following keys are available for use in defining drains using the --prefix flag for kato log drain add).
Systail keys are configurable.
apptail.<app.id>
The list above shows the default systail keys. These can keys can be modified with the kato config command to add arbitrary system log files to the stream or change the log file source for an existing key.
To retrieve the current list of log files being streamed:
$ kato config get logyard systail/log_files
To remove a log file from the stream:
$ kato config del logyard systail/log_files/dpkg
To add a new log file to the stream:
$ kato config set logyard systail/log_files/dpkg /var/log/dpkg.log
Restart the systail process after adding or removing log files:
$ kato process restart systail
Note
Do not remove the default Stackato log stream keys (i.e. anything in the systail list above) as this would affect the output of kato tail.