Streaming App Logs with Fluentd
Pivotal Web Services End of Availability Announced
For more information, see Frequently Asked Questions.
Fluentd is an open source log collector that allows you to implement unified logging layers. With Fluentd, you can stream app logs to different backends or services like Elasticsearch, HDFS and Amazon S3. This topic explains how to integrate Fluentd with Cloud Foundry apps.
In Cloud Foundry, create a syslog drain user-provided service instance as described in Using Third-Party Log Management Services.
Choose one or more apps whose logs you want to drain to Fluentd through the service.
Bind each app to the service instance, and restart the app.
Note the GUID for each app, the IP address of the Loggregator host, and the port number for the service.
Locate the port number in the syslog URL. For example:
This section assumes you have an active Fluentd instance running. If you do not have an active Fluentd instance, refer to the Fluentd Documentation/Install steps for more details.
Fluentd comes with native support for syslog protocol. To set up Fluentd for Cloud Foundry, configure the syslog input of Fluentd as follows.
In your main Fluentd configuration file, add the following
<source> @type syslog port 5140 bind 0.0.0.0 tag cf.app protocol_type udp </source>
Restart the Fluentd service.
Note: The Fluentd syslog input plugin supports
tcp options. Make sure to use the same transport that Cloud Foundry is using.
Fluentd will start listening for Syslog message on port 5140 and tagging the messages with
cf.app, which can be used later for data routing. For more details about the full setup for the service, refer to the Config File article.
If your goal is to use an Elasticsearch or Amazon S3 backend, read the following guide: http://www.fluentd.org/guides/recipes/elasticsearch-and-s3