Streaming Application Logs with Fluentd

Page last updated:

Fluentd is an open source log collector that allows you to implement unified logging layers. With Fluentd, you can stream application logs to different backends or services like Elasticsearch, HDFS and Amazon S3. This topic explains how to integrate Fluentd with Cloud Foundry applications.

Step 1: Create a Cloud Foundry Syslog Drain for Fluentd

  1. In Cloud Foundry, create a syslog drain user-provided service instance as described in Using Third-Party Log Management Services.

  2. Choose one or more applications whose logs you want to drain to Fluentd through the service.

  3. Bind each app to the service instance, and restart the app.

  4. Note the GUID for each app, the IP address of the Loggregator host, and the port number for the service.

  5. Locate the port number in the syslog URL. For example:

    syslog://logs.example.com:5140

Step 2: Set up Fluentd for Cloud Foundry

This section assumes you have an active Fluentd instance running. If you do not have an active Fluentd instance, refer to the Fluentd Documentation/Install steps for more details.

Fluentd comes with native support for syslog protocol. To set up Fluentd for Cloud Foundry, configure the syslog input of Fluentd as follows.

  1. In your main Fluentd configuration file, add the following source entry:

    <source>
      @type syslog
      port 5140
      bind 0.0.0.0
      tag cf.app
      protocol_type udp
    </source>
    
  2. Restart the Fluentd service.

Note: The Fluentd syslog input plugin supports udp and tcp options. Make sure to use the same transport that Cloud Foundry is using.

Fluentd will start listening for Syslog message on port 5140 and tagging the messages with cf.app, which can be used later for data routing. For more details about the full setup for the service, refer to the Config File article.

If your goal is to use an Elasticsearch or Amazon S3 backend, read the following guide: http://www.fluentd.org/guides/recipes/elasticsearch-and-s3

Create a pull request or raise an issue on the source for this page in GitHub