Streaming App Logs to Azure OMS Log Analytics (Beta)

Warning: The OMS Log Analytics Firehose Nozzle is currently intended for evaluation and test purposes only. Do not use this product in a production environment.

This topic explains how to integrate your Pivotal Web Services (PWS) apps with OMS Log Analytics.

Operations Management Suite (OMS) Log Analytics is a monitoring service for Microsoft Azure. The OMS Log Analytics Firehose Nozzle is a PWS component that forwards metrics from the Loggregator Firehose to OMS Log Analytics.

This topic assumes you are using the latest version of the Cloud Foundry Command Line Interface (cf CLI) and a working PWS deployment on Azure.

Step 1: Create an OMS Workspace in Azure

To create an OMS workspace, see Get started with Log Analytics in the Microsoft Azure documentation.

Step 2: Deploy the Nozzle to PWS

To deploy the OMS Log Analytics Firehose nozzle to PWS:

  1. Authenticate to your PWS instance. Run:

    cf login -a https://api.YOUR-DOMAIN -u YOUR-USERNAME --skip-ssl-validation
    

    Where:

    • YOUR-DOMAIN is your domain.
    • YOUR-USERNAME is your PWS username.
  2. To create a new PWS user and grant it access to the Loggregator Firehose using the UAA CLI (UAAC):

    1. Target your UAA server by running:

      uaac target uaa.YOUR-DOMAIN --skip-ssl-validation
      

      Where YOUR-DOMAIN is your domain.

    2. Obtain an access token for the admin client by running:

      uaac token client get admin
      
    3. Create a new user by running:

      cf create-user USERNAME PASSWORD
      

      Where:

      • USERNAME is a new username.
      • PASSWORD is a password.
    4. Grant the new user admin permissions by running:

      uaac member add cloud_controller.admin USERNAME
      

      Where USERNAME is the username you set in the previous step.

    5. Grant the new user permission to read logs from the Loggregator Firehose endpoint by running:

      uaac member add doppler.firehose USERNAME
      

      Where USERNAME is the username you set.

  3. Download the OMS Log Analytics Firehose Nozzle BOSH release from Github. Clone the repository and navigate to the oms-log-analytics-firehose-nozzle directory by running:

    git clone https://github.com/Azure/oms-log-analytics-firehose-nozzle.git
    cd oms-log-analytics-firehose-nozzle
    
  4. Set the following environment variables in the OMS Log Analytics Firehose Nozzle manifest:

    Environment Variable Description
        applications:
        - name: oms_nozzle
        ...
        env:
          OMS_WORKSPACE: YOUR-WORKSPACE-ID
          OMS_KEY: YOUR-OMS-KEY
    Enter the ID and key value for your OMS workspace.
    OMS_POST_TIMEOUT: 10s
    (Optional) Set the HTTP post timeout for sending events to OMS LogmAnalytics. The default value is 10 seconds.
    OMS_BATCH_TIME: 10s
    (Optional) Set the interval for posting a batch to OMS. The default value is 10 seconds.

    For more information, see Configure Additional Logging.
    OMS_MAX_MSG_NUM_PER_BATCH: 1000
    (Optional) Set the maximum number of messages to include in an OMS batch. The default amount is 1000.

    For more information, see Configure Additional Logging.
        FIREHOSE_USER: YOUR-FIREHOSE-USER
        FIREHOSE_USER_PASSWORD: YOUR-FIREHOSE-PASSWORD
    Enter the username and password for the Firehose user you created in Step 2c.
    API_ADDR: https://api.YOUR-DOMAIN
    Enter the URL of your API endpoint.
    DOPPLER_ADDR: wss://doppler.YOUR-DOMAIN:443
    Enter the URL of your Loggregator Traffic Controller endpoint.
    EVENT_FILTER: YOUR-LIST
    (Optional) Enter the event types you want to filter out in a comma-separated list. The valid event types are METRIC, LOG, and HTTP.
    IDLE_TIMEOUT: 60s
    (Optional) Set the duration for the Firehose keep-alive connection. The default time is 60 seconds.
    SKIP_SSL_VALIDATION: TRUE-OR-FALSE
    Set this value to TRUE to allow insecure connections to the UAA and the Traffic Controller. To block insecure connections to the UAA and Traffic Controller, set this value to FALSE.
    LOG_LEVEL: INFO
    (Optional) Change this value to increase or decrease the amount of logs. Valid log levels in increasing order include INFO, ERROR, and DEBUG. The default value is INFO.
    LOG_EVENT_COUNT: TRUE-OR-FALSE
    Set this value to TRUE to log the total count of events that the nozzle has sent, received, and lost. OMS logs this value as CounterEvents.

    For more information, see Configure Additional Logging.
    LOG_EVENT_COUNT_INTERVAL: 60s
    (Optional) Set the time interval for logging the event count to OMS. The default interval is 60 seconds.

    For more information, see Configure Additional Logging.
  5. Push the app by running:

    cf push
    

Step 3: View Logs in OMS Portal

Import the PWS OMS view to your OMS Portal to view visualized logs and metrics. You can also create alert rules for specific events.

Note: The OMS view of PWS is not yet available in the OMS Solutions Gallery. You can add it manually to view your logs in OMS Portal.

Import the OMS View

To import the OMS view:

  1. From the main OMS Overview page, go to View Designer.

  2. Click Import.

  3. Click Browse.

  4. Select the Cloud Foundry (Preview).omsview file.

  5. Save the view. The main OMS Overview page displays the Tile.

  6. Click the Tile to view visualized metrics.

For more information, see Create custom views by using View Designer in Azure Monitor in the Azure documentation.

Create Alert Rules

For more information about OMS Log Analytics alerts, see Overview of alerts in Microsoft Azure in the Azure documentation.

Set Alert Queries

This section includes example queries that operators can set in the OMS Portal.

  • The following query alerts the operator when the nozzle sends a slowConsumerAlert to OMS:

    Type=CF_ValueMetric_CL Name_s=slowConsumerAlert
    
  • The following query alerts the operator when Loggregator sends an LGR to indicate problems with the logging process:

    Type=CF_LogMessage_CL SourceType_s=LGR MessageType_s=ERR
    
  • The following query alerts the operator when the number of lost events reaches a certain threshold, specified in the OMS Portal:

    Type=CF_CounterEvent_CL Job_s=nozzle Name_s=eventsLost
    
  • The following query alerts the operator when the nozzle receives the TruncatingBuffer.DroppedMessages CounterEvent:

    Type=CF_CounterEvent_CL Name_s="TruncatingBuffer.DroppedMessages"
    

(Optional) Step 4: Configure Additional Logging

OMS Log Analytics Firehose Nozzle forwards metrics from the Loggregator Firehose to OMS with minimal processing, but the nozzle can push additional metrics to OMS.

Log Sent, Received, and Lost Events

If you set the LOG_EVENT_COUNT environment variable to TRUE in the manifest, the nozzle periodically sends the count of sent, received, and lost events to OMS. The value you set for the LOG_EVENT_COUNT_INTERVAL determines how frequently the nozzle sends the count.

Note: The nozzle does not count CounterEvents themselves in the sent, received, or lost event count.

The nozzle sends the count as a CounterEvent with a CounterKey of one of the following:

CounterEvent CounterKey
nozzle.stats.eventsReceived The number of events the Firehose has received during the interval
nozzle.stats.eventsSent The number of events the nozzle has successfully sent to OMS during the interval
nozzle.stats.eventsLost The number of events the nozzle has tried to send to OMS during the interval, but failed to send after 4 attempts

In most cases, the total count of eventsSent plus eventsLost is less than the total eventsReceived at the same time. The nozzle buffers some messages and posts them in a batch to OMS. Operators can adjust the buffer size by adjusting the OMS_BATCH_TIME and OMS_MAX_MSG_NUM_PER_BATCH environment variables in the manifest).

Log Slow Consumer Alerts

Note: The nozzle does not count ValueMetrics in the sent, received, or lost event count.

Loggregator sends the nozzle a slowConsumerAlert in the following situations:

  • WebSocket sends the error code ClosePolicyViolation (1008).

  • The nozzle receives a CounterEvent with the value TruncatingBuffer.DroppedMessages.

In either case, the nozzle sends the slowConsumerAlert event to OMS as the following ValueMetric:

ValueMetric MetricKey
nozzle.alert.slowConsumerAlert 1

(Optional) Step 5: Scale the Deployment

Scale the Nozzle

If the nozzle is unable to keep up with processing logs from the Firehose, Loggregator alerts the nozzle. When the nozzle receives the alert, it sends a slowConsumerAlert to OMS. If this happens, scaling up the nozzle minimizes data loss.

If an operator chooses to scale up their deployment, the Firehose evenly distributes events across all instances of the nozzle.

Operators can create an alert rule for the slowConsumerAlert message. For more information, see Create Alert Rules.

Scale Loggregator

Loggregator sends LGR log messages to indicate problems with the logging process.

Operators can create an alert rule for the LGR message. For more information, see Create Alert Rules.