Troubleshooting Application Deployment and Health

Page last updated:

Refer to this topic for help diagnosing and resolving common issues when you deploy and run applications on Cloud Foundry.

Common Issues

The following sections describe common issues you might encounter when attempting to deploy and run your application, and possible resolutions.

cf push Times Out

If your deployment times out during the upload or staging phase, you may receive one of the following error messages:

  • 504 Gateway Timeout
  • Error uploading application
  • Timed out waiting for async job JOB-NAME to finish

If this happens, do the following:

  • Check your network speed. Depending on the size of your application, your cf push could be timing out because the upload is taking too long. We recommended an Internet connection speed of at least 768 KB/s (6 Mb/s) for uploads.

  • Make sure you are pushing only needed files. By default, cf push will push all the contents of the current working directory. Make sure you are pushing only the directory for your application. If your application is too large, or if it has many small files, Cloud Foundry may time out during the upload. To reduce the number of files you are pushing, ensure that you push only the directory for your application, and remove unneeded files or use the .cfignore file to specify excluded files.

  • Set the CF_STAGING_TIMEOUT and CF_STARTUP_TIMEOUT environment variables. By default your app has 15 minutes to stage and 5 minutes to start. You can increase these times by setting CF_STAGING_TIMEOUT and CF_STARTUP_TIMEOUT. Type cf push -h at the command line for more information.

  • If your app contains a large number of files, try pushing the app repeatedly. Each push uploads a few more files. Eventually, all files have uploaded and the push succeeds. This is less likely to work if your app has many small files.

App Too Large

If your application is too large, you may receive one of the following error messages on cf push:

  • 413 Request Entity Too Large
  • You have exceeded your organization's memory limit

If this happens, do the following:

  • Make sure your org has enough memory for all instances of your app. You will not be able to use more memory than is allocated for your organization. To view the memory quota for your org, use cf org ORG_NAME.

    Your total memory usage is the sum of the memory used by all applications in all spaces within the org. Each application’s memory usage is the memory allocated to it multiplied by the number of instances. To view the memory usage of all the apps in a space, use cf apps.

  • Make sure your application is less than 1 GB. By default, Cloud Foundry deploys all the contents of the current working directory. To reduce the number of files you are pushing, ensure that you push only the directory for your application, and remove unneeded files or use the .cfignore file to specify excluded files. The following limits apply:

    • The app files to push cannot exceed 1 GB.
    • The droplet that results from compiling those files cannot exceed 1.5 GB. Droplets are typically a third larger than the pushed files.
    • The combined size of the app files, compiled droplet, and buildpack cache cannot total more than 4 GB of space during staging.

Unable to Detect a Supported Application Type

If Cloud Foundry cannot identify an appropriate buildpack for your app, you will see an error message that states Unable to detect a supported application type.

You can view what buildpacks are available with the cf buildpacks command.

If you see a buildpack that you believe should support your app, refer to the buildpack documentation for details about how that buildpack detects applications it supports.

If you do not see a buildpack for your app, you may still be able to push your application with a custom buildpack using cf push -b with a path to your buildpack.

App Deploy Fails

Even when the deploy fails, the app might exist on PWS. Run cf apps to review the apps in the currently targeted org and space. You might be able to correct the issue using the CLI or Apps Manager, or you might have to delete the app and redeploy.

Common reasons deploying an app fails include the following:

  • You did not successfully create and bind a needed service instance to the app, such as a PostgreSQL or MongoDB service instance. Refer to Step 3: Create and Bind a Service Instance for a RoR Application.
  • You did not successfully create a unique URL for the app. Refer to the troubleshooting tip App Requires Unique URL.

App Requires Unique URL

PWS requires that each app that you deploy has a unique URL. Otherwise, the new app URL collides with an existing app URL and PWS cannot successfully deploy the app. You can resolve this issue by running cf push with either of the following flags to create a unique URL:

  • -n to assign a different HOST name for the app.
  • --random-route to create a URL that includes the app name and random words. Using this option might create a long URL, depending on the number of words that the app name includes.

App Fails to Start

After cf push stages the app and uploads the droplet, the app may fail to start, commonly with a pattern of starting and crashing similar to the following example:

-----> Uploading droplet (23M)
...
0 of 1 instances running, 1 starting
0 of 1 instances running, 1 down
...
0 of 1 instances running, 1 failing
FAILED
Start unsuccessful

If this happens, try the following:

  • Find the reason app is failing and modify your code. Run cf events APP-NAME and cf logs APP-NAME --recent and look for messages similar to this:

    2014-04-29T17:52:34.00-0700   app.crash          index: 0, reason: CRASHED,
    exit_description: app instance exited, exit_status: 1
    
    These messages may identify a memory or port issue. If they do, take that as a starting point when you re-examine and fix your application code.

  • Make sure your application code uses the PORT environment variable. Your application may be failing because it is listening on the wrong port. Instead of hard coding the port on which your application listens, use the PORT environment variable.

    For example, this Ruby snippet assigns the port value to the listen_here variable: listen_here = ENV['PORT']

    For more examples specific to your application framework, see the appropriate buildpacks documentation for your app’s language.

  • Make sure your app adheres to the principles of the Twelve-Factor App and Prepare to Deploy an Application. These texts explain how to prevent situations where your app builds locally but fails to build in the cloud.

  • Verify the timeout configuration of your app. Application health checks use a timeout setting when an app starts up for the first time. See Using Application Health Checks. If an application fails to start up due to health check timeout, you might see messages in the logs similar to the following:

    2017-01-30T14:07:20.39-0800 [CELL/0]     OUT Creating container
    2017-01-30T14:07:20.65-0800 [CELL/0]     OUT Successfully created container
    2017-01-30T14:07:22.30-0800 [CELL/0]     OUT Starting health monitoring of container
    2017-01-30T14:08:23.52-0800 [CELL/0]     ERR Timed out after 1m0s: 
    health check never passed.
    2017-01-30T14:08:23.57-0800 [CELL/0]     OUT Destroying container
    2017-01-30T14:08:23.59-0800 [API/0]      OUT Process has crashed with type: "web"
    2017-01-30T14:08:23.59-0800 [CELL/0]     OUT Creating container
    2017-01-30T14:08:23.60-0800 [API/0]      OUT App instance exited with guid 91086440-bac0-44f0-808f-a034a1ec2ed0 
    payload: {"instance"=>"", "index"=>0, "reason"=>"CRASHED",
    "exit_description"=>"2 error(s) occurred:\n\n* 1 error(s) 
    occurred:\n\n* Exited with status 6\n* 2 error(s) 
    occurred:\n\n* cancelled\n* cancelled", "crash_count"=>1,
    "crash_timestamp"=>1485814103565763172,
    "version"=>"3e6e4232-7e19-4168-9583-1176833d2c71"}
    2017-01-30T14:08:23.83-0800 [CELL/0]     OUT Successfully destroyed container
    2017-01-30T14:08:23.84-0800 [CELL/0]     OUT Successfully created container
    2017-01-30T14:08:25.41-0800 [CELL/0]     OUT Starting health monitoring of container
    

App consumes too much memory, then crashes

An app that cf push has uploaded and started can crash later if it uses too much memory.

Make sure your app is not consuming more memory than it should. When you ran cf push and cf scale, that configured a limit on the amount of memory your app should use. Check your app’s actual memory usage. If it exceeds the limit, modify the app to use less memory.

Routing Conflict

PWS allows multiple apps, or versions of the same app, to be mapped to the same route. This feature enables Blue-Green deployment. For more information see Using Blue-Green Deployment to Reduce Downtime and Risk.

Routing multiple apps to the same route may cause undesirable behavior in some situations by routing incoming requests randomly to one of the apps on the shared route.

If you suspect a routing conflict, run cf routes to check the routes in your installation.

If two apps share a route outside of a Blue-Green deploy strategy, choose one app to re-assign to a different route and follow the procedure below:

  1. Run cf unmap-route YOUR-APP-NAME OLD-ROUTE to remove the existing route from that app.

  2. Run cf map-route YOUR-APP-NAME NEW-ROUTE to map the app to a new, unique route.

Gather Diagnostic Information

Use the techniques in this section to gather diagnostic information and troubleshoot app deployment issues.

Examine Environment Variables

cf push deploys your application to a container on the server. The environment variables in the container govern your application.

You can set environment variables in a manifest created before you deploy. See Deploying with Application Manifests.

You can also set an environment variable with a cf set-env command followed by a cf push command. You must run cf push for the variable to take effect in the container environment.

Use the cf env command to view the environment variables that you have set using the cf set-env command and the variables in the container environment:

$ cf env my-app
 Getting env variables for app my-app in org My-Org / space development as admin...
 OK

 System-Provided:
 {
   "VCAP_SERVICES": {
     "p-mysql-n/a": [
       {
         "credentials": {
           "uri":"postgres://lrra:e6B-X@p-mysqlprovider.example.com:5432/lraa
         },
         "label": "p-mysql-n/a",
         "name": "p-mysql",
         "syslog_drain_url": "",
         "tags": ["postgres","postgresql","relational"]
       }
     ]
   }
 }

 User-Provided:
 my-env-var: 100
 my-drain: http://drain.example.com

View Logs

To view app logs streamed in real-time, use the cf logs APP-NAME command.

To aggregate your app logs to view log history, bind your app to a syslog drain service. For more information, see Streaming Application Logs to Log Management Services.

Note: The Diego architecture does not support the cf files command.

Trace Cloud Controller REST API Calls

If a command fails or produces unexpected results, re-run it with HTTP tracing enabled to view requests and responses between the cf CLI and the Cloud Controller REST API.

For example:

  • Re-run cf push with -v:

    cf push APP-NAME -v

  • Re-run cf push while appending API request diagnostics to a log file:

    CF_TRACE=PATH-TO-TRACE.LOG cf push APP-NAME

These examples enable HTTP tracing for a single command only. To enable it for an entire shell session, set the variable first:

export CF_TRACE=true

export CF_TRACE=PATH-TO-TRACE.LOG

Note: CF_TRACE is a local environment variable that modifies the behavior of the cf CLI. Do not confuse CF_TRACE with the variables in the container environment where your apps run.

Analyze Zipkin Trace IDs

When the Zipkin feature is enabled in Cloud Foundry, the Gorouter adds or forwards Zipkin trace IDs and span IDs to HTTP headers. For more information about what the Gorouter provides in the HTTP header, see the HTTP Headers section of the HTTP Routing topic.

After adding Zipkin HTTP headers to app logs, developers can use cf logs myapp to correlate the trace and span ids logged by the Gorouter with the trace ids logged by their app. To correlate trace ids for a request through multiple apps, each app must forward appropriate values for the headers with requests to other applications.

Use Troubleshooting Commands

You can investigate app deployment and health using the cf CLI.

Some cf CLI commands may return connection credentials. Remove credentials and other sensitive information from command output before you post the output a public forum.

  • cf apps: Returns a list of the applications deployed to the current space with deployment options, including the name, current state, number of instances, memory and disk allocations, and URLs of each application.

  • cf app APP-NAME: Returns the health and status of each instance of a specific application in the current space, including instance ID number, current state, how long it has been running, and how much CPU, memory, and disk it is using.

    Note: CPU values returned by cf app show the total usage of each app instance on all CPU cores on a host VM, where each core contributes 100%. For example, the CPU of a single-threaded app instance on a Diego cell with one core cannot exceed 100%, and four instances sharing the cell cannot exceed an average CPU of 25%. A multi-threaded app instance running alone on a cell with eight cores can draw up to 800% CPU.

  • cf env APP-NAME: Returns environment variables set using cf set-env and variables existing in the container environment.

  • cf events APP-NAME: Returns information about application crashes, including error codes. Shows that an app instance exited. For more detail, look in the application logs. See https://github.com/cloudfoundry/errors for a list of Cloud Foundry errors.

  • cf logs APP-NAME --recent: Dumps recent logs. See Viewing Logs in the Command Line Interface.

  • cf logs APP-NAME: Returns a real-time stream of the application STDOUT and STDERR. Use Ctrl-C (^C) to exit the real-time stream.

  • cf files APP-NAME: Lists the files in an application directory. Given a path to a file, outputs the contents of that file. Given a path to a subdirectory, lists the files within. Use this to explore individual logs.

Note: Your application should direct its logs to STDOUT and STDERR. The cf logs command also returns messages from any log4j facility that you configure to send logs to STDOUT.

Access Apps with SSH

If you need to troubleshoot an instance of an app, you can gain SSH access to the app with the SSH proxy and daemon. See the Application SSH Overview topic for more information.

Send Requests to App Instance

To obtain debug data without SSH, you can make HTTP requests to a specific instance of an app by using the X-CF-APP-INSTANCE HTTP header. See the App Instance Routing section of the HTTP Routing topic for more information.

Create a pull request or raise an issue on the source for this page in GitHub