Understanding GrootFS Disk Usage
Page last updated:
This topic explains the concepts related to GrootFS disk space management in Pivotal Web Services (PWS).
GrootFS is the container root filesystem management component for Garden. A container root filesystem or rootfs is often referred to as an image.
A GrootFS store is the directory in which rootfs layers and container images are cached. This directory is configured by GrootFS and mounted on an XFS-formatted volume by the Garden job during BOSH VM creation.
Individual container root filesystems are provided via OverlayFS mounts.
Supplying GrootFS with an already formatted XFS volume for its store is not yet supported for BOSH-controlled deployments.
GrootFS stores are initialized to use the entirety of
/var/vcap/data. If the
reserved_space_for_other_jobs_in_mb is not set high enough,
or if there are many images with few shared volumes, the store can use up all available space.
The thresholder component calculates and sets a value so that GrootFS’s garbage collector can attempt to ensure that a small reserved space is kept free for other jobs. GrootFS only tries to garbage collect when that threshold is reached. However, if all the rootfs layers are actively in use by images, then garbage collection cannot occur and that space is used up.
Underlying layers in rootfs images are known as
volumes in GrootFS.
They are read-only and their changesets are layered together through an OverlayFS mount
to create the root filesystems for containers.
When GrootFS writes each filesystem volume to disk, it also stores the number of bytes written
to a file in a
The size of an individual volume is available in its corresponding metadata file.
GrootFS also stores the SHA of each underlying volume used by an image in the
For each container, GrootFS mounts the underlying
volumes using overlay to a point in the
This mount point is the rootfs for the container and is read-write.
On disk, the read-write layer for each container can be found at
/var/vcap/data/grootfs/store/privileged/images/CONTAINER-ID/diff for privileged containers.)
When GrootFS calls on the built-in XFS quota tooling to get disk usage for a container,
it takes into account data written to those
diff directories and not the data in the read-only volumes.
clean is called in GrootFS, any layers that are not being used by an existing rootfs are deleted from the store.
The cleanup only takes into account the
volumes folders in the store.
For example, imagine that there are two rootfs images from different base images, Image A and Image B:
- Image A Layers: - layer-1 - layer-2 - layer-3 - Image B Layers: - layer-1 - layer-4 - layer-5
They have a layer in common, layer-1. And after deleting Image B, layer-4 and layer-5 can be collected by clean, but not layer-1 because Image A still uses that layer.
For more information, see the following sections of