# Monitoring Docker Volume Usage

# Overview

Monitoring disk space usage with a breakdown by individual container ensures continuous service availability by preventing space leakages in containers with data (persistent) volumes.

While the Docker command line includes a way to obtain container sizes using the --size option, the command can take several minutes to complete while significantly overloading the host disk subsystem and slowing down Docker Engine API response times. The CLI output does not expose space usage by volume and requires parsing size units (KB, MB, GB).

docker ps -s --format "{{.ID}}\t{{.Names}}\t{{.Size}}"

a832af264060 JENKINS_atsd-api-test_ 132.3 KB (virtual 984.1 MB)
d7f87797f36e distracted_williams 323 MB (virtual 1.821 GB)
f53cf366a4c6 stupefied_hamilton 323 MB (virtual 1.821 GB)
d6d362779455 sad_spence 446.5 MB (virtual 1.944 GB)
fa510d6c9a9a ve-7.1.1 1.518 GB (virtual 2.244 GB)
75caaa907eab axibase_collector 5.942 GB (virtual 20.38 GB)
...

For example, on a Docker host where the /var/lib/docker size is 30 GB with 20 running containers and 80 in total, the initial execution of the docker ps -as command takes more than 2 minutes while fully loading the host I/O.

$ time docker ps -as
CONTAINER ID   IMAGE                             COMMAND                  CREATED         STATUS         PORTS                     SIZE
83010924db3c   axibase/atsd_package_validation   "/bin/bash /root/chec"   5 minutes ago   Up 3 minutes   atsd_package_validation   561.7 MB (virtual 818.9 MB)
...
real 2m51.218s
user 0m0.026s
sys 0m0.032s

Executing API requests with the &size=1 parameter typically requires even more time than the docker ps -as command and can cause timeout issues for API clients.

# Collecting Container Sizes with Axibase Collector

To gather container sizes using Axibase Collector, set Container Size Interval to an interval 60 minutes or longer.

To minimize the load on the disk subsystem, this interval is applied to running containers, while the size of inactive containers is collected less frequently, as specified by Property Interval.

# Container Size Metrics

Axibase Collector collects sizes for both individual containers as well as the total for running and all containers.

Metric Name	Description
docker.fs.size.rw	The total size of all the files in the container, in bytes. The entire filesystem of the container exported in tarball form is about this size.
docker.fs.size.rootfs	The size of the files which have been created or changed, if you compare the container to the base image. Just after creation, this number must be zero; as you modify (or create) files, this number increases.
docker.fs.total.size.rw	The total size of all the files for all containers, in bytes. Σ docker.fs.size.rw for all containers.
docker.fs.total.size.rootfs	The size of the files created or changed for all containers. Σ docker.fs.size.rootfs for all containers.
docker.fs.running.size.rw	The total size of all the files for all running containers, in bytes. Σ docker.fs.size.rw for running containers.
docker.fs.running.size.rootfs	The size of the files which have been created or changed for running containers. Σ docker.fs.size.rootfs for running containers.

ChartLab example

# Alternative Method

One of the "lesser evil" alternatives is to calculate disk usage of /var/lib/docker subdirectories using the du command. This requires superuser privileges.

sudo bash -c 'du -hs /var/lib/docker/volumes/*'

Note: Use the bash -c wrapper here as a permission-safe way to pass the * wildcard.

177M /var/lib/docker/volumes/dd1e8b1942e204054b8a56219e523834d35bd8e84283720daf227823eae9b21f
174M /var/lib/docker/volumes/de9a1746ea49e702accfb732ee74354c6291c2356a5319693868533fbeb40765
363M /var/lib/docker/volumes/df81a9b753aa9464935af03b8e3dc57fa53cacaa1910a80dc4f1e6b9f952fb77
1.7G /var/lib/docker/volumes/df8948c840a882070c9ed10f01bf3c6b594a0095f236f47af124cf9d76dee165
1.1G /var/lib/docker/volumes/e1f9859a2a58a8071805597e3e4ea71d45e67cda13dbb9630742e654c379d544
307M /var/lib/docker/volumes/eb3952dec29e293cfae8149b20c40859ac944723abd28666e093ab1d76b43a0c

The following collector script executes the ds command to calculate total disk usage of each subdirectory in the /var/lib/docker/volumes/ directory, as well as computes the percentage of the total size of the underlying file system, used by each volume.

# Running

• Print commands to stdout:

  sudo ./docker_volume_collect.sh
  

• Print commands to file:

  sudo ./docker_volume_collect.sh >> /tmp/docker-volumes.out
  

• Send commands to ATSD:

  sudo ./docker_volume_collect.sh > /dev/tcp/{atsd_hostname}/8081
  

# Sample commands

series e:f867330fd3e9f103388ed84325c4432a4dce2cdebfb7db22f46940bcecc81225 m:docker.volume.used=179724 d:2016-12-27T11:12:39Z t:docker-host="nurswghbs001"
series e:f867330fd3e9f103388ed84325c4432a4dce2cdebfb7db22f46940bcecc81225 m:docker.volume.used_percent=0.0175442 d:2016-12-27T11:12:39Z t:docker-host="nurswghbs001"
series e:nurswghbs001 m:docker.volume.total_used=32424032 d:2016-12-27T11:12:39Z
series e:nurswghbs001 m:docker.volume.total_used_percent=3.16514 d:2016-12-27T11:12:39Z

# Scheduling

• To send commands to ATSD on schedule, open crontab:

  su root
  crontab -e
  

• Add a task to collect data every 15 minutes:

  */15 * * * * /bin/bash -l -c '/opt/scripts/docker_volume_collect.sh > /dev/tcp/{atsd_hostname}/8081'
  

# Container Volume Metrics

Metric Name	Description
docker.volume.fs.size	Total size (used + available, in bytes) of the file system where the /var/lib/docker directory is located. Collected for the entire docker host.
docker.volume.total_used	Total space (in bytes) used by the /var/lib/docker directory. Collected for the entire docker host.
docker.volume.total_used_percent	Percentage of space used by the /var/lib/docker directory in the file system where the /var/lib/docker directory is located. Calculated as docker.volume.total_used/docker.volume.fs.size * 100. Collected for the entire docker host.
docker.volume.used	Space used by all files in the given volume (in bytes).
docker.volume.used_percent	Space used by files in the given volume as percentage of the total size of the file system where the /var/lib/docker directory is located. Calculated as docker.volume.used/docker.volume.fs.size * 100.

# Volume View

To display volume sizes, import the updated Entity View for Docker Volumes. Open Entity Views > Configure and click Import on the split-button at the bottom of the screen with the Replace Existing Entity Views option enabled.

# Volume Disk Rules

Import the rules file to raise an alert whenever a volume consumes more than 50% of total file system size.

Rule Name	Description
docker-host-volume-space-low	Raise an alert if the total size of the /var/lib/docker directory exceeds 60% of the total space on the file system.
docker-volume-space-leak	Raise an alert if the volume consumes more than 25% of the total space on the file system where /var/lib/docker is located.

# Monitor Volume Sizes using a Container

As an alternative to running the du script on the Docker host, you can launch an Axibase Collector container with the /var/lib/docker/volumes directory mounted in read-only mode.

• Replace the atsd_hostname placeholder with the actual ATSD hostname in the command below.

• Replace username and password with collector account credentials.

  docker run \
    --detach \
    --publish 9443:9443 \
    --restart=always \
    --name=axibase-collector \
    --volume /var/lib/docker/volumes:/var/lib/docker/volumes:ro \
    axibase/collector \
     -atsd-url=https://username:password@atsd_hostname:8443 \
     -job-enable=docker-socket
  

• Start the container.

• Copy the script into the container.

  Here, there are two alternatives:

  a) Copy the script file to the Docker host and then copy the file into the container:

  docker cp /tmp/docker_volume_collect.sh axibase-collector:/opt/axibase-collector/ext/docker_volume_collect.sh
  

  b) Download the script file directly into the container:

  docker exec axibase-collector wget https://raw.githubusercontent.com/axibase/axibase-collector/master/jobs/docker/docker_volume_collect.sh -P /opt/axibase-collector/ext
  

• Grant execute permissions to the script:

  docker exec axibase-collector chmod +x  /opt/axibase-collector/ext/docker_volume_collect.sh
  

• Run the script once manually to validate import:

  docker exec axibase-collector /opt/axibase-collector/ext/docker_volume_collect.sh docker_hostname tcp://atsd_hostname:8081
  

• Log in to ATSD and verify that the following metrics are available:

    docker.volume.fs.size
    docker.volume.total_used
    docker.volume.total_used_percent
    docker.volume.used
    docker.volume.used_percent
  

• Open the cron file in the container shell:

  docker exec -it axibase-collector crontab -e
  

• Add the following lines to cron schedule:

  # Replace 'docker_hostname' with the hostname of the Docker host
  # Replace 'atsd_host' with ATSD hostname or IP address
  # Run script every 15 minutes
  */15 * * * * /opt/axibase-collector/ext/docker_volume_collect.sh docker_hostname tcp://atsd_hostname:8081
  # Empty line is required at the end of this file for a valid cron file
  

• Save the cron file.