Monitor Your Containers with Sysdig

Quick. What are your containers doing right now? Do you know? How are they performing? What system calls and events are associated with those microservices you’ve deployed? If you don’t know, then consider yourself a bit behind the curve.

Fret not, there are tools available to help you dig out the details so that you can have every bit of information you need at your fingertips. One such tool is Sysdig. By design, sysdig collects system calls and events directly from the Linux kernel (as opposed to /proc) and does (by itself) what strace, tcpdump, htop, iftop, lsof, and Wireshark does. In other words, you can use one tool, instead of six. Even better, since 2015, Sysdig is aware of containers. So when you need to troubleshoot those microservices on Linux, Sysdig has your back. And although Sysdig is a command-line tool, it does include an ncurses user interface to make viewing this information even easier.

I’m going to walk you through the installation and usage of Sysdig on my server operating system of choice, Ubuntu Server 20.04. Of course, Sysdig can be installed on either Debian- or Red Hat-based distributions. To make this work, you’ll need a running instance of Ubuntu Server, as well as a user with sudo privileges. You’ll also need a container runtime engine (so you can deploy and monitor containers). I’ll be demonstrating with the Docker engine and a WordPress deployment.

Installing Sysdig

The first thing we’ll do is install Sysdig. To do that, log into your Ubuntu Server and issue the command:

curl -s https://s3.amazonaws.com/download.draios.com/stable/install-sysdig | sudo bash

That should install all of the dependencies as well as the latest release of Sysdig.

Deploying a WordPress Docker Container

Let’s deploy a WordPress Docker container, so we have something to monitor. This is a bit more involved than deploying a one-off container, but it’s worth knowing how to do it.

First, pull the MariaDB container with:

sudo docker pull mariadb

Next, create the necessary folders that will house the WordPress data (for persistent storage) with the commands:

sudo mkdir /opt/wordpress

sudo mkdir -p /opt/wordpress/database

sudo mkdir -p /opt/wordpress/html

Create the MariaDB container:

docker run -e MYSQL_ROOT_PASSWORD=PWORD1 -e MYSQL_USER=wpuser -e MYSQL_PASSWORD=PWORD2 -e MYSQL_DATABASE=wordpress_db -v /opt/wordpress/database:/var/lib/mysql --name wordpressdb -d mariadb

Where PWORD1 and PWORD2 are unique/strong passwords.

Pull the latest version of WordPress with:

docker pull wordpress:latest

Deploy the WordPress container:

docker run -e WORDPRESS_DB_USER=wpuser -e WORDPRESS_DB_PASSWORD=PWORD2 -e WORDPRESS_DB_NAME=wordpress_db -p 8081:80 -v /opt/wordpress/html:/var/www/html --link wordpressdb:mysql --name wpcontainer -d wordpress

Where “PWORD 2” is the password you set for the wpuser in the previous docker run command.

You can now point a browser to http://SERVER:8081 (Where SERVER is the IP address of the hosting server) and finish the WordPress installation.

How to Use Sysdig

Now that we have something to monitor, let’s see how Sysdig is used. I prefer using the ncurses command-line interface, so instead of using the sysdig command, we’ll use csysdig.

First, let’s get a listing of our currently running containers with:

sudo csysdig -vcontainers

If the only containers you’ve deployed are for the WordPress site, you should see two containers running:

• wordpress
• mariadb

The listing will also tell you how much CPU each container is using, the number of associated processes, threads, virtual memory, resident memory, files, the engine being used, and the container ID. So, already we’re seeing value in using Sysdig.

To exit from that view, use the [Ctrl]+ keyboard shortcut.

Let’s get even more information. To view every process associated with a container, issue the command:

sudo csysdig -pc

This will list (for each process found):

• PID (Process ID)
• PPID (Parent Process ID)
• VPID (Virtual Process ID)
• CPU (CPU used by the container)
• USER (the user that launched the container)
• Virtual Memory
• RES (Resident memory assigned)
• Files (files used by the container)
• Net (total network I/O used by the container)
• Container (container name)
• Command (command used by the container)

The problem with the above command is that it might give you too much information. Say, for example, you only want to view the information associated with one particular container (in our case the container named wpress. For that, you could issue the command:

sudo sysdig -pc -c topprocs_cpu container.name=wpcontainer

From that command you should see listed:

• CPU%
• Process
• Host_pid
• Container_pid
• Container.name

Maybe you need to check on the net I/O of a particular container. Again, sticking with our example, issue:

sudo sysdig -pc -c topprocs_net container.name=wpcontainer

Or

sudo sysdig -pc -c topprocs_net container.name=wordpressdb

The above commands will list:

• Bytes
• Process
• Host_pid
• Container_pid
• container.name

The one thing to know about the above commands is that you’ll only see data if there is actual network traffic.

What if you want to view the files associated with I/O for the WordPress container? For that issue the command:

sudo sysdig -pc -c topfiles_bytes container.name=wpcontainer

Again, this is another command that will only show output if files are being used.

You might also want to switch up the view in csysdig. While viewing one of the above commands, hit F2 to open the menu of existing csysdig views (Figure 1).

You should see container-centric views for things like Containers, Containers Errors, K8s Controllers, K8s Deployments, K8s Namespaces, etc.

But don’t think you have to first know how to issue the csysdig command with having to remember the necessary arguments and options. In fact, you can simply issue:

sudo csysdig

Once the command is running hit F2 (on your keyboard) to select the view you want. This makes it much easier to use the command (without having to remember the available options and switches).

Conclusion

Sysdig and csysdig are powerful tools that can help you monitor and troubleshoot your container deployments from the terminal window. It’s simple to install and use. To find out more about what csysdig can do for you, issue the command:

man csysdig

or

man sysdig