Container Security: A Troubling Tale, but Hope on the Horizon

As a data scientist studying container adoption, I spend a lot of time looking at what the facts have to tell us about the makeup of public containers that millions of developers use every day. So, this week I’m at KubeCon NA in Detroit, sharing our findings with the incredible Kubernetes community.

At Slim.AI, I spend my days exploring the realities developers encounter when running containers at scale. My mission is to provide a window into that world, and to that end, my team recently completed our second annual analysis of the world’s most popular public container images. Alongside this foundational dataset, we partnered with Dimension Research to survey more than 300 developers & DevOps professionals in a global randomized survey about supply chain security and how it relates to modern containerized applications.

The results from this body of research can be found in the Slim.AI Top Public Container Report 2022, and I shared the highlights during my keynote at KubeCon yesterday. Here’s an overview of what we discovered analyzing more than 800,000 unique container images over the past year and how our survey shed light on those findings. I suspect you’ll find the results as eye-opening as I did.

For starters, we found that:

• 60% of the top public containers have more vulnerabilities today than they did a year ago. And this is after a year of intense focus on the software supply chain and in the aftermath of multiple security incidents. Moreover, the remaining 40% of the containers in the study have not considerably improved in their security posture, either.

• 70% of the developers responding to our survey said their customers and end users are demanding that their containers have zero vulnerabilities. And on top of that, 88% stated that it is getting more challenging to ensure containers are free from vulnerabilities. The number one cited contributing factor is the complexity and numerous components with dependencies in these containers.

So this is how the story is unfolding: containers are becoming more vulnerable, customers are demanding perfection more than ever, and developers are feeling the pressure.

It will come as no surprise to you that containers are being used all the time, everywhere, for nearly every modern software project, across all verticals. In fact, developers are beginning to look at a technology like Docker as native to their workflows as tools like Git or an IDE. According to the Stack Overflow 2022 Developer Survey, Docker is the number one technology developers love and want to learn, followed by Kubernetes. We can also see this trend in the all-time pull volume on Docker Hub, which tripled in the last year. But while there are lots of tutorials out there for building and shipping containers, there’s precious little understanding around what’s in them. Much like cells in biology, containers are seen as these little, atomic units, building blocks of much larger systems: requests go in, data comes out. It’s just simple.

But then, something like Log4Shell happens and suddenly 2022 becomes the year of software supply chain security. As an industry, we have become more invested in not only the principal components and contributors in our software systems, but also their second-order effects. With each of these exploits, our industry achieves a renewed sense of awareness of the problem we collectively face.

So, how do we harness that renewed sense of awareness and sustainably change the security posture of containers at large? At Slim, our answer is a systematic focus on container security and optimization. My research team has been observing and deconstructing hundreds of thousands of containers since 2020 with a goal to understand how they are changing over time and what makes these containers developer-friendly and production ready.

Last year, around this time, we put a magnifying lens on the top publicly available containers on Docker Hub and published our first public container report last year. We were surprised to find that even the most commonly used public containers — we’re talking containers with 5, 6 and 7 billion pulls on Docker Hub each! — have large numbers of vulnerabilities.

We’ve now published a sequel to that report — the second annual Slim.AI Public Container Report (available for complimentary download here) — and in it, we’ve explored the delta on that data set from 2021 to today.

Finding #1: We saw more critical vulnerabilities than ever across all categories.

I already mentioned that 60% of top public containers have more vulnerabilities today than they did one year ago. Yes, we saw certain incidents getting resolved, but new incidents are detected three times faster than our “remediation rate.” Most notably, the issues we resolved were mostly negligible, low-severity vulnerabilities, whereas the new ones we found are mostly critical and high-severity: high-severity ones increased by 50%, followed by a 10% increase in critical vulnerabilities. Today the average public container has 287 vulnerabilities, 30% of which belong to a high/critical category, up from 20% last year. (And we thought 20% was way too high!)

Finding #2: Component complexity has also risen significantly in the last year.

We detected 13% more packages on average per container. The average container now has 387 packages, almost 400. Given how each package may have hundreds of thousands of dependencies, as shown by multiple academic studies, this number is supposed to be the tip of the iceberg. And it’s not just the package counts that are worrisome. We saw two and a half times more licenses and four times more layers on average. Scanning them using open source scanning tools takes almost two times longer, resulting in wasted time in our CI/CD systems.

Don’t get me wrong — a lot of these components are necessary for experimentation. These tools and packages can help developers build, debug, and test their applications. But they also represent complexity and attack surface. If we don’t make a conscious effort to remove unnecessary components and optimize containers prior to shipping them to production, we’re incurring massive technical debt that will need to be addressed down the road. This obviously implies the need for automated processes to ensure that bloated attack surfaces never make it to production.

Finding #3: There is a disconnect between executives and front-line engineers.

Forty-nine percent of executives in our survey think containers are slimmed and hardened, but those who do the actual work, the front-line engineers and managers, report significantly lower numbers. As mentioned above, our survey found that 88% of developers admit it is challenging to remove vulnerabilities. Moreover, less than 26% say they understand how to slim and harden containers.

Today, many companies and governments are demanding a world with zero vulnerabilities, but our research reveals just how out of reach that goal is given current tools and techniques.

Complexity Is Not the Enemy; Ignorance Is

Here’s the sobering bottom line: We are no more secure today than we were this time in 2021. Securing containers for production is not getting easier, yet customers are reacting to security breaches by demanding zero-vulnerability supply chains. As a result, developers, DevOps, and DevSecOps teams are feeling the pinch.

The silver lining, however, is that we have woken up, and we are more aware of these issues than ever before. Across the world, there are competent, relentless, brilliant teams losing sleep thinking about these problems, and so I’m confident that when I share our container analysis results in the future, I will be bringing much better news.