Are You Overspending on CVE Management?

Asking cloud native developers to “build now” has meant punting vulnerabilities (CVEs) in your cloud native supply chain to “later.” The new CVE Zero movement is rethinking how to approach container security debt.

Modern companies are addicted to the fast delivery of new features. This is why delivering software on a modern software team often feels like feeding fish to a frenzied group of sharks. It’s also why accepting all types of technical debt is so easy for software teams. Why not just feed the sharks today? They want it today, and they’re willing to pay.

Cloud native companies tend to feel this trade-off particularly acutely. Software developers choose base container images and other third-party images, and add custom code and configuration on top. There might be problems lurking inside those third-party container images, but the tests pass and the customers are happy.

It’s only later that issues emerge, when cloud native software teams discover tens of thousands of unique known CVEs in the fleet of containers powering their revenue-generating applications.

What’s the Real Cost of ‘Paying Later’ on CVEs?

It is increasingly accepted that most popular containers, the fundamental building blocks of cloud native applications, are riddled with CVEs. Industry analysis suggests that even the latest versions of popular containers have hundreds of CVEs. Research from the Ponemon Institute and other organizations is digging into how teams are using emerging DevSecOps patterns for managing and addressing vulnerabilities.

But all that great research doesn’t answer a fundamental question: How much does the “pay later” approach to accepting container vulnerabilities as a form of security debt cost? What is the actual penalty of identifying, triaging and remediating CVEs?

To give the security community more insight on the cost of CVEs, Chainguard Labs conducted deep interviews with software professionals who handle vulnerability management as part of their day-to-day duties at cloud native software companies. Our investigation dug deeper into all of the tasks and workflows associated with CVE management to estimate the annual time cost of CVE management.

CVE Management Costs Thousands of Hours Each Year

The findings were frightening, especially from the perspective of software leaders who need to maintain feature velocity to compete.

We learned that most cloud native software companies are spending thousands of hours each year on vulnerability management. For some companies, that’s 10,000 hours per year — or more. That’s an entire engineering squad doing nothing but identifying, triaging and remediating vulnerabilities.

The following table reveals this daunting reality. While the average appears to be around 1,000 hours per year, some organizations are spending 10 times more. This means that whole teams of software and security engineers are managing CVEs rather than shipping new software.

Teams rationalize deferring paying CVE debt in different ways.

One major factor is software consumers are voracious, demanding new features built rapidly. This means software engineers with tight timelines are begrudgingly accepting the cloud native default — containers with CVEs. If the functionality works, scanning for CVEs (much less fixing them) is an afterthought.

Another key factor is the software application developers who usually select a container image — often through making a few edits to a Dockerfile — are often not the ones bearing the downstream costs of vulnerability management.

Finally, creating software that is easy to update is difficult. While it’s at the core of the DevOps philosophy, it’s hard to do in practice. Changing a piece of software, even to fix a CVE, often risks product downtime and frustrated customers. Consequently, many software organizations find it painful to make even minor changes to their software. This means fixing a CVE can be deemed not worth the risk.

You Can’t Know When CVE Security Debt Must Be Paid

Every organization faces different timetables on when their CVE-driven security debts demand payment.

For some, the debt must be paid for compliance reasons. For instance, cloud native companies seeking to tap federal markets often run into compliance frameworks that require onerous amounts of paperwork when CVEs are present in containers (think FedRAMP).

For the particularly unfortunate, the debt comes due all at once as a consequence of hackers exploiting a CVE to access a system. That cost may be millions of dollars in reputational loss, lawsuits and ransomware. CVE-2017-5638, a vulnerability in Apache Struts, allowed remote code execution via HTTP headers. It produced one of largest data breaches in history, affecting almost 150 million U.S. citizens and 15 million U.K. citizens.

CVE-2021-44228, better known as Log4Shell, is another example of a CVE requiring an all-at-once security debt payment. As the undisputed heavyweight of all CVEs, Log4j is still so widespread that nearly all large organizations are running it, and many are still trying to find all the instances.

CVE Zero Is about Building Securely from the Start

Microservices and containers rewrote the rules for how developers get software components. Gone are the days of top-down approaches where technical leadership decides on Linux distributions, operating systems, application infrastructure and security service-level agreements (SLAs). Today, developers are doing this all in Dockerfiles and GitHub Actions.

But there are massive trust gaps between distributions and software packages. There are whole classes of vulnerabilities that live outside distributions and do not get picked up by security scanners. Meanwhile, most distributions and software packages ship with many known CVEs that create a ton of “false positive” noise from security scanners. The consequence of paying CVE security debt later is it’s very hard to get a clear picture of what CVEs are addressable in your environment. There is also a ton of thrashing involved to estimate time to triage an individual CVE, with so many variables between a simple batch version upgrade and migrating a codebase to a major new version of a dependency.

Paying vulnerability-driven security debt down whenever that bill comes due is an engineering management nightmare — lumpy and unpredictable work.

CVE Zero seeks to control the binary, the container image itself. It’s about shifting software building blocks to a zero-vulnerability approach. It’s about weeding out the bloat of unnecessary components from base images so the signals from security scanners become more reliable.