Zero-Day Vulnerabilities Can Teach Us About Supply-Chain Security

Was the Log4Shell vulnerability an anomaly or a precursor for the types of security incidents that organizations will face in the future?

While the worst predictions about CVE-2021-44228 didn’t come true, every organization has a software supply chain that can be rendered vulnerable. Embedding open source third-party libraries is common development practice because it greatly speeds things up, but each library can be a source of risk. Most development teams are willing to accept this risk because of the added benefits.

However, cybercriminals and nation-state actors know there is value in finding and exploiting vulnerabilities in open source code. As open source projects are terminally underfunded and understaffed, motivated attackers will continue to exploit these vulnerabilities. What lessons can we learn to manage and mitigate future threats?

One Year Later: 500,000+ Attacks Per Day

The Log4Shell vulnerability was one of the most severe and troublesome CVEs to be discovered in recent years. At the time of its disclosure, I could not have imagined a worse library to have a vulnerability, given that almost every enterprise Java developer uses the Log4j logging services framework.

More than a year later, many organizations have yet to implement a patch for this vulnerability on their systems for two common reasons:

• Businesses prioritize patching their external systems, but this makes up a small proportion of overall IT infrastructure. Many have yet to address their internal applications, which make up the majority of an organization’s infrastructure.
• Identifying or mitigating vulnerabilities in commercial off-the-shelf software takes time because you don’t have a deep understanding of what could be lurking within.

Despite being over a year old, the vulnerabilities associated with Log4j are an active threat. In fact, 12% of all CVE requests since December 2021 are related to Log4Shell, which equates to an average of 500,000 attack requests per day. About 7% of those malicious requests are successful, which underscores that this is still a viable threat for businesses that employ the logging framework in Java environments.

Attackers were able to quickly exploit the vulnerability because it was automated within days, and new permutations and encoding techniques were discovered, allowing for the bypass of weak defenses. It’s commonly exploited by bots and the Chrome browser, although requests also come from cURL, PhantomJS, Nessus Cloud, the Go HTTP library and Node.js.

Additionally, several vulnerability scanners are able to scan for vulnerable Log4j applications and make up a portion of the traffic we see scanning the internet for the Log4Shell vulnerability. While these vulnerability scanners are typically used with non-malicious intent, they can sometimes be used by attackers probing for vulnerabilities.

Based on an analysis of the payloads used to exploit the vulnerability, Imperva Threat Research generally categorizes them as:

• Probing: Attackers will often probe the application before sending the actual payload and will check if the application is vulnerable.
• Reverse Shell: This payload will open a communication channel between the vulnerable application and the hacker. After the hacker receives the communication, they can further explore the target system and remotely run any shell commands.
• Malware deployment: Attackers will attempt to deploy malware on vulnerable systems.
• Data exfiltration: Payloads attempt to exfiltrate information.
• Patching: Interestingly, Log4Shell can be used to deploy Java code that changes the configuration and doesn’t allow lookups. In other words, you can patch the Log4Shell vulnerability with a Log4Shell payload.

Buying Time: How Security Teams Can Aid Developers and Minimize the Impact 

Unfortunately, supply-chain risk is inevitable and zero-day incidents like Log4j will happen again. When a vulnerability is announced, it takes time for development teams to write, test and validate whether a fix will patch the issue without breaking existing functionality. To minimize the impact of these incidents and buy developers time to patch their systems, there are several controls security teams should implement:

• Deploy a web application and API protection (WAAP) solution: A WAAP solution is an effective, non-invasive security defense to protect production workloads. They provide an effective early level of mitigation and can inspect all content, payloads and requests for malicious content. What’s more, if you’re writing and building APIs with gRPC or GraphQL, a WAAP offering can mitigate automated threats like bad bots, distributed denial of service (DDoS) or API abuse — unlike a web application firewall, which is often blind to gRPC or GraphQL traffic.
• Invest in Runtime Application Self-Protection (RASP): Aside from being the only solution recognized by NIST (SP 800-53 Revision 5 | SI-7 (17)) as a recommended control to stop software supply chain attacks, RASP integrates security into DevOps via continuous integration, continuous delivery (CI/CD) processes, seamlessly becoming part of an organization’s secure software development life cycle (SSDLC). From attacks to injections or even weaknesses, RASP is capable of detecting and preventing attacks in real time. The technology can pinpoint attacks down to an exact line of code and automatically stop the exploitation of a vulnerability, which gives developers the time they need to patch vulnerabilities on their own schedule.
• Adopt a Micro-Segmentation Approach: Micro-segmentation divides networks into separate security segments at the level of specific workloads. This makes it possible to define security controls and restrict access to each segment, preventing attackers who compromise one segment from being able to perform lateral movement or privilege escalation to other segments.

Managing Supply-Chain Risk Starts with Secure Development

By implementing security controls, developers are permitted the time they need to patch a vulnerability. In theory, this process sounds straightforward, but that’s not always the case. There are two common challenges I hear from customers’ development teams when working in a triage situation:

• First-party software takes time to patch. Writing, testing and validating whether a fix will resolve a vulnerability can take trial and error. By definition, zero-days don’t have a patch, meaning it can often be days before developers can even begin the process of patching their applications. Furthermore, software needs to go through QA cycles before a true fix is identified. This is why security controls are necessary for blocking malicious activity before it reaches runtime. Additionally, developers must analyze their software development life cycle (SDLC) and augment it before a vulnerability is announced. An asset or application inventory should be a mandatory component so that when a vulnerability is disclosed, organizations know who owns the application and who to contact. Finally, developers should be mindful not just of the security threat of direct dependencies, but also of transitive dependencies, which are dependencies of dependencies, as these can also be vulnerable to exploitation.
• Commercial off-the-shelf software is a quagmire for security teams. Securing third-party or commercial-off-the-shelf software is one of the biggest cybersecurity challenges facing every organization. Unfortunately, most vendors don’t disclose the components and libraries that make up their software, making it difficult for organizations to know whether a vulnerability affects them once it’s disclosed. Historically, companies have focused on the risk introduced by their immediate set of vendors and the critical software they rely on. Yet, that posture is no longer sufficient.

Developers must come together to advocate for the adoption of several software standards such as the software bill of materials and Vulnerability-Exploitability eXchange. Additionally, industry-driven efforts like the OpenSSF Foundation are providing fundamental guidance for developers to ensure secure development becomes a habit, especially at a time when operational defenses are not yet available to fully address this challenge.

Conclusion

Thankfully, the public campaign to create awareness about the Log4Shell vulnerability encouraged organizations to patch systems and deploy preventative mechanisms. This mitigated the potential for a much larger, more catastrophic impact, including large-scale security incidents and data breaches.

The response to Log4j is a great example of how the private and public sectors can work collaboratively to address high-priority issues. What’s more, we see that security and development teams can work in concert to quickly respond and address risks.

We can credit the Log4j response, in part, to the preparation efforts many organizations made after the Apache Struts zero-day in 2017, the last large-scale software supply chain vulnerability. This experience highlighted the need for organizations to include software bills of materials (SBOMs) into their development processes, enabling them to know what and where dependencies are in the software supply chain.

Ultimately, Log4j was a zero-day. While there’s a big effort to get patched quickly once a zero-day is disclosed, it’s imperative that you have other controls in place to prevent software supply chain attacks before they’re even identified.