Log4j Scanner Blindspots

Thanks to the Apache Java logging library log4j‘s popularity and its ability to hide in code, we have landmines hiding in our infrastructure due to log4j’s Log4Shell security vulnerabilities. A day, an hour, doesn’t go by without new exploits blowing up. The good news is there are scanning tools that can find those vulnerable libraries before they go bang. The bad news is, Rezilion, a programming security company, has found that no single scanning tool can find all the security-compromised programs.

The problem with detecting Log4Shell within packaged software in production environments is that Java code can be nested a few layers deep into other files. A simple shallow search for it can miss it entirely. Adding insult to injury, log4j can be packaged in many different formats such as Java Archive Files (JARs), Tape Archive (TAR), Web Application Archive (WAR), Enterprise Application Archive (EAR) Service Application Archive (SAR), and on and on. Salting the wound Java code can be buried many levels down in these formats. It’s a real nightmare digging up the suspect code.

Use Multiple Tools

So, as Rezilion Vulnerability Research Lead Yotam Perkal explained, “In order to estimate how big the industry’s current blindspot is Rezilion’s vulnerability research team conducted a survey where multiple open source and commercial scanning tools were assessed against a dataset of packaged Java files where Log4j was nested and packaged in various formats. All formats are commonly used by developers and IT teams. Some scanners did better than others, but none was able to detect all formats.”

Yes, you read that last line right. There is no single scanner that can spot examples of the problem code. Your only choice is to use multiple scanners.

While Rezilion didn’t look at all the scanners I’ve mentioned in How to Find Dangerous Log4j Libraries, it’s a safe bet that none of them can find all the troubled lib4j libraries either. Again, you’re going to need to use multiple scanners in your security-hole hunt.

For example, Rezilion found that while tools can detect vulnerable Log4j instances in multiple Java binaries types with a range of file extensions, sometimes the names are the ones we’re searching for. For instance, “in the ElasticSearch binary detection evaluation, the JAR name we examined was elastic-search-sqlcli-7.4.2.jar, a filename that does not meet the basic identification pattern used by many scanners to detect Log4j (e.g.: log4j-core-.jar), yet it still contained vulnerable pieces of Log4j code.”

In addition, Perkal continued, source-code analysis has its limitations. “While this approach will effectively detect nested and packaged instances of the vulnerable component, it can’t be applied to packaged third-party software, which still constitutes most of the code in production. Furthermore, it has major blind spots and false positives when it comes to transient dependencies.”

You also need, Rezilion points out, “code-level visibility in runtime memory where the code isn’t packaged or nested.” Besides, Perkal concluded, “detection abilities are only as good as your detection method. Scanners have blindspots. Security leaders cannot blindly assume that various open source or commercial-grade tools will be able to detect every edge case — and in the case of log4j, there are a lot of edge instances in many places.”

Indirect Dependencies

Besides that, log4j can be called by indirect or transitive dependencies that the direct dependencies are calling. Indeed, Rezilion has found the majority of affected artifacts stem from indirect, transitive dependencies. “That means that in most cases, Log4j is not explicitly defined as a dependency of the artifact, but gets pulled in as a transitive dependency. Furthermore, for more than 80% of the packages, the vulnerability is deeper than one level down, with a majority affected five levels down (for some packages even as many as nine levels down).” Finding those can be a real nightmare.

Oh yes, we’ve all been finding that out in the last week. The question isn’t, “Where are we calling log4j?” It’s “where are we Not using log4j?”

Rezilion recommends using multiple approaches to find and patch the vulnerable code. The company is also, of course, willing to run a free assessment to quickly detect if you have undetected instances of Log4j and determine whether or not these instances are exploitable and require immediate action. Given that we’re all going to need all the help we can, I’d seriously consider this offer.

Good luck. Now, if you’ll excuse me, I’m going back to scanning.