Linux xz and the Great Flaws in Open Source

SEATTLE —  The Linux xz utils backdoor exploit exposes flaws in the open source ecosystem and how it works. That’s one takeaway from this On the Road interview on The New Stack Makers with John Kjell, director of open source at TestifySec.

The interview, hosted by tech evangelist Chris Pirillo, was recorded at Open Source Summit North America.

A maintainer for the Linux xz utils compromised the project by adding malicious code to a new release. Discovered earlier this spring, the orchestrated exploit illuminates how the conditions in the open source community allowed a maintainer, often the most trusted person on an open source project, to betray his colleagues most fundamentally.

The maintainer placed the backdoor malware in the upstream xz repository’s tarballs. A Microsoft engineer discovered the malware when determining why a download of xz utils using SSH took way too long. He looked into it a bit and found the malicious code.

Talk about betrayal. 

“This library is used with openSSH and so part of the code inside of it allowed backdoor to be inserted so that whenever someone would do an SSH connection, they could basically get in,” Kjell said. “They could either remotely execute code on that server, or they could directly log into that server.”

One Takeaway: Open Source Is Fragile

The most extraordinary fragility in open source is the people who face a weight of pressure to fix CVEs and make updates. Economic forces come into play. Company leadership requires updates to make new features available in their products.

The xz attackers knew of this pressure and used social engineering to get one of their members close to the project and eventually get accepted as a maintainer. The rest is pretty ugly.

Red Hat gave the vulnerability its highest threat rating. The backdoor can be anywhere. Why? It’s used with SSH and OpenSSH. So when the maintainer added the code, it spread far and wide in Debian, Ubuntu, etc. 

How this crew (nation-state actors?) got into the community shows our fragility and resiliency. Linus Torvalds said at the Open Source Summit that a community member found the obfuscated code soon after it was planted.

Still, the bad actors established trust, knowing that they could put pressure on the maintainer. They could be aggressive but also sympathetic by volunteering to help.

“They started just showing up and contributing to different open source projects,” Kjell said. “And one of the projects they contributed to was actually utils. And over time, they built up more trust, they, you know, did more commits, then, you know, the, the social engineering aspects started kind of cropping up.”

The economics of open source depend on contributions. Maintainers love their work and take pride in what they do. They all have their vulnerabilities, which opens them to predators who exploit the conditions that the maintainer faces. 

“This isn’t most people’s day job, they don’t get paid to do this” Kjell said. “They get badgered by these large upstream distros, companies selling these things, to have more releases to fix problems.

“And so, other accounts started badgering the sole maintainer of this project, saying you need to do more, you need to do it faster, you need to address all these concerns. And then this person walks in and says, ‘I’m happy to help.’ They had built up enough credibility beforehand to be introduced, and over time, that led to being able to handle sensitive bug reports and create their own releases. And that was how we ended up in the situation where we had the backdoor put in.”

A deep need for open source is as much an economic issue as a technical one. Organizations need open source software that manages every aspect of their business. They depend on maintainers but also put considerable pressure on them to keep the software patched and updated.

These economic conditions open the door, so to speak, exposing fragilities in the open source supply chain. Thank goodness people are also so resilient.