Solo.io Brings ‘Docker-Like Experience’ to eBPF with BumbleBee

Service mesh integration software provider Solo.io has released BumbleBee, a new open source project that it says “brings a Docker-like experience for eBPF to you.”

In their efforts to improve performance in certain realms such as metrics, Solo.io had turned to the extended Berkeley Packet Filter (eBPF) in order to “shortcut the HTTP stack,” said Solo.io CEO and founder Idit Levine. eBPF gives Linux users a way to run sandboxed programs within the kernel space, without changing kernel source code or loading modules. The act of writing an eBPF application, however, requires multiple tasks, and the Solo.io team found themselves spending time having to recompile the programs for each environment they ran in.

Portability Problem

Writing an eBPF program, explained Levine, involves multiple tasks, from writing the program itself, to writing the user-mode program that interacts with it, to compiling the C source to the eBPF byte code for that particular kernel, which leads to a portability problem.

“As the Linux kernel is not backward compatible, eBPF byte code that was compiled for one kernel cannot be used in a different version of the kernel. For example, if an eBPF program reads a field in a struct in the kernel in one version, and in a newer kernel the field order of that struct has changed, the program will now read the wrong field,” explained Levine in an email.

For a while, the solution was to distribute eBPF programs in source code form, for compiling on the machine it was to be run on. This also meant that the program’s dependencies would need to come with it. A recent change, however, made it possible to fix this portability problem in much the same way that Docker helps developers with application portability.

The addition of the BPF Type Format (BTF), explained Levine, “(along with some smarts added to clang) enables the BPF program loader to fix the BPF byte code to work correctly on different versions of the kernel. For example, if a BPF program accesses a struct, clang now stores all these struct access in a special location in the BPF program binary. libbpf can go to each of these struct accesses, and use BTF information from the current kernel (obtained at runtime) to fix these accesses to the correct offset.”

BumbleBee to the Rescue

With the addition of BTF, Solo.io created BumbleBee, which not only uses BTF to parse and bring to the user space the maps of eBPF programs, but also uses the Open Container Initiative (OCI) image specification for the eBPF applications, which means they can be stored and distributed more easily, and integrated into existing workflows.

“Now you can basically build it everywhere and run it everywhere, which gives you the ability to distribute it exactly like a Docker container,” said Levine. “It’s giving you the ability to basically, in one command line, take something that was on the registry in the cloud, and run it on your kernel, which is pretty insane.”

In addition to simplifying the distribution of eBPF programs, BumbleBee also automates the boilerplate, including the userspace code, leaving its users to just write the eBPF code itself.

In all, Levine said that she saw the effects of BumbleBee immediately in Solo’s use of the project.

“I think that what we saw internally in the company was a huge velocity gain. I feel that it’s not going to be only my employees, but it will be way more of the community. It’s exactly the effect that Docker has potentially, which is the ability to have the community writing eBPF and other people running it. I think that this is huge,” said Levine.

BumbleBee itself is a command-line tool that requires a Linux kernel version 5.4 or newer and knowledge of just a few commands — bee init, build, run, list, and push — to get started.