How BumbleBee Eases eBPF Observability with OCI

An open source tool, BumbleBee extends eBPF’s reach and enhances its adaptability to operate in various environments, thanks to use of the Open Container Initiative (OCI) image specification.

In essence, BumbleBee simplifies the process of building and running distributed eBPF programs, encapsulating them as OCI images, then publishing them into an OCI-compliant registry.

It also allows you to expose events as metrics, making it easier to hook up and visualize your metrics alongside other observability tools.

Lin Sun, senior director of open source and a member of the Istio technical oversight committee, headed a workshop at Open Source Summit in Bilbao last September to show how the technology could be used.

Sun demonstrated that BumbleBee uses BPF (Berkeley Packet Filter), a kernel-space code and transforms existing code into metrics.

With BumbleBee, you have the ability to make it easier to deploy runtimes with eBPF, including on multiple kernels by executing distributed eBPF programs using OCI images. Specifically, during the hands-on workshop, participants were able to use BumbleBee to port eBPF kernel space code into metrics and to perform other tasks.

Like Docker, But for eBPF

BumbleBee is designed to offer the Docker experience for eBPF developers, Sun said.

With BumbleBee, users can build eBPF programs as OCI images, publish your eBPF programs as the OCI images into any OCI-compliant registries. You can also run your eBPF programs as OCI images.

“That’s exactly the favorite Docker experience you are familiar with: You build your images, you push them to the registry, and then you can run your Docker images,” Sun said. So, that’s why we believe that BumbleBee really provides the eBPF community this Docker-like experience, allowing you to learn, build and publish for your friends, families and coworkers to consume your eBPF programs.

BumbleBee uses the libBPF kernel space code, and takes existing eBPF kernel space code into metrics, as mentioned above.

👁 Image

Write Once Run Everywhere

BumbleBee makes use of the compile once – run everywhere (CO-RE) framework, so you have to run BumbleBee on newer Linux kernels that support CO-RE. You can also migrate some of the existing eBPF programs originally written to work with bcc-to-libBPF tools associated with older kernels to work with BumbleBee (Linux 5.8 and newer version of Linux are the requirements to use ring buffer).

👁 Image

Here is what was covered at the workshop: Firstly, an application was built and deployed with BumbleBee. This involved creating the application using BumbleBee and pushing it to the OCI registry.

Additionally, attendees were able to witness, through the Prometheus user interface (UI), the kernel metrics that BumbleBee collected. Workshop participants also saw how to bundle multiple kernel-space code components and package them as a single OCI image.

The project for the first workshop — the only one covered here — was to detect out-of-memory “oomkills” programs that terminated programs that overextended their memory allotments. In this program, with bcc, you typically have the user space code and the kernel space code so that there are two files per tool, such as:

oomkill.c
oomkill.bpf.c

The program was originally written to use HashMap, which does not perform as well as ring buffer. Sun explained the necessity of migrating HashMap to ring buffer. During the workshop, the program was also migrated from the perf buffer to the ring buffer. This is because ring buffer allows for better use of resources on a CPU level, Sun said.

OOM was first used during the early days of Linux to detect when application memory is running low. For Kubernetes, OOM is run to make sure code running on clusters is not running out of memory and if they are, older instances replace the current ones running in the cluster. However, tests are run to determine the viability of the swap.

A pod usually gets “oomkilled” if it does not have the requisite memory to run the code or if it is over committed. Again, eBPF was used in this workshop to detect oomkills.

👁 Image

With the “Bee Build” command, I first successfully compiled “oomkill.c” and then wrote it to “oomkill.o” and saved the eBPF OCI image to the local host:

👁 Image

Oomkill events revealed with BumbleBee:

👁 Image

Prometheus offers its observability info for the oomkills detected with eBPF:

👁 Image

Conclusion

Downloading BumbleBee from GitHub and installing it locally can pose more challenges than simply running it pre-installed from a virtual machine on the cloud. However, I assume it’s very doable.

Once everything is installed, getting it to run proved relatively seamless. I was able to successfully run BumbleBee, which is used to take eBPF code and push it to an OCI registry, providing observability data through monitoring.

This included “oomkill” data, and it was followed up with Prometheus as well. All of this was accomplished smoothly.

Furthermore, eBPF extends to various areas and encompasses a wide range of growing use cases, including observability and security. Since tool makers are building on eBPF,  using observability data to detect security vulnerabilities, offer ways to mitigate and resolve these vulnerabilities, and even address attacks in some cases, eBPF is becoming increasingly worthwhile for adoption.