Linux Technology for the New Year: eBPF

In the year to come, we will start to see a change in the Linux kernel architecture, as a new component, eBPF (Extended Berkeley Packet Filter), starts taking over more monitoring, security and networking duties from individual kernel modules.

eBPF is “Linux’s newest superpower,” said SAP Labs’ developer Gaurav Gupta, during a talk that he gave about using the technology for low-overhead tracing at KubeCon in Copenhagen earlier this year.

A virtual machine for the Linux kernel, eBPF could set the stage for advanced, low-overhead tracing inside the kernel itself, offering insight into I/O and file system latency, CPU usage by process, stack tracing and other metrics useful for debugging. It could also play a role in system security, potentially offering a way to thwart DDOS attacks, to monitor for intrusion detection, and even replace IPtables as a way to enforce firewalls. It also offers a cleaner alternative to installing drivers.

“In the future, you will see a lot more eBPF programs instead of kernel modules,” said Netflix Kernel and Performance Engineer Brendan Gregg, at the All Things Open conference held in Raleigh, North Carolina in October. “You’ll start to see a lot more surprising stuff written in eBPF.” It is a step towards moving Linux to the microkernel model, where more functionality is defined and run in the user-space, rather than kernel space.

As organizations make their way to the latest Linux kernels they can enjoy these new benefits. You need at least Linux kernel version 4.4 or, preferably, 4.9 to run eBPF.

Originally called the Berkeley Packet Filter, the technology originally served as an HTTP packet filter for the Berkeley Software Distribution (BSD). Plumgrid extended its use for software-defined networking, providing a way for the Linux kernel to execute customized operations on the user’s behalf.

This has typically been a job for modules, though their use requires recompilation of the kernel upon installation, and they can also crash the kernel as well. Far beyond just redirecting packets, eBPF can attach to any kernel event, or any socket. It can run simple programs. It is basically a sandbox for the kernel, one providing built-in security. eBPF programs are compiled to bytecode, so that the developer doesn’t have to worry about different underlying architectures.

Writing raw eBPF code is difficult, Gregg admitted. It resembles an assembly programming language with no looping and other limitations. Fortunately, a number of packages can writing programs easier. With the LVMM, developers can create eBPF programs in C. The BPF Compiler Collection (BCC) toolkit offers a Python and Lua environments, which can be good for building out complex programs.

Those looking to capture metrics using eBPF should take a look at a project started by IOVisor Project, called bpftrace, a high-level tracing language for eBPF. “It boils down the program to just the probes and the logic,” Gregg said, noting that bfptrace offers the same low-overhead, dynamic tracing as Dtrace did for Solaris. Linux has had no shortage of tracing tools, though “getting them merged into the kernel has been a challenge,” Gregg admitted, referring to KTap and SystemTap.

Gregg envisions more engineers building out new GUIs built on the base tools.

In a 2017 infrastructure software conference, WeaveWorks’ Alfonso Acosta demonstrated how eBPF could be used to monitor all the traffic going through an NGINX server, without adding any instrumentation to NGINX itself.