eBPF Has a Bright Future in Infrastructure Development

For the next decade, eBPF will be “a strategic platform choice for infrastructure developers,” predicted Cisco Distinguished Software Engineer Daniel Borkmann, in a keynote talk at the Linux Open Source Summit about the state of the technology he helped create, including some prognostication of how it could be used in future times.

eBPF is revolutionary because it shortens development cycles of getting new technologies into the kernel, Borkmann posited. Until now, getting some sort of extension added to the Linux kernel has taken years, with scant assurance of success. Now, eBPF allows users to add their own kernel technologies.

“We are going from a long innovation cycle to a short innovation cycle,” said Borkmann, who has also been a 15-year Linux core committer. And more providers of infrastructure tools are taking advantage of the in-kernel technology.

eBPF is good at fine-tuning performance across the stack, from networking to the CPUs themselves, providing a better user experience. It’s also good at implementing policies.

Meta’s Facebook uses eBPF extensively in its Layer 4 load balancer, with 100 different eBPF applications doing various tasks. It is used in all Android phones for security policies and traffic control and for bridging to IPv6 networks.

eBPF for Virtual Machines

In the realm of networking, the eBPF-based Netkit driver has proved popular for identifying namespaces within the kernel itself, formerly the task of the much slower network drivers.

This work actually began in the Kubernetes community, as a way to minimize container communication latency, Borkmann said. Borkmann participated in the Cilium project, a network packet processor built on eBPF.

“As soon as we landed this in the Linux kernel, the hyperscalers also got interested, and they rolled this out into their fleet,” Borkmann said. For instance, Meta managed to shave its P99 latency for namespace resolution from 12 seconds down to 100 milliseconds using Netkit.

The next step will be to bring eBPF to the world of virtual machines.

Many organizations now have two platforms, one based on Kubernetes and an older legacy-based one based on virtual machines (VMs). Each has its own monitoring, metrics and logging.

eBPF could help consolidate these two worlds, Borkmann said. Currently, work is underway to integrate NetKit into KubeVirt, which allows users to manage virtual machines as Kubernetes pods.

👁 Image

Netkit could optimize the network throughout.

eBPF for the Kernel

You can think of eBPF as a “universal assembly language,” Borkmann said, though it is one most developers will never see, opting for an interface to their programming language of choice.

Two major just-in-time compilers (LLVM and GCC) support a variety of front-end languages (C, Rust, etc), which they convert into eBPF bytecode. The instruction set has been standardized by the IETF Working Group (allowing for a version of eBPF for Windows). There’s a verifier to ensure user code doesn’t destabilize the system.

Another use case for eBPF has been to optimize the kernel itself.

Process scheduling is another emerging use case that benefits from a new Linux kernel utility, sched-ext, which enables the creation of an eBPF-based kernel thread scheduler.

Hyperscalers such as Meta and Google are already using eBPF-based schedulers. Fine-grained scheduling policies have optimized these companies’ workloads by 10% or so.

SteamOS also uses an eBPF scheduling approach to optimize gaming.

eBPF could help in AI workload balancing: What part of the AI stack is chewing up resources? An eBPF-based flamegraph would show the full call graph. The longer the bar, the more cycles are being chewed up.

Finally, Borkmann suggested that eBPF could set the stage for live patching the kernel, where it would provide safety checks for the patches being applied.

👁 Image

Disclosure: The Linux Foundation paid for the reporter’s trip to the Open Source Summit.