Strobelight: Meta’s eBPF Profiler Framework for Massive Infra

Imagine, if you will, that you’re in charge of monitoring all of Meta’s massive infrastructure. It’s a scary thought isn’t it? Now, in a groundbreaking development, Meta’s engineering team has successfully leveraged eBPF technology to enhance their fleetwide profiler, Strobelight.

When I say “enhance,” I don’t mean some minute improvement that only a dedicated performance engineer could love. No, I mean a “20% reduction in CPU cycles, equating to a 10-20% reduction in the number of required servers for Meta’s top services.” That’s a serious savings in compute, which equals a serious saving in money.

Strobelight, Meta’s fleetwide profiler framework, is designed to provide comprehensive profiling capabilities across the company’s large-scale infrastructure. It’s made up of multiple subprofilers that collect various types of performance data, including CPU, GPU, and memory profiles. The framework’s number one job is to identify performance bottlenecks and optimize resource utilization across Meta’s fleet of machines.

The key to Strobelight’s recent success lies in integrating eBPF. It enables efficient, low-overhead monitoring and system event tracing directly within the Linux kernel. By leveraging eBPF, Strobelight can now collect performance data with minimal impact on system resources.

To be exact, according to the eBPF Foundation, eBPG enables Meta to track CPU time spent in function calls and execution paths; call stacks for native and non-native languages (e.g., Python, Java, and Erlang); off-CPU time and service request latency analysis; and AI/GPU profiling and memory tracking.

eBPF for the Savings

Besides, saving compute time and cash, the use of eBPF in Strobelight has led to, the eBPF Foundation claims, 15,000 servers’ worth of annual capacity savings from a single one-character code change. Color me impressed. It’s also enables faster debugging and performance analysis. This allows engineers to prevent regressions before they reach production.

With eBPF, Strobelight can now track GPU memory allocations and detect memory leaks more efficiently. According to Riham Selim, a Meta software engineer, eBPG enables at any point of time how memory allocation is happening for each GPU,

Mind you, eBPF isn’t perfect. Selim noted. It lacks visibility into GPU internals; the sheer volume of data can be overwhelming;  and it lacks application-specific understanding. So, for example, you’ll need to add observability code to a PyTorch program rather than relying on eBPF alone.

So, it’s important to understand that Strobelight is far more than just eBPF. According to Meta, “Strobelight is … not a single profiler but an orchestrator of many different profilers (even ad-hoc ones) that runs on all production hosts at Meta, collecting detailed information about CPU usage, memory allocations, and other performance metrics from running processes.”

Indeed, all together Strobe Light has 42 different profilers. Most, but not all of these, are based on eBPF.

Safe Injection of Custom Code

So it is that, eBPF is vital for Strobelight. The Meta engineers note, “eBPF allows the safe injection of custom code into the kernel, which enables very low overhead collection of different types of data and unlocks so many possibilities in the observability space that it’s hard to imagine how Strobe Light would work without it.”

Don’t ask me, how you could do it. I know a thing or two about observability and, without eBPF. I wouldn’t even know where to start. Fortunately, since we have eBPF, we needn’t worry about this.

Want to try it for yourself? You can. Most of StrobeLight was recently open sourced under the Apache 2 License. However, Meta has yet to open source Strobelight’s profilers and libraries. The company promises it will do this since by opening them up they’ll become “more robust and useful.” Be that as it may, there’s already enough open here to make Strobelight worth exploring for anyone that wants to keep a weather eye on massive infrastructure systems.