Where Does the Time Go? Rust’s Problem with Slow Compiles

While many are sold on the magic of the Rust programming language, one persistent complaint keeps popping up: the long time it can take to compile Rust programs.

“I really feel like the Rust community suffers from a collective Stockholm Syndrome when it comes to compile times. They are bad. I know it’s hard work to fix this, but the compiler is very far off of what good productivity demands,” wrote Armin Ronacher, creator of Python Flask, on X (the platform formerly known as Twitter). His criticism of Rust’s slow build times has been one of many.

On Monday, Oxide held a virtual meetup on Discord to discuss the issue.

The Zen of Compile Time

Oxide CTO Bryan Cantrill is no stranger to long compile times.

He recalled at one point in his time at Sun Microsystems that compiling an OS kernel would take more than 24 hours to complete. That was a bit outrageous, he admits satisfaction from a lengthy build, appreciating that “it’s taking a long time to compile because it’s solving really hard problems for me.”

But he acknowledges there is also an inherent frustration with long build times, a sinking feeling that this is time better spent elsewhere, both for man and machine.

“One of the things that’s most frustrating is when you, the programmer, really feel like Rust is doing work it doesn’t need to do,” Cantrill said.

So the company set its engineers on the task of trying to figure out why Rust apps took so long. As this online discussion revealed, it turned out to be quite the side quest…

“Why is my compile taking so long?!” If you are a Rust programmer, you have asked yourself this question many times. Tomorrow, @ahl and I will be joined by our colleagues Sean Klein and Rain Paharia to dig into Rust build times. Join us, 5p Pacific! https://t.co/X7p4dlD5IU

— Bryan Cantrill (@bcantrill) January 21, 2024

How Oxide Uses Rust

A hardware company specializing in gear for on-premises clouds, Oxide used Rust to build its control plane (“Omicron“) as well as other infrastructure software (everyone at Oxide seems to love the advanced capability of Rusts packaging system, Cargo).

These projects started small but as they grew larger, their compile times slowed. This could be particularly frustrating when making a single change and looking to see the results, which involves lots of repeated builds, Cantrill said.

Worse, as the compile times grew, it made became increasingly more difficult which parts of the build were gobbling up all the time. Build times could not be shortened if there were no tools to reveal why they were taking so long to build in the first place.

The Rust compiler was written as a classic batch compiler (though it has been expanded with some incremental capabilities) meaning you have to sit through the compile time of the entire app, even if you only made an incremental change. Hit cargo-build, and the resulting build time could go for 48 seconds or three minutes or longer. But the programmer will only stay for so long before doing something else entirely, breaking up their flow, Cantrill observed.

How Rust Compiles Code

cargo build –timings has a flag that, when called, provides a build graph of, crate by crate, what is being built, and how long it takes to build each one.

What is happening inside these crates remains a bit of a mystery though, and so can be a challenge to optimize for, said Oxide software engineer Sean Klein, “There is not one answer for where you go next. There’s a lot of different answers,” he said.

Tools like cargo-llvm-lines, or cargo-bloat can show why the resulting binaries are the size they are, but these measurements are proxies for estimating compile times themselves.

One obvious issue in Monomorphization, which is either a feature or a bug of Rust, depending on how you look at it. If a small generic function is used in multiple places across an application, Rust will compile that generic for each specific case. Result: Fast programs, slow compile times.

Oxide engineer Steve Klabnik created a 10-line function that eliminates all these repeat builds of generics. “By making this small change, you can help the compiler not do as much work,” the pull request documentation reads. It shaved five seconds from the build time of Omicron.

Also working on this issue is Oxide engineer Rain Paharia, who has spent a lot of time looking at the issue, both at Oxide and at their previous role at Meta.

When a crate is rebuilt, then all of its dependents are also rebuilt transitively, they said.

For instance, the widely used syn-crate, a parsing library has about 15 features, any number of which may be called by a program’s Procedural Macros (a handy to feature used to extend program code at compile time).

At compile time all those features get rebuilt, regardless of which ones are actually called…

“All of a sudden, you have this combinatorial explosion of feature or feature sets that can be built. And because syn is so core, every single thing that depends on syn, also gets rebuilt,” Paharia said. “It is kind of a disaster scenario at the moment. This is not good at all.”

Problem Solved?

Paharia built some tooling that works around the problem, such as cargo-hakari, a command-line application that uses a blank crate to speed compilations by 20-25%. The crate specifies a union of all the features used everywhere in the program, so they are compiled only once, instead of multiple times.

They found that cargo-hakari cut build times somewhat … but not entirely. So Paharia dug in with a new (unstable) Rust feature called the unit graph, which specifies each atomic step in the build process, building off the dependency graph for the app.

Pouring over the output, Paharia found that, despite their preventive measures, the compiler was still rebuilding many duplicate objects, due to a variety of obscure behaviors, such as how Rust handles panics for plug-ins, and how Rust handles Proc Macros in general.

Also, importing non-Rust code, through Build Scripts, can consume hella resources. The Oxide team also seems to collectively love Build Scripts, even as they suffer through the performance hits they incur.

Work Still to be Done on Rust Compile Times

Rust does have a “self-profiling” flag that will let you know how long it take to build a particular macro. It provides, as JSON, a timeline of how long each action took, in terms of ” internal compiler passes,”  which is not that useful without a lot of subsequent analysis

“It’s a little informative, but it’s not as informative of as like, which module were you working on?” Klein said. he noted that there is a lot more tooling that still could be built around this output, which would provide more insight into compile times.

So why are your Rust compile times so slow? Eightball says: check back later.

The full discussion can be enjoyed here:

(Update: The post has been updated with Sean Klein’s correct name)