LLVM Begins Landing Distributed ThinLTO "DTLTO" Support
The LLVM compiler toolchain has begun upstreaming support for Distributed ThinLTO "DTLTO" as a new means of handling ThinLTO compilations for leveraging link-time optimizations.
ThinLTO is the more scalable and incremental approach for handling link-time optimizations by LLVM. With Distributed ThinLTO, the distribution of backend ThinLTO compilations can be done via external distribution systems.
The DTLTO Design Overview explains of Distributed ThinLTO:
That page is also noting pretty good performance results out of DTLTO:
As of yesterday, the initial support for Distributed ThinLTO in COFF LLD was upstreamed to LLVM Git for the LLVM 22 version now under development.
This commit lands that initial LLD COFF integration. It's not over the finish line yet for DTLTO but great to see the code beginning to land upstream. Hopefully for the LLVM 22 release next year this initial Distributed ThinLTO support will be ready for developers.
ThinLTO is the more scalable and incremental approach for handling link-time optimizations by LLVM. With Distributed ThinLTO, the distribution of backend ThinLTO compilations can be done via external distribution systems.
The DTLTO Design Overview explains of Distributed ThinLTO:
"DTLTO enables the distribution of backend ThinLTO compilations via external distribution systems, such as Incredibuild. Existing support for distributing ThinLTO compilations typically involves separate thin-link (--thinlto-index-only), backend compilation, and link steps coordinated by a modern build system, like Bazel. This "Bazel-style" distributed ThinLTO requires a modern build system as it must handle the dynamic dependencies specified in the summary index file shards. However, adopting a modern build system can be prohibitive for users with established build infrastructure.
In contrast, DTLTO manages distribution within LLVM during the traditional link step. This approach means that DTLTO is usable with any build process that supports in-process ThinLTO."
That page is also noting pretty good performance results out of DTLTO:
"We have access to a large farm of computers on Windows. For a link of clang.exe on a modest Windows development machine (AMD64 16 cores, 64GB RAM) DTLTO (via sn-dbs.py) was approximately 4 times as fast as multi-threaded in-process ThinLTO.
To estimate the overhead from DTLTO vs in-process ThinLTO, we measured the difference in the time taken to link Clang with in-process ThinLTO using one thread per core, and DTLTO using one local process per core. On both Windows and Linux the overhead was approximately 6%.
Note that, to facilitate review, this PR elides performance optimizations where possible."
As of yesterday, the initial support for Distributed ThinLTO in COFF LLD was upstreamed to LLVM Git for the LLVM 22 version now under development.
This commit lands that initial LLD COFF integration. It's not over the finish line yet for DTLTO but great to see the code beginning to land upstream. Hopefully for the LLVM 22 release next year this initial Distributed ThinLTO support will be ready for developers.