Linux 6.16 Performance Regression Tracked Down In New Futex Code
Sent out this morning as part of this week's "locking/urgent" pull request is a performance regression fix ahead of today's Linux 6.16-rc5 release. This latest performance regression in the Linux kernel is around the new Futex code merged this cycle with a big performance hit being observed in scheduler benchmarks.
The locking/urgent pull request sent out this morning is disabling the FUTEX_PRIVATE_HASH option merged back during the start of the Linux 6.16 merge window. The FUTEX_PRIVATE_HASH functionality was introduced as part of the work on supporting task local hash maps that landed alongside some nice Futex improvements in Linux 6.16.
The FUTEX_PRIVATE_HASH code ended up causing some significant performance regressions though so is being disabled disabled for Linux 6.16 and then for Linux 6.17 the kernel developers will attempt to re-introduce it with a proper performance fix.
Meta engineer Chris Mason reported the performance regression that was able to mimic real-world performance workloads, unlike some earlier Futex benchmarks that showed some performance regressions just in micro-benchmarks. So FUTEX_PRIVATE_HASH with today's pull request is being gated by the "BROKEN" Kconfig option so it's effectively disabled for the rest of the Linux 6.16 cycle.
Chris Mason reported the performance regression with a pretty sizable performance hit:
The requests per second dropping by 36% on a high-end AMD EPYC 9005 "Turin" server. Ouch.
So with this pull request expected to hit Linux Git today ahead of the Linux 6.16-rc5 release, the regression will be avoided.
The locking/urgent pull request sent out this morning is disabling the FUTEX_PRIVATE_HASH option merged back during the start of the Linux 6.16 merge window. The FUTEX_PRIVATE_HASH functionality was introduced as part of the work on supporting task local hash maps that landed alongside some nice Futex improvements in Linux 6.16.
👁 Disabling the regressed code
The FUTEX_PRIVATE_HASH code ended up causing some significant performance regressions though so is being disabled disabled for Linux 6.16 and then for Linux 6.17 the kernel developers will attempt to re-introduce it with a proper performance fix.
Meta engineer Chris Mason reported the performance regression that was able to mimic real-world performance workloads, unlike some earlier Futex benchmarks that showed some performance regressions just in micro-benchmarks. So FUTEX_PRIVATE_HASH with today's pull request is being gated by the "BROKEN" Kconfig option so it's effectively disabled for the rest of the Linux 6.16 cycle.
Chris Mason reported the performance regression with a pretty sizable performance hit:
"On my big turin system, this commit slows down RPS by 36%. But even a VM on a skylake machine sees a 29% difference."
The requests per second dropping by 36% on a high-end AMD EPYC 9005 "Turin" server. Ouch.
So with this pull request expected to hit Linux Git today ahead of the Linux 6.16-rc5 release, the regression will be avoided.