VOOZH about

URL: https://thenewstack.io/greg-kroah-hartman-lessons-for-developers-from-20-years-of-linux-kernel-work/

⇱ Greg Kroah-Hartman: Lessons for Developers from 20 Years of Linux Kernel Work - The New Stack


TNS
SUBSCRIBE
Join our community of software engineering leaders and aspirational developers. Always stay in-the-know by getting the most important news and exclusive content delivered fresh to your inbox to learn more about at-scale software development.
REQUIRED
It seems that you've previously unsubscribed from our newsletter in the past. Click the button below to open the re-subscribe form in a new tab. When you're done, simply close that tab and continue with this form to complete your subscription.
The New Stack does not sell your information or share it with unaffiliated third parties. By continuing, you agree to our Terms of Use and Privacy Policy.
Welcome and thank you for joining The New Stack community!
Please answer a few simple questions to help us deliver the news and resources you are interested in.
REQUIRED
REQUIRED
REQUIRED
REQUIRED
REQUIRED
Great to meet you!
Tell us a bit about your job so we can cover the topics you find most relevant.
REQUIRED
REQUIRED
REQUIRED
REQUIRED
REQUIRED
Welcome!

We’re so glad you’re here. You can expect all the best TNS content to arrive Monday through Friday to keep you on top of the news and at the top of your game.

What’s next?

Check your inbox for a confirmation email where you can adjust your preferences and even join additional groups.

Follow TNS on your favorite social media networks.

Become a TNS follower on LinkedIn.

Check out the latest featured and trending stories while you wait for your first TNS newsletter.

PREV
1 of 2
NEXT
VOXPOP
As a JavaScript developer, what non-React tools do you use most often?
Angular
0%
Astro
0%
Svelte
0%
Vue.js
0%
Other
0%
I only use React
0%
I don't use JavaScript
0%
Thanks for your opinion! Subscribe below to get the final results, published exclusively in our TNS Update newsletter:
NEW! Try Stackie AI
From clobbered drafts to real-time sync
Apr 14th 2026 10:00am, by David Moore
TypeScript 6.0 RC arrives as a bridge to a faster future
Mar 14th 2026 9:00am, by Darryl K. Taft
Mastra empowers web devs to build AI agents in TypeScript
Jan 28th 2026 11:00am, by Loraine Lawson
2020-11-29 06:00:26
Greg Kroah-Hartman: Lessons for Developers from 20 Years of Linux Kernel Work
profile,
Linux / Software Development / Tech Culture

Greg Kroah-Hartman: Lessons for Developers from 20 Years of Linux Kernel Work

Greg Kroah-Hartman, the Linux Foundation fellow currently responsible for stable Linux kernel releases shared the lessons he's learned as a kernel developer that are applicable to other developers at this year's Linux App Summit.
Nov 29th, 2020 6:00am by David Cassel
👁 Featued image for: Greg Kroah-Hartman: Lessons for Developers from 20 Years of Linux Kernel Work

👁 898px-Greg-Kroah-Hartman in 2011 (photo by Sebastian Oliva via Wikipedia).jpeg

Greg Kroah-Hartman, the Linux Foundation fellow currently responsible for stable Linux kernel releases shared the lessons he’s learned as a kernel developer that are applicable to other developers at this year’s Linux App Summit.

Kroah-Hartman has been helping develop the Linux kernel for over 20 years — “way too long,” he jokes — but reminded the audience that he has done some userspace work and maintains some small Linux userspace packages. He started by showing how he could succinctly distill the essence of the talk into a single five-word slide:

“Don’t make your users mad.”

“Let’s go into a little more detail, though,” he adds with a laugh. “I’ll try to explain why you don’t want to make users mad, and what happens when you do.”

“So, all this stuff comes down to a pretty simple thing: who are you writing code for?”

It’s important because what you should do depends on why you’re doing it, he explains, starting with the easiest scenario. If you’re writing code for yourself, “Throw it over the wall and have fun.”

But if you’re writing it for others, “Now you have to care about users.”

“Within reason,” he adds, sharing a corner-case example from what he calls the infamous XKCD cartoon illustrating how users can complain about even the most essential of fixes — like “the CPU no longer overheats when you hold down spacebar.”

“Everything I say here is ‘within reason’, because there are always some users you can never please, no matter what. And those people you can safely ignore.”

Kroah-Hartman explains that’s one of Linus Torvalds’ most deeply-held convictions: don’t break userspace.

“Other operating systems have this rule as well — it’s a very solid rule — because we always want you to upgrade. And we want you to upgrade without worrying about it. We don’t want you to feel scared. If you see a new release, and we say, ‘Hey, this fixes a bunch of problems,’ we don’t want you to feel worried about taking that. That’s really really important — especially with security.”

This leads to what Kroah-Hartman called Greg’s rule #1: “Users will not update if they are afraid you will break their current system.”

“We’ve learned this the hard way…”

Evolving with Users

This obviously informs the way that changes ultimately get made. “The trick is you just support everything until there are no users,” Kroah-Hartman explained. “Figuring out when there are and are not users is up to you.”

Kroah-Hartman told a remarkable story about the time where the Linux kernel developers realized one particular architecture was only being used by exactly two machines in existence. “One of them broke, and the other one — we actually paid the developer to replace it, because we didn’t want to support that thing anymore.”

“But otherwise, we have to keep supporting users. Because we made that contract. We made that contract that we gave them a functionality. We cannot remove it.”

He points out that Linux can still support 25-year-old binaries, so “People think Linux has stagnated. But in reality, we’ve created new functionalities right next to the old functionalities, that you can do different things — you can take advantage of stuff.”

He provided another very prominent example of an application evolving slowly that a few notice: KDE.

In the Real World

Another piece of advice for application developers: Don’t take away things that work. “If somebody is using this functionality, keep it there. It’s that simple… Don’t think by removing it, you’re doing them a favor.”

When he turned his attention to library developers, his first slide simply read: “I pity you.”

“You never know if an API is really useful, until you have too many people using it to ever be able to change it.”—Greg Kroah-Hartman

“This is the hardest job ever,” he said with a laugh. “I really, really feel sorry for you… It’s one of the hardest things to do.”

“You can never know if your API actually works until you have a lot of people using it. And by the time you have a lot of people using it, then you realize all the problems in it — and then you can’t change it.” He said, laughing. “Getting it right the first time is almost impossible.”

👁 Slide from Greg Kroah-Hartman talk at Linux App Summit 2020 - library developers should not deprecate

There is one alternative — but it still involves honoring the users who are still using that old library that you want to deprecate. If you really need to make a fresh start — “do that as a whole new library,” he advised.

“And then you have no users! And then you don’t know if anybody’s using your API, you don’t know if you got it right, until you have users, and then it’s broken — and the cycle continues.”

He also offered an important tip for developers who actually want their users to switch over to that new library. “You need to document it really well — because if you don’t Stack Overflow is going to try to do it.” He laughs knowingly, remembering a flood of bad drivers and submissions that were apparently inspired by a 10-year-old Stack Overflow post. “You don’t want Stack Overflow to be your primary documentation. Provide good documentation.”

But then he added seriously, “If you violate those things, you cause people to lose trust. If I rebuild and see ‘deprecated’ all over the place or APIs changed, it’s just horrible. You don’t want to do that. If you violate that trust, you won’t have that user anymore.”

If you do make a change, make sure there truly is a compelling reason. “You have to provide enough reason and enough goodness to force somebody to take the time to learn to do something else. That’s very rare.”

His example of this was systemd, which unified a variety of service configurations and initialization processes. “They did it right. They provided all the functionality, they solved a real problem that was there. They unified all these existing tools and problems in such a way that it was just so much better to use, and it provided enough impetus that everybody was willing to do the work to modify their own stuff and move to the new model.

“It worked. People still complain about it, but it worked. Everybody switched… It works well. It solves a real problem.

“That was an example of how you can provide a compelling reason to move on — and make the change.”

TRENDING STORIES
David Cassel is a proud resident of the San Francisco Bay Area, where he's been covering technology news for more than two decades. Over the years his articles have appeared everywhere from CNN, MSNBC, and the Wall Street Journal Interactive...
Read more from David Cassel
SHARE THIS STORY
TRENDING STORIES
The Linux Foundation is a sponsor of The New Stack.
TNS owner Insight Partners is an investor in: Real.
SHARE THIS STORY
TRENDING STORIES
TNS DAILY NEWSLETTER Receive a free roundup of the most recent TNS articles in your inbox each day.
The New Stack does not sell your information or share it with unaffiliated third parties. By continuing, you agree to our Terms of Use and Privacy Policy.