Flexible IO Tester: Practical Testing Tool for Developers, Creative Outlet for Its Creator

Amazon Web Services (AWS) sponsored this post.

For some people, open source is a full-time job. For others, it’s a side gig. For Jens Axboe, it’s both.

Axboe is the lead developer and maintainer of the Linux kernel’s block IO subsystem. Related to this work, he has written an impressive array of code, from different IO schedulers to developing and maintaining ATAPI, to SATA NCQ, to (most recently) io_uring. Oh, and he also happened to build the Flexible IO Tester (fio), which has become the industry standard for modeling and stress-testing storage workloads.

As seems often to be the case with the best open source projects, Axboe wasn’t trying to create an industry standard. He just needed an easy way to reproduce storage workloads in order to look for problems with the Linux code. Developers were filing bugs, but he was struggling to simulate the workloads that generated them. Fio enabled the developer to describe the workload so that Axboe could build and run it without having to write simple applications to test various configurations or changes to the kernel code.

The other thing fio did was for Axboe, personally — it gave him a creative outlet, a place to have fun. I interviewed Axboe as part of the Open Source Builders Series, to glean insights into effective open source community management, the most valuable kinds of contributors, and more.

A Creative Side Project

While Axboe has been fortunate to collect a paycheck for giving away open source software, he’s never been paid for fio. That’s always been a side project for him, he says. For the first several years, fio was a labor of love—something Axboe worked on alone. Even as he built new functionality into fio to help with his Linux kernel work, it remained separate from his full-time job (doing systems engineering for a variety of companies, currently Facebook), which has always suited him just fine.

According to Axboe, fio is a creative outlet where he controls everything. This isn’t because Axboe is a control freak who can’t get along with others, but it’s simply a natural human desire to express himself. “When you work on other people’s projects, which the Linux kernel ultimately is, there’s a rigorous review process, which we need,” he explains. “Also, other people need to sign off on your code, and you have to convince Linus to accept it. Plus kernel programming is just a little more difficult, and debugging is harder. So fio has been a way for me to creatively do things in [the] userspace that was just easy. Whenever I got a little tired of the kernel side of things, or if I needed a little break, I’d develop new things on fio, because it was just fun and liberating.”

This creative outlet persists for Axboe, even though he doesn’t author most of the code in fio these days. Mostly he’s merging code from others, while working on more complicated tasks that might arise from new storage hardware, for example. Regardless, fio remains a creative outlet for Axboe, similar to other open source projects he’s created to help him get work done.

Attracting the Contributor Class

If Axboe today is able to mostly maintain fio by merging others’ code, it’s because he’s done a good job of attracting talented developers to work on fio. In his estimation, any project that wants to thrive over a number of years needs to attract new blood. The alternative is for the project to become dormant. Why would this happen? Sometimes the original developer might get busy working on other things or could simply tire of the project. If a vibrant community is in place, the risk to the project is minimized.

Sharing the burden of leading a project is also important. For Axboe, open source projects don’t become burdensome in terms of code, but rather in terms of code contributors — people, in other words. This side project attracts users who can become demanding. “It’s supposed to be a labor of love, something you do for fun and because you find it exciting,” he says, but suddenly this side project has you inundated with support issues.

This is perhaps why Axboe sees the contributor world as split between givers and takers. The latter category includes people who just use the software. If something doesn’t work, they may or may not file a bug report. “Working with these users can suck up a lot of time,” Axboe says, “And this is also where you sometimes get slightly demanding requests.” From Axboe’s perspective, he’s not in the project simply to have tons of users, as he thinks that metric is useless by itself. “It’s a nice side effect, but I’d put that very low on my list,” he says. Users who simply file bug reports bury him in problems without offering assistance in digging out.

(It’s perhaps worth noting the stark difference between Axboe’s experience with such “free riders” and that of Rich Felker, creator of musl libc. Felker loves users of all kinds because adoption ensures that others will need to do the work to integrate with musl. So perhaps in his case, users of the software don’t create the headaches they do for Axboe. My sense is that Axboe’s reaction is the more common.)

The second category of user, however, is different. This group uses the software, and is capable of making changes. They send in a patch (or patches), fixing issues they ran into, or adding features that they need.

“I love these users!” Axboe says. “They help drive the project forward, in the true spirit of open source, by contributing back. The software and project is better for it, and it makes their life easier, too.” And this class does not only include coders. “The biggest hole fio has — and probably a lot of other open source projects — is someone writing quality documentation. My documentation isn’t great.”

So, I asked, how can a project attract more of the kind of users who also contribute — code, documentation, and so on?

On Leadership

The first and foremost need within a project, Axboe says, is a strong leader. Someone who isn’t too afraid to make decisions, and someone who has the “technical chops” to make those decisions.

“It’s very important that people have a certain amount of technical respect for the leader of the project,” he explains, “So whenever he or she has to make difficult decisions, they’re generally acknowledged as being a good decision.” Absent such leadership, in his estimation, nothing else can save the project. It will drift and then, most likely, die.

Another key component of open source success is communication. “Having an environment that is welcoming to new contributors is very important,” Axboe says, such that newcomers feel comfortable asking questions or proposing changes to the project.

On that note, if you’d like to get involved with fio, it’s always good practice to start by reading the documentation. Wish the documentation were better? So does Axboe! He’d love to see your contributions. Once you’re ready to start tweaking the code and submit a pull request, find fio on GitHub.

Visit the AWS Open Source Blog to learn how open source projects can apply for AWS promotional credits. Come build with us.

The Linux Foundation is a sponsor of The New Stack.

Feature image via Pixabay.

At this time, The New Stack does not allow comments directly on this website. We invite all readers who wish to discuss a story to visit us on Twitter or Facebook. We also welcome your news tips and feedback via email: feedback@thenewstack.io.