Open Source Builders: How Redis Upended the Database Market

Amazon Web Services (AWS) sponsored this post.

If Salvatore Sanfilippo was going to upend the database market, he at least could have had the decency to have a database pedigree. You know, 10 years engineering Oracle or SQL Server, then he quit to follow his dreams, etc. But no. Sanfilippo says that when he started writing for Redis in 2009 he was a database amateur and new in the database world. In fact, his inexperience in databases led him to break all sorts of sacrosanct rules of “good” database engineering.

Thank goodness.

Today, Redis is one of the world’s most popular databases, and we’re all the beneficiaries of Sanfilippo’s rule-breaking. Though there’s no “Redis Inside” sticker affixed to the various products and services that depend on Redis, odds are you use Redis daily because Uber, Instacart, Slack, Hulu, Twitter, Instagram, and many more companies are using it. Nor is Sanfilippo going to tell you how you should use Redis. “What Redis is good for is not my choice—it’s the application developer that knows [best],” he says.

To better understand how Sanfilippo came to create this open source staple of enterprise computing, and where it goes from here, I chatted with the Redis founder.

No DBA Expertise Required

Prior to writing Redis, Sanfilippo had made a name for himself in security circles, as well as other areas. In security he created hping, a network security tool; and Idle Scan, a port scanning technique. He also created Visitors, a weblog analyzer; the Jim interpreter, a small-footprint implementation of the Tcl programming language; and a few device drivers. He wasn’t, in short, sitting around. He was busy.

Part of that busyness related to his efforts to scale his startup and, specifically, a real-time analytics service called lloogg.com. Sanfilippo struggled to make it scale with a relational database (MySQL). It worked, but he says that the amount of hardware needed for 10,000 users was too much compared to the economics of the application. The idea struck him that an in-memory approach would be ideal and, more pertinently, that his workload — pushing data to a list then trimming it to a fixed size — simply didn’t fit the relational model. He began building Redis.

Creating Redis wasn’t about picking between different databases. Sanfilippo had to think differently about the underlying data model. He ended up building an in-memory NoSQL database that is both data store and cache, and breaks all sorts of database “rules” along the way.

Indeed, Sanfilippo’s approach was roundly criticized by the database intelligentsia. First, people dinged Redis for being in-memory. They also dismissed the Redis model of persisting on disk using the fork system call and Lua scripts using SHA1s of their bodies instead of names. They were, perhaps, right about how traditional databases were built.

They were dead wrong about the future of databases.

How to Win Friends and Influence People

Sanfilippo says that while the database experts mostly criticized his work, it left him free to innovate a new kind of database. Not surprisingly, he chose to innovate in the open, given that open source had long been a core part of him. “In general I believe in sharing things with others,” he says. According to Sanfilippo, his career was spawned by installing Slackware 1.2.3 when he was 18, and he realized there was a C compiler inside and a lot more. “In practical terms, my whole story as a software developer started with open source, so for me it is the obvious way,” he says.

But open source isn’t just a matter of licensing for Sanfilippo.

“I’m convinced that software is a human process and needs people telling stories,” Sanfilippo says, “So I was recounting the story of Redis everywhere.” Quickly people listened and started embracing Redis, initially within the startup community, and later within established enterprises.

Though initially, Sanfilippo notes he valued his freedom more than the help of so-called database experts, over time a community has gathered around Redis. Sanfilippo says that, after 10 years, “The Redis core is definitely a collaboration between many individuals.”

Not that he’s taken his hand off the wheel: Sanfilippo admits at times he has been very bad at delegating, to the point that he still writes all the Redis documentation himself.

Though Sanfilippo welcomes contributions to Redis, he says that from the community you get a mix of good and bad ideas. For example, someone might be trying to use the software to solve a different problem from those Sanfilippo hopes to conquer. Also, there’s the fact that system software is hard. “You tend to get serious contributions only from folks that are paid to write such contributions most of the time,” he says, although of course there are exceptions. Contributing to a serious database system like Redis is not something you can do Sunday morning to relax, he admits.

You Choose

Sanfilippo has had his share of stress over the past decade, wondering whether Redis would succeed and whether he’d have enough money to cover his needs. Fortunately, he says those concerns resolved themselves automagically because Redis instead continued to grow in popularity. Today Sanfilippo is fortunate to get to devote all of his time to Redis as an employee of Redis Labs.

Just don’t expect him to tell you how you should use Redis.

“What Redis is good for is not my choice; it’s the application developer that knows better,” he says, explaining that every application has its goals, guarantees it must provide, and latency and scalability concerns. Sanfilippo says he wants Redis to be used when it solves a problem: be it a primary database, just an index for another database, some smart caching, messaging, or whatever.

Instead of “pre-baking” the future for Redis, Sanfilippo says he has provided a set of building blocks, so people can find ways to assemble them creatively. Redis is also easy to modify. “So often with some changes, developers are able to make Redis do things that it was not intended for,” Sanfilippo says. Which is OK.

Sanfilippo doesn’t see himself as some authoritative leader. “I like more to design new things and code them,” he says, and if others follow, great. Because with Redis, concludes Sanfilippo, “the developer is the queen or the king — it’s up to them.”

Visit the AWS Open Source Blog to learn how open source projects can apply for AWS promotional credits.

Redis Labs is a sponsor of The New Stack.

Feature image via Pixabay.