New Open Source Standard Brings Consistency to Webhooks

Ken Ruf spends a lot of time thinking and reading about the challenges with webhooks as part of his job at Svix, a “webhooks as a service” company. Most of the complaints are from developers who are receiving webhooks, rather than sending them, he told The New Stack.

“We would see a lot of times people just complaining about webhooks and how they suck,” said Ruf. ”From watching a lot of the conversation, our hypothesis was that the big problem is fragmentation. So many people are sending in so many different ways that people receiving them have to basically redo everything every time when they have a new source that they want to receive webhooks from.”

Webhooks differ from APIs in that they are used primarily for real-time data and to trigger automation workflows. Use cases include chat messages, payment alerts, inventory updates, order status changes, and task creation events such as customer onboarding. With webhooks, the receiving application subscribes to events by providing a url endpoint to the source application.

“Webhooks act as HTTP callbacks that enable services to notify one another about events,” wrote Vincent Le Goff, a senior staff software engineer at API gateway provider Kong. Le Goff sits on the new standard steering committee on behalf of Kong. “They function as a ‘reverse API,’ where instead of a client initiating requests to a service via an API call, the service proactively triggers a webhook to push updates to the client. For example, a service might trigger webhooks for events like ‘a user has paid’ or ‘task is complete.’”

APIs, by contrast, are more often used for two-way data exchanges and tend to involve some data latency. API polling is like the Bart and Lisa Simpson in the back of a car — always asking “Are we’re there yet,” Ruf said. Webhooks are quieter — more like Maggie, who awaits arrival without all the chatter. Another analogy he offered is that webhooks are like having a doorbell to alert you to a guest — you don’t have to constantly check the front door because you receive a message when the guest arrives.

“They’re very flexible,” Ruf said. “It’s really anytime you want to trigger a workflow in your system based on an event that happens in another product or another application.”

But until last month, webhooks lacked a standard design approach. For its State of Webhooks report, Svix looked at ten factors across 100 provider documents and found that not one had the exact same implementation across those ten factors, Ruf said. That’s an issue for developers, particularly as webhooks become more pervasive.

“What happens is I have most of my code, but I have to change it because they don’t have this one thing out of the ten, and then because all of them are different, … I have to change a bit again, over and over, instead of just being able to have different versions of the same endpoint for different providers,” he said. “You literally need to copy most of it and then change a few things here and there.”

One example of the problem: There’s variation in how often webhooks automatically retry failed messages. The State of Webhooks report found that 67% of services offered automatic retries, with the most common amount of retries offered being 5 — most offer between 3-10 retries. The best practice is an exponential back off, Ruf said.

This lack of standards led Ruf and Tom Hacohen, founder and CEO of Svix respectively, to decide to create a set of standards for webhooks. Last year, the pair formed a coalition to do just that.

“What we really wanted to do was get people who are sending and receiving lots of webhooks at scale, to really add weight to the committee, weight to the standard itself,” Ruf said.

In addition to Hacohen, the technical steering committee includes representatives from:

• Zapier, a web applications integration company;
• Twilio, a web communication company;
• Lob, a direct mail system company and Svix customer;
• Mux, a video streaming company;
• ngrok, a unified ingress platform;
• Supabase, the open source Firebase alternative; and
• Kong, a service mesh and API gateway, which has some webhook functionality built in, according to Ruf.

Last month, the body issued the open source Standard Webhooks specification on GitHub, as well as launching a site, Standard Webhooks, which offers information about contributing to the standard, the governing body, and open source tools to verify webhooks and to simulate a standard webhooks message.

The standards will help developers who have to to set up endpoints, he said. Adopting the standards will make it easier to reuse code instead of custom coding everything from scratch, Ruf said.

“When you are going through and trying to create a new endpoint for a new webhook from another application, you can reuse a lot more of the webhook code that you’ve already written,” he said. “Right now, you’re basically needing to write everything from scratch. So that’s one benefit of the standardization and one thing that we’re looking to try to accomplish is make it easier for people to adopt webhooks from a variety of different providers.”

The standards specify, among other things:

• Ideal payload size for webhooks (smaller than 20kb);
• Webhook metadata;
• Webhook headers; and
• Signature scheme.

The standard also sets the bar for what makes a quality webhook by establishing best practices, he added. For instance, as it stands, whether or not a webhook triggers an authentication request is up to the individual developer.

“Right now, people are all over the place, and it’s a real pain to try and receive webhooks from different providers, but also we want to make it as easy as possible for people to offer a good webhook solution, because that’s like another pain point,” he said.

The standard not only outlines that authentication should be part of a webhook process, but it offers an opinion on what authentication method is best for webhooks: Hash-based message authentication code (HMAC) signatures.

Ruf is not aware of any competing standards that are specific to webhooks. There’s a specification called Cloud Events that describes event data in a common way but it only briefly addresses webhooks, he said.

“We read their actual standard and we thought that there’s a lot more that needs to be done there. It’s not specific enough,” he said, adding that their goal is to provide a complementary standard that’s compatible with some of the work that’s already been done.

“We’re just trying to make these developers’ lives easier when they have to implement workbooks, whether they’re implementing it for their company, to send it to their users, or if they’re just trying to receive the webhooks from other people to trigger workflows automations inside their product,” Ruf said. “If we can get everybody to adopt it, then it makes everybody’s lives easier.”