API Rating Agency Brings Consistency to API Measurements

“In the world of open everything, it’s sometimes challenging to remember who everyone is and who the stakeholders are.”

At the livestreamed APIdays London conference held earlier this month, APIMetrics CEO David O’Neill compared the application programming interface or API ecosystem to the restaurant business. Some of the stakeholders are sweating in the kitchen — like your technical team; some are serving — the product managers; some are inspecting and reviewing it — the regulators setting the quality standards. For both situations, the time-to-first-byte (TTFB) is all the matters. Food consumers want to get their food on time, as expected, and API consumers want to get their responses on time, as expected.

But, while hygiene standards and Yelp reviews are pretty widely accessible, quality standards for APIs are still a mystery. We lack meaningful ways to handle disputes.

“There’s an evolving trust gap between the technical standards, the governing body and the people measuring it,” O’Neill said. “Docs are too much, too little, or plain wrong, missing things or misusing standards.” Or they are in PDFs that are impossible to copy-paste code from.

And if we continue to measure only at the API gateway, it’s like having the back-kitchen taste the food, but nobody outside, who has to actually eat the food, gets a say. Without any sort of regulatory or business barriers to access. And no insight into how different stakeholders measure and consume APIs.

“If you can’t agree on what you’re measuring, you will not be able to reach an agreement for other things as well,” O’Neill said. And how can you prove yourself in an argument with one of the big cloud providers, anyway? Metrics may be objective measurements, but they can be really subjective, depending on which side of an integration you’re on and what you’re measuring.

So, how can stewards of the API economy, measure and gain trust over time?

API Rating Agency

In comes the API Rating Agency, a cross-industry advisory group to create an industry-wide agreement on how to create consistency around what to measure, in production from the outside in. This group will set policy for things that are not yet in industry standards:

• How to measure
• What to measure
• What’s good governance and what’s not
• What needs to be in an API contract

As O’Neill put it, “Parties in dispute need somebody outside the delivery chain to mediate. Agreement on what is good or bad in the open ecosystems.”

This is a mix of hard metrics — performance, how, what, why, where and which in production — and soft metrics — API spec, docs, and best practices like non-performance metrics. They will also look to set standards around service level agreements (SLAs) and API deprecation policies. And to check if it matches sector standards, like open banking.

Lorinda Brandon, vice president of engineering at SaaS management platform provider BetterCloud, became a founding board member of the API Ratings Agency to help drive the next phase in the industry’s API maturity. “As someone whose dev teams are reliant on third-party APIs, it’s key to have insights into how they’re performing and how that performance trends over time. Without standards for these API metrics, everything is open to interpretation,” she told The New Stack.

More data simply means more ways of measuring it all. Some of that, like for loans, will open up to more inclusive alternative data, giving millions access to banking for the first time. For others, that will just cause more headaches if we don’t clarify who is measuring what.

“We’re going to have a world where there will be more than one way to measure a credit score.” Similarly, O’Neill told The New Stack in a follow-up interview that they will stop short of calling them API standards. Modeled after the International Open Data Charter, he explained, they will be more general best practices.

It’s not about deciding what’s good or not, but, at the company level, knowing what does or does not meet specifications. And trying to help solve the current confusion that if an API consumer measure that spec one way, and Azure Cloud measures it another way, how do you say who is right or wrong? Because Azure will want to dive into what’s gone wrong. API service providers are keen to figure out what’s not working, but currently, that’s near impossible.

“It doesn’t help you if someone outside your stack can’t get your stuff to work with their systems and you can’t see what the problem is when they’re saying they can’t connect, can’t get access tokens, or your API is giving an error that they can’t see internally. If you only look in one place, you only see the good and people outside your organization can see the bad,” O’Neill said.

The objective of the API Ratings Agency, sometimes called TARA, is to break down barriers between different organizations and proprietary methods to agree on the best way to measure things in the same way so that the measurements become more meaningful.

“APIs have become too critical too quickly for us to not pay attention to some of these things, because too much time has been wasted discussing things that ought to be clear,” O’Neill said. It’s time IT is permanently treated as part of the business.