The New Monitoring for Services That Feed from LLMs

How do you monitor the services that feed from large language models (LLMs)?

That was the question for a discussion with Adrian Cockcroft, who oversaw the development of the Netflix cloud on Amazon Web Services and who is now retired and, lucky for The New Stack, working with us a bit as a podcast co-host and writer/analyst. Look for Cockcroft at Nvidia’s GTC event in March, where he will represent The New Stack.

We are going to a video-first approach with our shows. Check out the presentation that goes with the discussion on our YouTube page. The audio version will still be available on The New Stack Makers podcast feed.

Questions we discussed this week included:

• Why use monitoring in AI apps that use LLMs?
• What are the new golden signals?
• What signals do you need to consider so your apps behave properly?
•  How do they complement existing open source standards that we’ve seen emerge over the years, such as open telemetry?
• What questions should people be asking about when they’re building a scaling operation?
• Is it going to work?

Cockcroft pointed initially to the issues with APIs from LLMs.

“We’re starting to see that some of these API calls take a long time to run,” he said. “They’re expensive to run. So as you start getting these things built, people are saying, ‘Well, OK, I built it using the generic APIs from OpenAI or Anthropic or whatever. But I need to go train my model or do fine-tuning on one of the open source models, perhaps, or license a model and train it.’ And now you’re into something that looks like a very expensive batch job.”

A New Development Model

This new development model feels quite new to everyone, Cockcroft said. Docker and containers forced teams into a new way of running things a decade ago. People learned new tools and new ways of doing things. And that’s how it is now with building and monitoring services using LLMs.

LangChain, for instance, compares to middleware as it sits between the LLM and the service. It’s like the Java Database Controller (JDBC), which served as an interface that connected Java programs to Java databases, said Janikiram MSV, a consultant and frequent TNS contributor in a recent discussion.

LangChain does what JDBC does by connecting the underlying service with the LLM, Cockcroft said.

“And LangChain will talk to whatever model you want behind it,” he said. “And then LangChain also has built into it the various components, checking that your output makes sense and looking at all of the sorts of performance.”

What’s interesting? The LangChain team found that it needed monitoring. So, they started building monitoring capabilities with LangChain. From that work, LangSmith emerged, a monitoring technology and LangChain’s first product. LangChain recently announced it raised $25 million, in a funding round led by Sequoia Capital.

The monitoring tool that’s been around a little longer is called LangKit, built by WhyLabs. Cockcroft said.

“And that is a bit more developed in some ways but probably less integrated,” he said. “So what we’ve got here is the typical kind of thing that happens in open source that we’ve seen over the years: there’s an open source version of something, it becomes successful, then a company is built around it. And that company then starts figuring out, OK, I’m looking for real customers; how can I add value? What are the problems we’ve got in more commercial environments?

“So, if you want to use LangChain, you can do it, no problem. But if you’re an enterprise wanting to integrate it into your line, typically you want to have enterprise monitoring, and you’ve got some support. There are all these things that enterprises want. And you want to have a whole bunch of tooling around that.”