Metrics that Matter with Serverless Inference

Introduction

When teams evaluate serverless LLM (large language model) inference models and providers, the comparison often collapses to a single number, the median tokens per second. It is an easy number to publish and an easy one to rank, and for some workloads it is exactly the right number to optimize. But it is one measurement among many, and on its own it describes only a narrow slice of what “performance” means once a workload reaches production.

The reason is that different workloads feel different bottlenecks. A nightly batch summarization job relies on sustained throughput, so median tokens per second is a fair measure for it. A user-facing chat interface, however, is governed by how fast the first token appears and how consistent that feels, not by the steady-state rate. A production service handling real traffic is governed by its worst requests, its error rate, and its cost per completed answer, none of which are captured by a median throughput figure. Optimize the wrong metric and you can ship something that benchmarks beautifully and behaves badly.

This article covers the metrics that actually matter for production serverless inference, what each one measures, and which workloads should care about it. The goal is to help you pick the measurements that match your use case.