Ollama has become the default answer when someone asks how to run a local LLM, and for good reason. It's easy, it works across platforms, and it hides enough of the ugly parts that you can go from nothing to a working model in minutes. llama.cpp sits underneath a huge amount of the local AI world too, especially if you're using GGUF models, so neither one is going away.

The problem is that the easy button stops being enough once a local model becomes part of a workflow. You start caring about serving APIs, batching, structured outputs, cache behavior, Mac-specific acceleration, mobile deployment, or whether you're silently leaving performance on the table. I still think Ollama is the easiest way to start running local LLMs, but it's not where I want to stay when I'm building something more serious.

The alternatives are messier, but they give you control over the parts Ollama tries to hide. If you're running agents, pointing several apps at the same model, working on a Mac, or trying to make a consumer GPU behave like a proper inference box, the runtime starts to matter as much as the model.

vLLM and SGLang turn local models into infrastructure

Serving a model is different from chatting with one

vLLM is the first tool I'd look at when you want a local model to act less like a desktop app and more like an inference service. It has an OpenAI-compatible API server, high-throughput inference, continuous batching, prefix caching, chunked prefill, structured outputs, tool calling and reasoning parsers, and support for a lot of quantization formats.

Those features matter when the model is being called by coding tools, agents, RAG experiments, or multiple apps at once. A single prompt in a terminal doesn't need much scheduling logic, but a local endpoint being hit repeatedly certainly does. Especially when those requests share context, run long, or need to avoid wasting VRAM on cache management.

vLLM's best-known feature is PagedAttention, which manages the model's key-value cache more efficiently. The goal is to keep GPU memory from becoming the bottleneck when multiple requests are active or when context gets large. It's not going to make every local setup faster, but it's the reason why vLLM shows up so often across the internet, especially in higher-throughput deployments.

SGLang sits in the same broad category, but its identity is more tied to structured generation, repeated prompt patterns, and agent-style workloads. Its feature list includes RadixAttention for prefix caching, prefill-decode disaggregation, speculative decoding, continuous batching, paged attention, chunked prefill, tensor and expert parallelism, and multi-LoRA batching.

Free-form prose is fine in a chat box, but it becomes a problem when your program expects JSON, a schema, or a tool call in a particular format. SGLang is built for repeated prompts, constrained outputs, and cache reuse, which are all much easier to care about once the model is driving tools instead of just answering questions.

I wouldn't install either of these before getting acquainted with simpler tools, as they add quite a bit of setup work and assume a certain level of understanding to use. However, they make the most sense once you're using other software that expects a more enterprise-esque endpoint. If your local LLM has become backend infrastructure for your home lab, vLLM and SGLang are closer to the right shape of what you need.

vMLX gives Macs a more serious local app

Apple Silicon doesn't need to pretend it's CUDA

Mac users have always had a slightly different local LLM story. Apple Silicon's unified memory makes large models more practical than you might expect on a laptop, but the software stack isn't the same as it is on a Linux box with an Nvidia GPU. You can run llama.cpp with Metal, and that works well, but there are good reasons to want tools built with Apple's stack in mind from the start.

vMLX is interesting because it aims to sit closer to the app experience people want from Ollama or LM Studio, while borrowing ideas from more serious serving stacks. Its own pitch calls out prefix caching, paged KV cache, continuous batching, and MCP tools. It's a very different pitch from "download a model and chat with it," and it's why it deserves to be treated as more than just another Mac wrapper.

MLX is Apple's array framework for Apple Silicon, with lazy computation, dynamic graphs, CPU/GPU execution, and a unified memory model where arrays live in shared memory. MLX-LM then gives you text generation, Hugging Face integration, quantization, and fine-tuning on top of that, while MLX-VLM covers vision-language models on the same general stack. vMLX is the app-level tool, while MLX-LM and MLX-VLM are the lower-level options when you want to work closer to the model. Granted, I wouldn't describe any of this as a universal replacement for vLLM or SGLang, because it isn't, but it's still a great tool to have if you're a Mac user.

vMLX is best understood as the Mac-native path through the local LLM world, rather than a CUDA tool awkwardly mapped onto Apple Silicon. The memory model, GPU stack, and app expectations are different enough that native tools like these can genuinely provide benefits.

MLC-LLM and ExLlamaV3 target specific hardware problems

Phones, browsers, and consumer GPUs don't all want the same runtime

MLC-LLM is built around machine-learning compilation and deployment across a variety of different platforms. Its support includes web browsers through WebGPU and WASM, iOS and iPadOS through Metal on Apple A-series GPUs, and Android through OpenCL on Adreno and Mali GPUs.

MLC fills a different role from a normal server runtime, even though it can expose OpenAI-compatible APIs. It's built for more niche use-cases, and WebLLM runs inference directly in the browser with WebGPU acceleration and no server. It also supports streaming, JSON mode, and structured JSON generation.

MLC isn't what I'd pick for serving a big model to a home lab full of apps. Its appeal is deployment into places that don't look like normal LLM hosts: places like browsers, phones, tablets, and embedded apps. It's aimed at a totally different class of local AI projects than what vLLM and SGLang are aimed at.

ExLlamaV3 is specialized in the other direction. It's the current version of the ExLlama line now that ExLlamaV2 is archived, and it's esssentially an inference library built specifically for running local LLMs on modern consumer GPUs. The priorities are fitting the model, keeping context usable, avoiding wasted VRAM, and getting acceptable speed without enterprise hardware.

Its EXL3 quantization format, tensor-parallel and expert-parallel inference for consumer hardware, continuous dynamic batching, speculative decoding, cache quantization, multimodal support, and LoRA support are all built for that goal. TabbyAPI gives it an OpenAI-compatible server too, so it can still slot into apps that expect a normal local endpoint.

There's more than just Ollama and llama.cpp out there

Try something more specialized

If you deploy your own local LLMs, Ollama and llama.cpp are both fine to start with and even continue with. However, if you find yourself wanting more, there's a whole world out there of software you can try out that might be better suited to your needs. For example, MLC and ExLlamaV3 don't solve the same problem, but both are more specialized than Ollama. MLC is for deployment across awkward targets. ExLlamaV3 is for getting more out of a consumer GPU. They're not the first tools I'd recommend to someone starting out, but they make sense once your hardware or deployment target starts dictating the runtime.

Then there's llama-swap, part of the llama.cpp package of model serving tools, and it's useful if you run multiple OpenAI or Anthropic-compatible local servers and want a routing layer between them. Then there's TensorRT-LLM, which is the optimized path made by Nvidia for Nvidia cards, LMDeploy is a real serving and deployment toolkit, Lemonade is a model serving platform built with AMD in mind, KTransformers handles heterogeneous CPU/GPU inference, and LocalAI covers more modalities and hardware targets.

Ollama is still the tool I'd point someone to if they just want to get started. llama.cpp is still foundational, and dismissing it as basic would be unfair given how much it can do on its own. But once local models become part of a real workflow, the runtime is no longer merely a stepping stone. The server, cache, batching model, quantization path, and platform backend start deciding what you can actually build.