📚 Related: Best Local LLMs for Mac (2026) · Apple M5 Pro/Max for Local AI · Ollama on Mac (0.30 setup) · Mac vs PC for Local AI

This guide is the foundational how-to for running local LLMs on Apple Silicon. It covers the mechanics that make Mac different (unified memory), the runtime decision specific to Apple Silicon (MLX vs Ollama vs llama.cpp Metal), how to verify Metal is doing its job, how to turn a Mac Mini into a silent always-on AI server, and the troubleshooting you’ll actually hit.

It deliberately doesn’t cover two things: which model to run on your specific Mac tier (that’s the Best Local LLMs for Mac 2026 page, which has the current 2026 picks across every memory tier from 8GB to 192GB), and whether to buy an M5 Pro or Max (that’s the M5 Pro/Max for Local AI page, which has the Neural Accelerator breakdown and upgrade-from-M4 framing). Those are model-pick and hardware-buying jobs; this page is the how-it-works underneath.

If you want the one-line version: install Ollama, run ollama run qwen3.6, your Mac is already an AI workstation. The rest of this guide is why that works, when it doesn’t, and how to make it work better.

Unified Memory: The Mechanic That Makes Mac Different

Every Apple Silicon Mac uses a unified-memory architecture. On a PC, your GPU has its own dedicated VRAM (typically 8-24 GB on consumer cards), and the system has its own DRAM, and the two communicate over PCIe — every byte the GPU needs to read from system RAM has to be copied across that bus, which is expensive. On Mac, there is no separate GPU memory. The CPU and GPU share a single pool of high-bandwidth memory, and both can address every byte of it directly. No copy, no transfer overhead. That’s it. That’s the architectural superpower.

What this means in practice for local LLM inference:

• Your entire RAM is potentially loadable model. An M4 Max with 128 GB unified memory can hold model weights that no consumer discrete GPU can touch. A 70B model at Q4_K_M (~40 GB) fits comfortably; a single RTX 3090 (24 GB VRAM) cannot load the same model without offloading layers to system RAM, which kills throughput.
• Memory pressure is not the same as swap. macOS reports memory pressure in Activity Monitor; yellow or red pressure does not mean the system has started swapping to disk yet, but it does mean macOS is squeezing existing allocations harder to make room. Sustained heavy pressure can eventually push to swap, which is where Mac LLM performance collapses. The rule of thumb: keep pressure green by managing what else is running, not by relying on macOS to be clever about it.
• Memory bandwidth matters more than chip generation. This is the counterintuitive one. An M3 Max at ~400 GB/s generates tokens faster than an M4 Pro at 273 GB/s on the same model, despite the M4 Pro being a newer chip. Token generation in LLM inference is memory-bandwidth-bound, not compute-bound. Bandwidth wins.

A quick reference for the chip-to-bandwidth lookup (used for orientation, not for picks — see the Best Local LLMs for Mac 2026 page for what to run on each):

The M5 Max at 614 GB/s and M5 Pro at 307 GB/s shipped on March 11, 2026, with Neural Accelerators in every GPU core delivering roughly ~4× prompt-processing speedup vs M4 per Apple’s own MLX research and community benchmarks (token generation, by contrast, is bandwidth-bound and tracks bandwidth changes — community benches put M5 Max token gen at roughly +28% over M4 Max). The Mac Studio M5 Max / M5 Ultra refresh is delayed to roughly October 2026 or later per Mark Gurman / supply-chain reporting; the bottleneck is memory and storage shortages industry-wide. If you need a desktop Mac for local AI in the next 6 months, M4 Max or M3 Ultra now, not M5 Ultra Wait. The full M5 architectural analysis lives in the M5 Pro/Max for Local AI guide.

MLX vs Ollama vs llama.cpp Metal: The Mac Runtime Decision

The three runtimes most Mac local-AI users will encounter. The picture in mid-2026 is meaningfully different from where it was in 2025.

MLX — the consensus speed pick

Apple’s own ML framework, designed specifically for unified memory. MLX operates directly on shared memory with no CPU↔GPU copy overhead, runs on Metal, and benefits the most from M5’s Neural Accelerators (which live inside the GPU cores MLX targets). Community benchmarks across late 2025 / early 2026 consistently put MLX at 20-50% faster than Ollama or llama.cpp on Apple Silicon for the same model when MLX-native conversions are available. Apple’s own MLX research backs this up directionally; community measurements (llmcheck.net, Andrew Ooo’s Ollama 0.19 MLX writeup, Yage.ai analysis) tighten the numbers per model.

The mlx-community Hugging Face org ships MLX-converted variants of nearly every mainstream model within days of release. Unsloth also ships MLX builds now (e.g. unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit).

Use MLX directly when you want maximum tok/s and you’re comfortable with the Python CLI:

pip install mlx-lm
mlx_lm.generate --model mlx-community/Qwen3.6-35B-A3B-4bit --prompt "Hello"

Ollama — the easy button (and “slow on Mac” is mostly false post-0.30)

Ollama is the simplest path: install once, ollama run <model>, done. The “Ollama is slow on Mac” critique you may have read was largely accurate against Ollama 0.18 and earlier on Apple Silicon, when Ollama ran exclusively through llama.cpp’s Metal backend.

That critique is mostly outdated as of mid-2026, because:

• Ollama 0.19 (March 31, 2026) swapped llama.cpp Metal for MLX as the inference engine on Apple Silicon for safetensors models, doubling decode speed on qualifying hardware (~58 → ~112 tok/s on representative benches, per the Ollama MLX preview announcement and Andrew Ooo’s review).
• Ollama 0.30 layered llama.cpp Metal back in alongside MLX and auto-routes by file format: GGUF files go through llama.cpp Metal, safetensors go through MLX. The user doesn’t have to pick; the right backend is selected per model. GGUF compatibility restored, MLX speed preserved for safetensors. Most of the historical Mac-side Ollama complaints don’t apply on 0.30.

Use Ollama when you want the simplest setup or you need an API server for other apps. For the full 0.30-specific install + setup picture, see the Ollama on Mac setup guide.

curl -fsSL https://ollama.com/install.sh | sh
ollama run qwen3.6

llama.cpp Metal — the engine underneath, useful directly

llama-server with Metal acceleration is what Ollama uses for GGUF on Apple Silicon. Most people don’t need to drive it directly, but two cases make it worth knowing:

• Brand-new models where MLX conversions haven’t landed yet. mlx-community usually catches up within a few days, but llama.cpp mainline (currently in the b9670+ range; see the releases page) often supports a fresh architecture before MLX does.
• Fine-grained flag control that Ollama doesn’t expose. Custom quants, specific grammar settings, MTP speculative decoding (now merged in mainline llama.cpp as of May 16, 2026), the --mmproj flag for split vision projector files.

./llama-server -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M --jinja -ngl 99

LM Studio — the GUI option with MLX backend

LM Studio (current 0.3.38+ releases) ships a native MLX engine on Apple Silicon for safetensors and uses llama.cpp Metal for GGUF, mirroring the Ollama 0.30 backend story but in a graphical interface. Use LM Studio when you prefer a desktop app over the terminal, want to browse models visually, or need fine control over sampling parameters without writing YAML configs. See the Ollama vs LM Studio comparison for the deeper breakdown.

When to pick which

For broader cross-platform runtime comparisons (vLLM, concurrent serving, the 24GB-3090-vs-Mac tradeoff), see llama.cpp vs Ollama vs vLLM (2026).

The Memory-Tier Rule of Thumb

The math, not the picks. Your model file should be no more than 60-70% of your total unified memory. That leaves room for macOS itself (2-3 GB), the KV cache (which grows with context length), framework overhead, and whatever else is running.

A 20 GB model on a 48 GB Mac is comfortable. A 20 GB model on a 24 GB Mac is on a knife’s edge — fits, but you’ll hit memory pressure as soon as context grows or another app demands RAM. If you’re at the limit, drop to a lower quant or a smaller model. A snappy 14B beats a swapping 32B every time.

Quick reference for the rule:

This is the math. For specific picks (which model name to install at each tier — Qwen 3.6-35B-A3B, Gemma 4 26B-A4B, DeepSeek V4-Flash, etc.), the Best Local LLMs for Mac 2026 page is the current reference. The VRAM Requirements guide and the VRAM Calculator cover the per-model math.

Performance Shape: Bandwidth-Strong, Compute-Weak

Mac local-AI performance has a specific shape, and naming that shape honestly is more useful than any specific benchmark number.

Token generation is bandwidth-bound. Mac wins for models that don’t fit on a consumer GPU.

Token generation throughput is dominated by how fast the runtime can stream model weights through memory. At ~400-800 GB/s (M-series Max / Ultra), Mac falls short of a discrete GPU’s bandwidth (RTX 3090 at ~936 GB/s, RTX 5090 at ~1,792 GB/s). For any model that fits in 24 GB of VRAM, a single discrete consumer GPU generates tokens faster than any Mac.

The crossover comes at the model size where a consumer GPU has to offload. A 70B Q4 model is ~40 GB — it doesn’t fit on a 24 GB GPU, so the GPU has to offload layers to system RAM over PCIe, which drops throughput dramatically (commonly to single-digit tok/s). A Mac with 64 GB+ unified memory loads the same 70B Q4 entirely in fast unified memory and runs it at the bandwidth limit of the chip — significantly faster than the offloaded GPU. Mac wins the 70B+ tier on consumer hardware, not by being faster per-byte but by avoiding the offload cliff.

Prompt processing is compute-bound. This is where M5’s Neural Accelerators change the picture.

Prompt processing (the prefill phase before the first generated token) is compute-bound, not bandwidth-bound. Mac has historically been slow here vs NVIDIA — long-context RAG, large code analysis, agentic tool calls with big system prompts all suffer.

The M5 Pro/Max generation changes this specifically. Neural Accelerators embedded in every GPU core (10 on base M5, 20 on M5 Pro, 40 on M5 Max) bring large speedups specifically on the compute-bound prefill phase. Apple’s own MLX research shows time-to-first-token improvements of 3.3-4× on base M5 vs M4 across a range of model sizes; community benches on Pro/Max indicate similar 4× scaling on prompt processing. Token generation gets a smaller bandwidth-tracking bump (~+28% on M5 Max per llmcheck.net community benches).

So the practical 2026 framing:

• For chat-style workloads (short prompts, lots of generated tokens), M-series Mac performance is bandwidth-tracked and improves linearly with bandwidth. Mac wins for big models, NVIDIA wins for models that fit on the GPU.
• For prompt-heavy workloads (RAG, code analysis, agentic harnesses with large context), M5 Pro/Max via MLX is a genuinely different category from M4 era. Pre-M5 Mac users feel the prefill penalty; M5 users mostly don’t.

The M5 Pro/Max for Local AI guide has the full architectural breakdown of why this shift happened. The llama.cpp vs Ollama vs vLLM guide covers the broader single-user-vs-concurrent dimension across hardware.

Mac Mini as a Headless AI Server

This is one of the best-kept secrets in local AI hardware. A Mac Mini configured as a headless always-on inference server is small, silent, sips power, and runs models that would need a multi-GPU PC. For home labs, family-sized self-hosted AI, agent backends, and 24/7 inference workloads, it’s the most cost-effective option on the market.

The qualities that make it work:

• Silent and small. Idle is near-fanless. Under sustained LLM load the fan spins up but stays in the “background noise” range. Sits in a media cabinet, on a shelf, anywhere.
• Low idle power. 5-15W idle, 30-60W under AI inference load. Annual electricity cost is dramatic vs a desktop PC with a 350W GPU.
• Unified memory advantage applies the same way. A Mac Mini M4 Pro 48GB runs the same 27B / 35B-A3B MoE models a Mac Studio does, just slower on the bandwidth-tracked metrics.
• macOS reliability. Set it up, leave it running. macOS uptime is well-suited to “headless box in the corner” use.

Setup as a headless server

1. Enable Remote Login. System Settings → General → Sharing → Remote Login (SSH).
2. Install Ollama.

   curl -fsSL https://ollama.com/install.sh | sh
   

3. Make Ollama listen on the network (default is localhost only):

   launchctl setenv OLLAMA_HOST "0.0.0.0:11434"
   

4. Prevent sleep. System Settings → Energy → Prevent automatic sleeping (and disable “Put hard disks to sleep when possible” if you have an external SSD).
5. Optionally: lock-screen the display and forget it.

Hit it from any device on your network:

curl http://mac-mini-ip:11434/api/generate -d '{"model":"qwen3.6", "prompt":"Hello"}'

For the deeper Ollama-on-Mac setup specifics including TLS, reverse proxies, and the 0.30-specific configuration, see the Ollama Mac setup guide.

Power consumption — why this matters for 24/7 use

Compare to a PC with an RTX 3090 idle-to-full-load (50-350W): roughly $50-400/year depending on duty cycle. The Mac Mini’s ability to run 24/7 for $35 a year electricity is one of the strongest practical arguments for it as a home-lab AI server.

For configuration recommendations and which Mac Mini SKU fits which workload, the Best Local LLMs for Mac 2026 “Best Mac for Local AI 2026” section is the current reference.

Metal Acceleration: Verify and Troubleshoot

Metal is Apple’s GPU framework, the equivalent of CUDA on NVIDIA. Every Apple Silicon Mac supports Metal, and every local LLM tool on Mac uses it automatically when correctly configured. You generally don’t have to enable anything. But you should verify it’s working before trusting performance numbers.

Verifying Metal is being used

In Ollama:

ollama ps

The Processor column should show GPU, not CPU. If it says CPU, something is misconfigured.

In LM Studio: The bottom status bar shows GPU utilization while a model is loaded and running.

In Activity Monitor: Window → GPU History. You should see activity spikes when generating tokens. If GPU History stays flat during generation, Metal isn’t being engaged.

When Metal doesn’t work

The common failure modes:

• macOS version too old. Metal for LLM inference requires macOS 12.6+ on M1, macOS 13.3+ on M2/M3/M4, and current macOS on M5. Update if you’re behind.
• Tool version too old. Older Ollama and LM Studio versions had narrower Metal compat. Update both. Ollama 0.30.x is the current floor for the auto-routing behavior covered above.
• Memory pressure pushing to swap. If macOS is actively swapping, performance collapses regardless of Metal status. Activity Monitor → Memory → Memory Pressure should be green. If it’s yellow or red, close apps before troubleshooting further.
• Wrong backend selected for the model format. On Ollama 0.30+ this is auto-routed (GGUF → llama.cpp Metal, safetensors → MLX), but on older versions or with manual MLX setups, you may have to match backend to format.

For broader Ollama-specific issues across platforms, the Ollama Troubleshooting Guide covers the cross-platform problem set.

Troubleshooting

The Mac-specific problems people actually hit.

“Model won’t load (out of memory)”

The model file plus context plus macOS is more than your unified memory. Fixes in order of effort:

1. Close memory-hungry apps (Chrome, VS Code, Slack — Electron apps cost more than you’d expect).
2. Use a lower quantization (Q4_K_M instead of Q6, Q3_K_M instead of Q4 if quality permits).
3. Reduce context length: ollama run <model> /set parameter num_ctx 4096. 4K context is plenty for most chat; 8K+ only when you need it.
4. Use a smaller model. A 14B at Q6 often beats a 32B at Q2 on real tasks.
5. If you’re hitting this repeatedly, your Mac’s memory tier may not fit your workload — see the memory-tier rule above.

“Painfully slow generation”

If you’re getting <10 tok/s on a small model that should be fast, Metal probably isn’t engaged. Check ollama ps for GPU vs CPU. If it’s on CPU:

• macOS too old → update.
• Tool misconfigured → reinstall.
• Heavy memory pressure → close apps. macOS deprioritizes GPU work when squeezed.

If Metal is on and it’s still slow, your model file is probably too large for your bandwidth class. A 70B model on an M4 Pro (273 GB/s) is bandwidth-bound to single-digit tok/s no matter what runtime you use — that’s not a misconfiguration, that’s the chip.

“Garbled or wrong output”

Usually a corrupted model download. Re-pull:

ollama rm <model>
ollama pull <model>

If the problem persists across re-pulls, the model file might be using a quant your runtime doesn’t fully support yet (rare for mainstream Qwen / Llama / Gemma; more common for brand-new architectures). Try a different quant variant from a different uploader (unsloth, bartowski, mradermacher).

“Prompt processing takes forever on long context”

If you’re feeding 8K+ token prompts on M1-M4 hardware, prefill is going to feel slow. That’s the compute-bound part of Mac LLM inference, where Mac has historically been weakest vs NVIDIA. Mitigations:

• M5 Pro/Max upgrades address this category specifically — Neural Accelerators speed prefill by 3-4× per Apple MLX research.
• Use MLX directly for prompt-heavy workloads where supported (MLX makes better use of compute resources than llama.cpp Metal does on prefill).
• For genuinely long contexts (32K+), consider whether RAG or chunked summarization could replace single-prompt processing.

For broader cross-platform troubleshooting, see Local AI Troubleshooting.

What This Guide Doesn’t Cover (and Where to Go Instead)

Three things deliberately routed elsewhere in the Mac cluster:

• Which specific model to run on your Mac tier. → Best Local LLMs for Mac 2026. Current picks across every memory tier from 8 GB through 192 GB Ultra, with the 2026 model lineup (Qwen 3.6-27B / 35B-A3B, Gemma 4 26B-A4B / 31B, Llama 4 Scout, DeepSeek V4-Flash).
• Whether to buy an M5 Pro or M5 Max (and upgrade-from-what-you-have framing). → Apple M5 Pro/Max for Local AI (2026). Neural Accelerator architecture, the 4× prompt-processing claim breakdown, M5 vs M4 upgrade math, M5 Ultra Mac Studio delay.
• Mac vs PC at a category level. → Mac vs PC for Local AI. The broader buying-decision framing across platforms.

For specific model family deep dives:

• Qwen 3.5 Local Guide — the open workhorse Qwen lineup
• Qwen 3.6 Local Guide — the newer-open-tier 27B and 35B-A3B
• Qwen Models Guide — the family across versions
• Run Your First Local LLM — the absolute beginner path

The Bottom Line

Apple Silicon Macs are genuinely good at local AI, but the picture has nuance you only see once you understand unified memory’s mechanics. Mac wins for any model that doesn’t fit on a consumer discrete GPU’s VRAM. Discrete consumer GPUs win for anything that fits in 24 GB at usable quant. The crossover is sharp, not gradual.

The mid-2026 runtime story is more boring than it used to be in a good way: install Ollama 0.30+, run a model, the right backend is selected for you. If you want to dig deeper, MLX directly is the consensus speed pick on supported models. If you want a GUI, LM Studio. If you want flag-level control or a brand-new architecture that hasn’t hit MLX yet, llama.cpp Metal directly.

The Mac Mini as a 24/7 headless inference server may be the most underrated single use case in this space. Silent, small, $35/year electricity, runs Qwen 3.6-35B-A3B-class MoE models on 48 GB unified memory all day. If you’ve been considering a dedicated home AI box, that’s it.

# The 30-second Mac AI workstation:
curl -fsSL https://ollama.com/install.sh | sh
ollama run qwen3.6

For what to actually run, see Best Local LLMs for Mac 2026. For M5 Pro/Max hardware analysis, see the M5 guide. For the cross-platform runtime decision, see llama.cpp vs Ollama vs vLLM.

Related Guides

• Best Local LLMs for Mac (2026) — current model picks by RAM tier
• Apple M5 Pro/Max for Local AI (2026) — M5 architecture + upgrade decision
• Ollama on Mac (0.30 setup) — Mac-specific Ollama install + tuning
• llama.cpp vs Ollama vs vLLM (2026) — cross-platform runtime decision
• Ollama vs LM Studio — GUI vs CLI Mac runtime
• Ollama Troubleshooting
• Local AI Troubleshooting Guide
• Run Your First Local LLM — beginner path
• Mac vs PC for Local AI
• VRAM Requirements for Local LLMs
• Local AI Planning Tool — VRAM Calculator
• Qwen 3.5 Local Guide
• Qwen 3.6 Local Guide

Get notified when we publish new guides.