The Definitive GPU Ranking for LLMs: Token Generation & Prompt Processing Performance

At Hardware Corner, we set out to create a data-driven benchmark hierarchy for local LLM inference – focusing on the two workloads that define real-world performance: prompt processing and token generation. Using llama.cpp’s latest llama-bench on Ubuntu 24.04 with CUDA 12.8, we measured a wide range of GPUs across model sizes, context lengths, and quantization levels. The goal is simple: identify which GPUs deliver the best performance for local LLM users.

These benchmarks aren’t theoretical. Every number you’ll see here was captured on-hardware in our lab, using consistent configurations and repeatable testing. Whether you’re upgrading a single-GPU desktop or scaling a dual-GPU setup, this guide shows what actually matters for real inference workloads – and where your money gets you the most tokens per second.

GPU Token Generation Benchmarks

To measure real-world performance, we tested a range of consumer and professional GPUs with various open-weight models at different context lengths. All benchmarks were run on Ubuntu 24.04 using the latest version of llama.cpp’s llama-bench, with CUDA 12.8. We used 4-bit (Q4_K_XL) quantization for all of the benchmarks. The following data shows the token generation speeds we recorded.

👁 Scatter plot showing GPU token generation performance vs VRAM for 8B model at 16k context. Larger circles represent higher GPU prices, comparing models like RTX 5090, RTX 4090, RTX 4080, and RTX Pro 6000

Key Findings

• Best Budget GPU: The NVIDIA RTX 5060 Ti (16GB) offers the best price-to-performance for running capable models.
• Best Value for VRAM: A used NVIDIA RTX 3090 (24GB) is the top choice for a single-GPU setup, enabling larger 30B+ models at an unbeatable price-to-VRAM ratio.
• Best for Cheap Scaling: A dual RTX 5060 Ti setup is a smart, budget-friendly path to 32GB of VRAM for running 34B models.
• Top Consumer Performance: The RTX 5090 (32GB) is the new speed leader for consumer GPUs, significantly outperforming previous generations.
• For Massive Models: The RTX PRO 6000 (96GB) is the only viable option for running the largest 70B-120B models, where memory capacity is the main priority.

GPU Benchmark Comparison: Token Generation Speed for 8B–123B Models from 16K to 131K Context

This benchmark table compares GPUs based on their token generation speed (tokens per second) when running large language models (LLMs) at a 16K context length. Each column represents a model – from Qwen3 8B to Mistral Large 123B at 4-bit quantization – and each value indicates the GPU’s generation throughput.

16GB GPUs performance with LLMs

The 16GB VRAM tier represents a critical sweet spot for many budget-conscious enthusiasts. It unlocks the ability to run larger and more capable models, such as 7B models with long context or 20B quantized models, without requiring a massive investment. In this competitive bracket, the price-to-performance ratio is the key metric that separates the good deals from the expensive hardware.

The NVIDIA RTX 5060 Ti 16GB stands out as the top value pick for budget builders. Priced around $430, it delivers 51.41 t/s—well above the RTX 4060 Ti 16GB’s 34.31 t/s at a similar cost. While premium cards like the RTX 4080 SUPER boast higher raw performance, their $1,200+ price tags cripple their performance-per-dollar ratio. The 5060 Ti strikes the sweet spot: enough VRAM, solid speed, and an accessible price.

👁 Token generation speeds (tokens/sec) for 16 GB GPUs running 8B, 14B, and 20B LLMs at 16k context. Each bar shows performance across model sizes, highlighting how newer 50-series cards, especially the RTX 5060 Ti, deliver strong efficiency relative to older or higher-priced GPUs.

The RTX 5070 Ti 16GB offers impressive power, but at nearly double the cost ($812), its gains—43 t/s versus 66 t/s on a 20B model—don’t justify the jump for users limited to 16GB VRAM. It’s faster, yes, but not transformative, with benefits mainly in handling extremely long prompts.

👁 This mirror bar chart compares each GPU’s purchase price (left) against its cost efficiency (right). While high-end models like the RTX 4080 SUPER command steep prices, midrange cards such as the RTX 5060 Ti and 5070 Ti deliver far better value per token generation speed — highlighting the sweet spot for local LLM builders.

Where the 5060 Ti truly shines is in a dual-GPU setup. Two cards ($860 total) deliver 32GB VRAM that is 8GB more than a single RTX 3090 for roughly the same price. This combo enables running larger 34B models, making it a smart, scalable path for enthusiasts chasing maximum model size on a budget.

The 24GB Sweet Spot

The 24GB VRAM tier is the sweet spot for a versatile single-GPU LLM setup, and the used NVIDIA RTX 3090 reigns supreme. At around $800 second-hand, it offers unmatched VRAM capacity for the price. That extra memory is what enables running 30B–32B parameter models, something simply impossible on 16GB GPUs. For enthusiasts chasing maximum model size without breaking the bank, the 3090’s price-to-VRAM ratio is unbeatable.

In terms of speed, the RTX 3090 holds its own. Even though it was released in 2020, the RTX 3090 is still a flagship card that has massive 384-bit memory bus and fast 19.5 Gbps memory deliver an impressive 936.2 GB/s of bandwidth, which is essential for fast token generation. This, combined with its generous 24 GB of VRAM, makes it an excellent choice for local LLM workloads. It hits 87.45 t/s on 8B models, nearly identical to the newer RTX 5070 Ti.

👁 Comparison chart of NVIDIA RTX 3090, 3090 Ti, 4090, and 5090 GPUs showing LLM token generation performance and price-to-performance ratios. Highlights that the RTX 5090 leads in speed, while the RTX 3090 delivers the best cost-per-token performance for local LLM inference.

The trade-off is clear: slightly less speed in exchange for vastly greater model capacity. Even on heavy loads, like a 20B model with a 131k context, the 3090 still pushes over 62 t/s, proving both capable and efficient.

The real strength of the 3090’s 24GB lies in its flexibility. It can manage extended contexts, such as an 8B model at 131k or a 30B MoE at 65k, far beyond what 16GB cards can handle. For users who prioritize long-context performance and large-model capability, the used RTX 3090 remains the smartest and most balanced single-card solution available.

The 32GB VRAM Tier

Once you move beyond 24GB of VRAM, the focus shifts from performance-per-dollar to raw capability. This is the realm of high-end consumer and select prosumer GPUs, including upcoming models like the RTX Pro 4500 Blackwell and RTX 5000 Ada. We have currently tested only the RTX 5090, which serves as the new reference point for this tier.

Although we are commenting primarily on performance, it’s important to note that 32GB sits in a bit of an in-between segment. It won’t unlock entirely new models compared to 24GB GPUs; it will load the same models, but with longer contexts. The real question is whether you want to pay extra to run Qwen3 14B, Qwen3 30B A3B, or Qwen3 32B at extended contexts such as 131K, 147K, or 45K instead of being limited to 86K, 65K, or 16K. That additional VRAM enables deeper context windows and more complex inference runs rather than new model access.

👁 A horizontal bar chart comparing the performance of the NVIDIA RTX 3090 and RTX 5090 GPUs on various large language models, including Qwen3 30B, Qwen3 32B, and gpt-oss 20B. The comparison is shown across different context sizes: 16k, 32k, and 65k. The RTX 5090 consistently demonstrates higher performance than the RTX 3090 in all benchmarked configurations.

The RTX 5090 delivers top-tier performance for a consumer card. Built on the GB202 architecture^[1], it combines 32GB of ultra-fast GDDR7 memory with immense compute throughput. In our benchmarks, it reached 145 t/s on Qwen3 8B and maintained strong efficiency even on larger 30B and 32B models.

However, when directly comparing token generation speeds between the RTX 5090 and the RTX 3090 at the same context lengths, the difference is not dramatic for most workloads – the 3090 still performs very well. For instance, the RTX 3090 runs Qwen3 30B A3B MoE at 87 t/s on a 32K context, while the 5090 reaches 110 t/s. For gpt-oss 20B at 32K, the 3090 produces 112 t/s versus the 5090’s 215 t/s. The newer card is faster, but the 3090’s speeds remain entirely workable for most local inference tasks.

For users running models up to 34B, the RTX 5090 stands as the new performance leader and the most balanced choice before stepping into the inflated pricing of the prosumer segment.

The 48GB VRAM GPUs

The 48GB tier sits firmly in the prosumer category, where prices are higher but the jump in capability is substantial. This class unlocks the 70B models and includes the RTX 4090 48GB, RTX A6000, and RTX 6000 Ada desktop GPUs.

Among these, one product stands out for its uniqueness, the RTX 4090 48GB. This Chinese-modded variant repurposes standard RTX 4090 GPUs by combining functional chips with custom PCBs designed to accept double-sided memory configurations. The result is a 48GB version of the 4090 that offers exceptional value for its capacity, priced around $3,100.

In performance terms, it maintains excellent throughput, achieving 137 t/s on 30B models – just below the RTX 5090 but ahead of the RTX A6000 and RTX 6000 Ada in price-adjusted efficiency. For comparison, the RTX 6000 Ada posts 120 t/s, while the older A6000 lags further behind at 75 t/s. The 4090 48GB effectively bridges the gap between consumer and workstation hardware, delivering the cheapest path to 48GB without sacrificing much speed.

The 96GB VRAM Tier: Extreme Capacity for Large-Scale Models

At the top of the chart sits the RTX Pro 6000 96GB. This GPU represents the only viable option for users who need to load and run the largest LLMs ( 100B+ parameter range) without offloading to system memory or relying on multi-GPU setups.

👁 Bar chart comparing NVIDIA RTX 5090 and RTX Pro 6000 WS GPU token generation speeds across large language models from 8B to 123B parameters at 16k context. The RTX 5090 achieves higher throughput on smaller models, while the RTX Pro 6000 WS shows better performance scaling with larger LLMs.

In our testing, the Pro 6000 reached 28 t/s on Llama 70B and 182 t/s on gpt-oss 120B MoE workloads. Its purpose is clear: not to dominate smaller benchmarks, but to make massive models feasible for local inference. For professional creators, AI researchers, and organizations building in-house model infrastructure, it remains the single most capable single-GPU solution available.

GPU Prompt Processing Benchmarks

Prompt processing performance defines how quickly a GPU can ingest and prepare the context window before token generation begins. It’s the phase that often bottlenecks large models, especially with extended contexts or complex prompts. At Hardware Corner, we benchmarked the prompt processing (prefill) stage separately to highlight how GPU compute performance, architecture efficiency, and VRAM capacity influence throughput during model initialization and prefill.

👁 Scatter plot showing GPU performance vs VRAM for 8B model at 16k context. Larger circles represent higher GPU prices, comparing models like RTX 5090, RTX 4090, RTX 4080, and RTX Pro 6000.

These results reveal where each VRAM tier truly stands. If you’ve ever wondered why your system slows down at the start of an inference run, or which GPU tier delivers the best context-loading efficiency, this section has your answer.

Key Findings

• RTX 5060 Ti (16GB): Best entry-level value; handles 13B–20B models efficiently for its price.
• RTX 3090 (24GB): Top price-to-capacity pick; stable at long contexts up to 131k tokens.
• RTX 5090 (32GB): Fastest consumer GPU; near-pro throughput at far lower cost.

GPU Benchmark Comparison Table: Prompt Processing Speed for 8B – 123B Models form 16K – 131K Context

This benchmark table compares GPUs based on their token generation speed (tokens per second) when running large language models (LLMs) at a 16K context length. Each column represents a model – from Qwen3 8B to Mistral Large 123B at 4-bit quantization– and each value indicates the GPU’s processing throughput.

Prompt Processing Performance of 16GB GPUs

For local LLM enthusiasts, the 16GB VRAM tier marks a crucial step up, enabling 13B to 20B models that simply don’t fit on 8GB or 12GB cards. Here, “value” means reaching that VRAM capacity for the lowest cost possible. The RTX 5060 Ti 16GB stands out as the new entry-level benchmark, offering the memory needed to unlock larger models without a steep price tag.

While the RTX 4060 Ti 16GB once launched cheaper, it’s now harder to find and nearly the same price second-hand. Worse, NVIDIA’s limited 288 GB/s bandwidth cripples its performance, making it slower than even the RTX 3060 16GB for prompt processing. Overall, it’s a poor value next to newer alternatives like the 5060 Ti.

Across the 16GB landscape, diminishing returns are clear. Cards like the RTX 4080 or 4070 Ti SUPER deliver over twice the throughput of the 5060 Ti but cost two to three times more. For value-oriented builders, that premium is wasted on marginally faster prompt ingestion. The savings from a 5060 Ti build are better spent on system RAM, storage, or even a second GPU.

Every 16GB card (from the 5060 Ti to the 4080 SUPER) can load the same 14B model with a 32k context. The main difference lies in how fast they process that context. The RTX 5060 Ti 16GB strikes the best balance: affordable, capable, and practical. It’s the true workhorse of this VRAM tier, prioritizing accessible capacity over inflated performance premiums.

Prompt Processing Performance of 24GB GPUs

The 24GB tier is where single-GPU LLM setups become truly capable. It provides enough headroom for 30B–32B models, longer contexts, and smoother inference without resorting to multi-GPU complexity. At this level, the goal shifts from simply fitting a model to running it efficiently across large prompts and deep contexts.

The RTX 3090 defines this class. With 24GB of VRAM and a wide 384-bit bus, it balances capacity and throughput better than anything near its price. In tests with Qwen3 8B Q4_K at 16k context, it reached 2572 t/s in prompt processing. While newer cards like the RTX 4090 and 5090 are 2.6–2.7× faster, they cost over three times more, making them harder to justify for most local users.

The 3090’s real strength is stability at scale. It maintains solid speeds up to 65k context on 30B models and can stretch to 131k on 8B versions without significant drop-off. That bandwidth and VRAM combination keeps large prompt ingestion fast and prevents the stuttering common on lower-capacity cards.

For local inference, the RTX 3090 remains the practical benchmark. It’s not the newest or fastest, but it delivers where it matters, ample VRAM, steady prompt handling, and the best price-to-capacity value in the 24GB tier.

Prompt Processing Performance of 32GB+ GPUs

The 32GB+ VRAM tier represents the top end of local LLM performance, where raw capacity and bandwidth take priority over price. For now, this segment includes the RTX 5090 (32GB) and RTX PRO 6000 (96GB), as testing of upcoming 48GB GPUs is still in progress.

The RTX 5090 marks a major leap for consumer hardware. With 32GB of GDDR7 VRAM, it finally breaks past the long-standing 24GB limit. That extra memory allows dense 32B models to run with double the context (roughly 32k tokens) while maintaining excellent speed. In prompt processing, it sustains around 6,900–7,000 tokens per second on 8B models at 16k context, keeping pace with professional cards at a fraction of the cost. For most local users, the 5090 delivers unmatched performance-per-dollar and broad compatibility with large quantized models.

The RTX PRO 6000 (96GB) sits in a different class. It’s built for extreme workloads—models exceeding 120B parameters and contexts over 100k tokens. Its massive VRAM pool ensures stability and throughput even on the heaviest loads, reaching over 7,500 tokens per second on smaller benchmarks and maintaining smooth scaling at huge context lengths. However, at around $8,000, its price limits its appeal to developers or researchers who truly need the capacity.

In short, the RTX 5090 defines the peak of consumer LLM performance, while the PRO 6000 exists for those running models too large for anything else. Until the 48GB cards arrive, these two GPUs set the standard for what’s possible in the 32GB+ prompt processing class.

Conclusion

Speed is only part of the story. These benchmarks reveal clear performance tiers, but the right GPU depends entirely on how and what you plan to run. If your focus is fast, local chat or lightweight coding, a 16GB card like the RTX 5060 Ti offers remarkable efficiency and affordability. For deeper context handling or larger 30B–32B models, the 24GB tier(especially the RTX 3090) remains the most balanced and cost-effective single-GPU solution.

When performance and flexibility truly matter, the entry point for serious LLM work begins around 24GB of VRAM. At that level, you can comfortably run 30B-scale models with usable 45k-token contexts at solid generation speeds, coding, and content workflows. Beyond that, the RTX 5090 and its 32GB of GDDR7 set a new standard for consumer performance, while professional-grade cards like the RTX PRO 6000 (96GB) unlock the 70B–120B range with near-perfect prompt throughput and multi-user headroom.

Ultimately, choosing a GPU for LLMs isn’t just about tokens per second, it’s about matching the hardware to your workload. Whether you’re running a compact local assistant or a production-grade MoE system, the data makes one thing clear: speed benchmarks guide you, but context defines your real performance.