VOOZH about

URL: https://willitrunai.com/gpus/h20-96gb?gpus=8

⇱ AI Models for NVIDIA H20 96GB — What Runs on 96GB VRAM

NVIDIA

NVIDIA H20 96GB

Hopper DatacenterDatacenterHopperSXMCUDA

Operating mode

Choose the operating mode for this hardware

Use this to bias workload recommendations toward responsiveness, background autonomy, lighter serving, or multi-GPU scale-out.

Current mode

Balanced

Balanced for general local use. Keeps the ranking neutral across personal and serving workflows.

See Full AI Tier List for NVIDIA H20 96GB →

About this GPU for AI

The NVIDIA H20 is a China-export-compliant Hopper GPU purpose-built for LLM inference in the Chinese market. Its 96 GB of HBM3 at 4.0 TB/s significantly exceeds the H100's 80 GB, while compute is throttled to 148 TFLOPS FP16 — roughly 15% of the H100 — to stay under U.S. export performance thresholds. Despite the compute cap, inference for memory-bandwidth-bound LLM workloads is competitive: the H20 delivers up to 20% higher peak tokens-per-second than H100 at low batch sizes due to its superior memory capacity and bandwidth. Chinese cloud providers built massive H20 clusters for LLM serving workloads.

Beyond LLMs

AI Capability Matrix

What AI tasks this GPU can handle — from text generation to image and video creation.

CapabilityStatusRepresentative ModelDetail
LLM Chat (7B)Runs nativelyLlama 3.1 8B Q4—
LLM Coding (30B)Runs nativelyQwen 3 30B Q4—
LLM Large (70B)
hbm-memorymassive-vramexport-regulatedinference-optimized

Specifications

Compute
FP16148 TFLOPS
INT8296 TOPS
ArchitectureHopper
Memory
VRAM96 GB
Bandwidth4000 GB/s
General
FamilyHopper Datacenter
SegmentDatacenter
InterconnectSXM
Compute PlatformCUDA
MSRP$12,000

Key Features

96 GB HBM3 — 4,000 GB/s bandwidth148 TFLOPS FP16 (throttled for export compliance) / 296 INT8 TOPSFull Hopper MIG support: up to 7 isolated instancesNVLink 4.0 (900 GB/s) — multi-GPU scaling retained400W TDP (vs. H100's 700W)Export-regulated: designed for China market, now facing further export restrictions

For AI Workloads

Strengths
  • 96 GB HBM3 fits 70B models at FP16 with substantial KV cache — no quantization required for large inference
  • 4 TB/s bandwidth delivers faster token generation than H100 for memory-bound workloads
  • 400W TDP — significantly lower power draw than H100 for inference deployments
  • Full NVLink 4.0 retained for multi-GPU inference scaling
Considerations
  • Compute throttled to ~15% of H100 — training runs are dramatically slower
  • Subject to ongoing U.S. export restrictions; availability outside China is minimal
  • Not available on Western cloud providers; primarily accessible via Chinese cloud platforms
  • The export regulatory environment around H20 shipments remains active and uncertain

Architecture

Hopper

Hopper is NVIDIA's datacenter-focused architecture succeeding Ampere. Built on TSMC 4N, it introduces the Transformer Engine with automatic FP8/FP16 mixed-precision training, HBM3/HBM3e memory, and NVLink 4.0 for multi-GPU scaling. The H100 flagship delivers up to 3x the AI training performance of A100.

AI Relevance

The Transformer Engine automatically manages FP8 precision for optimal training speed without accuracy loss. With up to 141 GB HBM3e (H200), Hopper GPUs can hold the largest open-weight models entirely in GPU memory, making them the workhorse of AI datacenters.

Process: TSMC 4NPlatform: CUDATensor Cores: Gen 4Precisions: FP64, FP32, TF32, FP16, BF16, FP8, INT8

Recommendations by Workload

Chat

S

Qwen 3.5 27B

Qwen 3.5 27B matches Chat and keeps a practical fit profile. It is a recent-generation family, which helps on current local SOTA workloads. It fits natively with comfortable headroom. Context coverage stays within the requested workload envelope. Known distribution channels: huggingface, ollama, lm-studio.

Decode 133.2 tok/s · 131K ctx · llama.cppEST.
41.0 GB / 96.0 GB VRAM

Coding

S

Qwen3-Coder-Next

Qwen3-Coder-Next is a specialized fit for Coding. It is a recent-generation family, which helps on current local SOTA workloads. It fits natively with comfortable headroom. Context coverage stays within the requested workload envelope. Known distribution channels: huggingface, ollama, lm-studio.

Decode 170.9 tok/s · 217K ctx · llama.cppEST.
77.6 GB / 96.0 GB VRAM

Agentic Coding

Full Model Compatibility

👁 Alibaba
Qwen 3.5 122B A10B
S96
122B87.4 GB130 tok/s73K ctx
moe
👁 Alibaba
Qwen3-Coder-Next
S95
80B60.8 GB219 tok/s256K ctx
moe
👁 Alibaba
Qwen 2.5 VL 72B
S95

Just out of reach

Models you could run with an upgrade

High-quality models that need a bit more memory

Image & Video Generation

Diffusion Model Compatibility

51 of 52 models can generate images or video on your NVIDIA H20 96GB

ModelMax ResolutionGen TimeGrade
SD TurboImage512×512300msS
Stable Diffusion 1.5Image512×768500msS
Realistic Vision v5.1Image512×768500msS
DreamShaper 8Image512×768500msS
LCM DreamShaper v7

Multi-GPU scaling

NVIDIA H20 96GB — Up to 8× via NVLink

Scale out with multiple GPUs for larger models. PCIe interconnect with 22% scaling overhead.

ConfigEffective memoryModels that fitEst. bandwidth
1× NVIDIA96 GB351/3744,000 GB/s
2× NVIDIA192 GB359/3746,240 GB/s
4× NVIDIA384 GB366/37412,480 GB/s
8× NVIDIA768 GB373/37424,960 GB/s

Model counts use default quantization at coding workload settings. Multi-GPU scaling factor: 0.78× per additional GPU.

Upgrade paths

Upgrade from NVIDIA H20 96GB

See what you unlock with more powerful hardware

Upgrade options

Upgrade options

👁 NVIDIA
8× NVIDIA H20 96GBMulti-GPU
8 × 96 GB = 768 GB effectivevia NVLink
A
Unlocks 22 additional models that do not fit on the current setup.Unlocks Qwen 3.5 397B A17B, Kimi K2.5, Kimi K2.6+19 more · +111% faster avg

Unlocks 22 additional models that do not fit on the current setup.

Lifts average decode speed across fitting models by about 111%.

NVLink gives this scale-out path a cleaner inter-GPU story than PCIe-only builds.

~$12,000 MSRP

👁 NVIDIA
NVIDIA H200 141GBNext step up
141 GB VRAM (+45)4800 GB/s (+800)
B
Unlocks 2 additional models that do not fit on the current setup.Unlocks Qwen 3 235B A22B, MiniMax M2.7+10% faster avg

Unlocks 2 additional models that do not fit on the current setup.

~$30,000 MSRP

👁 NVIDIA
NVIDIA B200 180GBNVIDIA upgrade
180 GB VRAM (+84)8000 GB/s (+4000)
B
Unlocks 8 additional models that do not fit on the current setup.Unlocks DeepSeek V4 Flash, Qwen 3 235B A22B, MiniMax M2.7+5 more · +34% faster avg

Unlocks 8 additional models that do not fit on the current setup.

Lifts average decode speed across fitting models by about 34%.

~$30,000 MSRP

AMD Instinct MI325X 256GBBiggest leap
256 GB VRAM (+160)6000 GB/s (+2000)
B
Unlocks 12 additional models that do not fit on the current setup.Unlocks Qwen 3.5 397B A17B, DeepSeek V4 Flash, Qwen 3 235B A22B+9 more · +12% faster avg

Unlocks 12 additional models that do not fit on the current setup.

Lifts average decode speed across fitting models by about 12%.

~$20,000 MSRP

AMD Instinct MI350X 288GBBest value
288 GB VRAM (+192)8000 GB/s (+4000)
B
Unlocks 13 additional models that do not fit on the current setup.Unlocks Qwen 3.5 397B A17B, DeepSeek V4 Flash, Qwen 3 235B A22B+10 more · +25% faster avg

Unlocks 13 additional models that do not fit on the current setup.

Lifts average decode speed across fitting models by about 25%.

~$8,000 MSRP

Frequently Asked Questions

Compare with similar

NVIDIA H20 96GB vs RTX PRO 6000 Blackwell Workstation Edition 96GBNVIDIA H20 96GB vs RTX PRO 6000 Blackwell Server Edition 96GBNVIDIA H20 96GB vs NVIDIA GH200 96GB
Compare this GPUCompare with another GPU →
96
GB
VRAM
4kGB/s
Bandwidth
148TFLOPS
FP16 Compute
296TOPS
INT8 Inference
$12,000 MSRP
NVIDIA H20 96GBCategory AvgNVIDIA H200 141GB
Runs natively
Llama 3.1 70B Q4
—
Image Gen (SDXL)Runs nativelySDXL 1.0 FP16~~2.1s per image
Image Gen (Flux)Runs nativelyFlux.1 Dev FP16~~9.4s per image
Image Gen (SD 3.5)Runs nativelySD 3.5 Large FP16~~11.4s per image
Video Short (25f)Runs nativelyLTX Video 2B~~1.8s/frame
Video Long (100f)Tight fitWan Video 14B~~5.3s/frame
S

Qwen3-Coder-Next

Qwen3-Coder-Next is a specialized fit for Agentic Coding. It is a recent-generation family, which helps on current local SOTA workloads. It fits natively with comfortable headroom. Context coverage stays within the requested workload envelope. Known distribution channels: huggingface, ollama, lm-studio.

Decode 218.8 tok/s · 256K ctx · llama.cppEST.
62.2 GB / 96.0 GB VRAM

Reasoning

S

Qwen3-Coder-Next

Qwen3-Coder-Next matches Reasoning and keeps a practical fit profile. It is a recent-generation family, which helps on current local SOTA workloads. It fits natively with comfortable headroom. Context coverage stays within the requested workload envelope. Known distribution channels: huggingface, ollama, lm-studio.

Decode 170.9 tok/s · 217K ctx · llama.cppEST.
77.6 GB / 96.0 GB VRAM

RAG

S

Qwen 3.5 27B

Qwen 3.5 27B matches RAG and keeps a practical fit profile. It is a recent-generation family, which helps on current local SOTA workloads. It fits natively with comfortable headroom. Context coverage stays within the requested workload envelope. Known distribution channels: huggingface, ollama, lm-studio.

Decode 133.2 tok/s · 131K ctx · llama.cppEST.
45.7 GB / 96.0 GB VRAM
72B59.3 GB80 tok/s33K ctx
dense
👁 Mistral
Devstral 2 123B Instruct
S95
123B90.9 GB47 tok/s31K ctx
dense
👁 Mistral
Mistral Small 4 119B
S94
119B88.5 GB141 tok/s38K ctx
moe
👁 Cohere
Command A 111B
S92
111B82.1 GB52 tok/s73K ctx
dense
👁 OpenAI
GPT-OSS 120B
S92
117B86.8 GB49 tok/s46K ctx
dense
👁 Alibaba
Qwen 3.6 35B A3B
S92
35B36.0 GB412 tok/s250K ctx
+1moe
👁 Alibaba
Qwen3-Coder 30B A3B Instruct
S92
30.5B30.6 GB490 tok/s256K ctx
moe
👁 Mistral AI
Pixtral Large 124B
S91
124B91.5 GB47 tok/s29K ctx
dense
👁 Alibaba
Qwen 3.5 27B
S91
27B30.1 GB213 tok/s131K ctx
dense
👁 Alibaba
Qwen3-VL 30B A3B Instruct
S91
30B30.3 GB507 tok/s256K ctx
moe
👁 Alibaba
Qwen 3.5 35B A3B
S91
35B33.3 GB448 tok/s131K ctx
moe
👁 Alibaba
Qwen 3.6 27B
S91
27B27.9 GB132 tok/s262K ctx
+1dense
👁 Alibaba
Qwen 3 32B
S90
32B33.9 GB181 tok/s131K ctx
dense
👁 Mistral
Leanstral 119B A6B
S90
119B91.9 GB130 tok/s24K ctx
moe
👁 Mistral
Magistral Small 2507
S89
24B27.6 GB238 tok/s131K ctx
dense
👁 Mistral
Devstral Small 2 24B Instruct
S89
24B27.6 GB238 tok/s256K ctx
dense
👁 Alibaba
Qwen 3 30B A3B
S89
30.5B30.6 GB490 tok/s131K ctx
moe
👁 NVIDIA
Nemotron 3 Nano 30B
S89
30B31.2 GB190 tok/s131K ctx
dense
👁 Google
Gemma 4 31B
S88
30.7B43.9 GB113 tok/s73K ctx
dense
👁 Mistral
Devstral Small 1.1
S88
24B27.6 GB238 tok/s131K ctx
dense
👁 Alibaba
Qwen 3.5 9B
S88
9B18.2 GB126 tok/s131K ctx
dense
👁 Alibaba
Qwen 3 14B
S87
14B21.5 GB196 tok/s131K ctx
dense
👁 NVIDIA
Nemotron Cascade 2 30B A3B
S87
30B31.7 GB501 tok/s262K ctx
moe
👁 Microsoft
Phi-4-reasoning-plus 14B
S87
14.7B22.5 GB206 tok/s33K ctx
dense
👁 OpenAI
GPT-OSS 20B
S86
21B25.8 GB622 tok/s128K ctx
moe
👁 Alibaba
Qwen 3 8B
S86
8B17.6 GB112 tok/s131K ctx
dense
👁 LG AI
EXAONE 4.0 32B
A84
32B33.9 GB179 tok/s131K ctx
dense
👁 Alibaba
Qwen 3.5 4B
A84
4B15.1 GB56 tok/s131K ctx
dense
👁 Google
Gemma 4 26B A4B
A83
25.2B29.5 GB526 tok/s256K ctx
moe
👁 Mistral
Ministral 3 14B
A82
14B21.5 GB196 tok/s262K ctx
multimodal
👁 NVIDIA
Nemotron Nano 8B
A81
8B17.3 GB112 tok/s131K ctx
dense
👁 Microsoft
Phi-4 Mini Reasoning 4B
A80
3.8B14.3 GB53 tok/s131K ctx
dense
👁 Jina AI
Jina Embeddings v3
A74
0.57B13.6 GB8 tok/s8K ctx
dense
👁 BAAI
BGE M3
A73
0.57B12.8 GB8 tok/s8K ctx
dense
👁 Alibaba
Qwen 3.5 397B A17B
F0
397B255.5 GB7 tok/s4K ctx
moe
👁 Moonshot AI
Kimi K2.5
F0
1000B627.9 GB3 tok/s4K ctx
moe
👁 Moonshot AI
Kimi K2.6
F0
1000B627.9 GB3 tok/s4K ctx
+1moe
👁 DeepSeek
DeepSeek V4 Pro
F0
1600B874.4 GB2 tok/s4K ctx
moe
👁 DeepSeek
DeepSeek V4 Flash
F0
284B169.8 GB23 tok/s4K ctx
moe
👁 Z.ai
GLM-5.1
F0
754B489.5 GB3 tok/s4K ctx
moe
👁 Z.ai
GLM-5
F0
744B483.4 GB3 tok/s4K ctx
moe
👁 DeepSeek
DeepSeek V3.2
F0
671B420.3 GB4 tok/s4K ctx
moe
👁 Alibaba
Qwen 3 235B A22B
F0
235B156.7 GB25 tok/s4K ctx
moe
👁 Alibaba
Qwen3-Coder 480B A35B Instruct
F0
480B306.2 GB5 tok/s4K ctx
moe
MiniMax M2.7
F0
230B154.6 GB29 tok/s4K ctx
moe
👁 DeepSeek
DeepSeek Coder V2 236B
F0
236B213.1 GB14 tok/s4K ctx
moe
👁 DeepSeek
DeepSeek R1 671B
F0
671B479.4 GB4 tok/s4K ctx
moe
👁 DeepSeek
DeepSeek V3.1 671B
F0
671B479.4 GB4 tok/s4K ctx
moe
Image
512×768
200ms
S
PixArt-SigmaImage1024×1024~2.1sS
FramePack I2VVideo1280×720~3.8s/frameS
SDXL TurboImage512×512300msS
SDXL LightningImage1024×1024800msS
Stable Diffusion XL 1.0Image1024×1024~2.1sS
Playground v2.5Image1024×1024~3.1sS
RealVisXL v5.0Image1024×1024~2.3sS
DreamShaper XLImage1024×1024~2.3sS
Juggernaut XL v9Image1024×1024~2.3sS
Animagine XL 3.1Image1024×1024~2.3sS
Pony Diffusion V6 XLImage1024×1024~2.3sS
Animagine XL 4.0Image1024×1024~2.3sS
Illustrious XLImage1024×1024~2.3sS
Wan Video 2.1 1.3BVideo480×832~1.5s/frameS
Stable Diffusion 3.5 MediumImage1024×1024~3.6sS
Flux.2 Klein 4BImage1024×1024600msS
LTX Video 2BVideo1280×720~1.8s/frameS
KolorsImage1024×1024~4.2sS
Stable CascadeImage1024×1024~5.2sS
AuraFlow v0.3Image1536×1536~9.4sS
Stable Diffusion 3.5 LargeImage1024×1024~11.4sS
Stable Diffusion 3.5 Large TurboImage1024×1024~2.1sS
CogVideoX 2BVideo720×480~1.8s/frameS
HunyuanVideoVideo720×1280~3.8s/frameS
ChromaImage1024×1024~2.1sS
Z-Image TurboImage1536×1536~2.1sS
Flux.1 DevImage1024×1024~9.4sS
Flux.1 SchnellImage1024×1024~1.8sS
LTX Video 13BVideo1280×720~3.8s/frameS
Flux.1 Kontext DevImage1024×1024~10.4sS
AnimateDiff v1.5.3Video512×768900ms/frameS
Cosmos Diffusion 7BVideo1024×576~3s/frameS
CogVideoX 5BVideo720×480~2.6s/frameS
Wan2.2 TI2V 5BVideo832×480~2.6s/frameS
Flux.2 Klein 9BImage1024×1024~1sS
Flux.1 Fill DevImage1024×1024~8.8sS
Mochi 1 PreviewVideo848×480~3.4s/frameS
HunyuanVideo 1.5Video720×1280~3.2s/frameS
Helios 14BVideo1280×720~3.9s/frameS
SkyReels V2 14BVideo1280×720~3.9s/frameS
Wan Video 2.1 14BVideo720×1280~3.9s/frameS
Wan Video 2.2 14BVideo720×1280~3.9s/frameS
Qwen ImageImage1024×1024~3.5sS
Qwen Image EditImage1024×1024~3.5sS
Flux.2 DevImage1024×1024~1m 38sS
MAGI-1Video1280×720~4.9s/frameS
HunyuanImage 3.0Image256×256~6.2sF

Image models estimated at 1024×1024 (28 steps, FP16). Video models estimated at 768×512 (25 frames, 30 steps, FP16). Actual performance varies with runtime and system load.

Buying advice

Should you buy NVIDIA H20 96GB for local AI?

Excellent choice for local AI

Runs 36 of 50 top models well — a strong all-rounder for local inference.

96.0 GB

VRAM

$12,000

MSRP

$125/GB

Cost per GB VRAM

Best models for this GPU

What will limit you first

This setup is broadly balanced for this model.

No major red flags

This recommendation has enough memory headroom and acceptable estimated speed for the selected workload.

Best upgrade itinerary

Unlocks 2 additional models that do not fit on the current setup.

Want more headroom? NVIDIA H200 141GB (141.0 GB VRAM) is the next step up.