Voozh

Qwen3-Coder 30B A3B Instruct30.5B

Q4_K_M23.4 GB VRAM54.4 tok/sNeeds offload

SExcellent

Qwen 3.5 27B27B

Q4_K_M22.9 GB VRAM23.6 tok/sNeeds offload

SExcellent

Qwen 3.6 27B27B

Q4_K_M20.7 GB VRAM17.9 tok/sTight fit

SExcellent

Qwen3-VL 30B A3B Instruct30B

Q4_K_M23.1 GB VRAM56.3 tok/sNeeds offload

SExcellent

Qwen 3.5 9B9B

Q4_K_M11.0 GB VRAM70.5 tok/sRuns great

SExcellent

Magistral Small 250724B

Q4_K_M20.4 GB VRAM26.4 tok/sTight fit

SExcellent

Devstral Small 2 24B Instruct24B

Q4_K_M20.4 GB VRAM26.4 tok/sTight fit

SExcellent

Qwen 3 14B14B

Q4_K_M14.3 GB VRAM45.5 tok/sRuns great

SExcellent

Qwen 3 30B A3B30.5B

Q4_K_M23.4 GB VRAM54.4 tok/sNeeds offload

SExcellent

Nemotron 3 Nano 30B30B

Q4_K_M24.0 GB VRAM15.8 tok/sNeeds offload

SExcellent

Qwen 3.5 4B4B

Q4_K_M7.9 GB VRAM56.0 tok/sRuns great

SExcellent

Qwen 3 8B8B

Q4_K_M10.4 GB VRAM79.3 tok/sRuns great

SExcellent

Phi-4-reasoning-plus 14B14.699999809265137B

Q4_K_M15.3 GB VRAM43.2 tok/sRuns great

SExcellent

Devstral Small 1.124B

Q4_K_M20.4 GB VRAM26.4 tok/sTight fit

SExcellent

GPT-OSS 20B21B

Q4_K_M18.6 GB VRAM69.1 tok/sRuns great

SExcellent

Nemotron Cascade 2 30B A3B30B

Q4_K_M24.5 GB VRAM39.9 tok/sNeeds offload

SExcellent

Phi-4 Mini Reasoning 4B3.799999952316284B

Q4_K_M7.1 GB VRAM53.2 tok/sRuns great

AGreat

Jina Embeddings v30.5720000267028809B

F166.4 GB VRAM8.0 tok/sRuns great

AGreat

BGE M30.5680000185966492B

F165.6 GB VRAM8.0 tok/sRuns great

AGreat

Nemotron Nano 8B8B

Q4_K_M10.1 GB VRAM79.3 tok/sRuns great

SExcellent

Ministral 3 14B14B

Q4_K_M14.3 GB VRAM45.3 tok/sRuns great

SExcellent

Gemma 4 26B A4B25.200000762939453B

Q4_K_M22.3 GB VRAM58.5 tok/sTight fit

SExcellent

Codestral 2 25.0822B

Q4_K_M19.2 GB VRAM27.1 tok/sRuns great

SExcellent

mxbai Embed Large0.33500000834465027B

F165.5 GB VRAM4.7 tok/sRuns great

AGreat

Mistral Small 3.2 24B24B

Q4_K_M20.4 GB VRAM26.4 tok/sTight fit

AGreat

Snowflake Arctic Embed L0.33500000834465027B

F165.5 GB VRAM4.7 tok/sRuns great

AGreat

CogVLM2 19B19B

Q4_K_M17.3 GB VRAM33.4 tok/sRuns great

SExcellent

LFM2 24B24B

Q4_K_M20.4 GB VRAM26.4 tok/sTight fit

AGreat

Granite 4.1 30B30B

Q4_K_M25.5 GB VRAM14.0 tok/sNeeds offload

AGreat

Nomic Embed Text v1.50.13699999451637268B

F164.1 GB VRAM2.0 tok/sRuns great

BGood

InternVL2 8B8B

Q4_K_M10.1 GB VRAM79.3 tok/sRuns great

AGreat

Mistral Small 24B24B

Q4_K_M20.4 GB VRAM26.4 tok/sTight fit

AGreat

Phi-4 14B14B

Q4_K_M14.9 GB VRAM45.3 tok/sRuns great

AGreat

Qwen 3 4B4B

Q4_K_M7.9 GB VRAM56.0 tok/sRuns great

AGreat

BGE Large EN v1.50.33500000834465027B

F165.5 GB VRAM4.7 tok/sRuns great

BGood

Mistral Small 3.1 24B24B

Q4_K_M20.4 GB VRAM26.4 tok/sTight fit

AGreat

Qwen 2.5 14B14B

Q4_K_M14.8 GB VRAM45.5 tok/sRuns great

AGreat

Ministral 3 8B8B

Q4_K_M10.4 GB VRAM79.3 tok/sRuns great

AGreat

MiniCPM-V 2.6 8B8B

Q4_K_M10.1 GB VRAM79.3 tok/sRuns great

AGreat

Gemma 3 12B12B

Q4_K_M15.5 GB VRAM39.1 tok/sRuns great

AGreat

Qwen 2.5 VL 7B7B

Q4_K_M8.4 GB VRAM91.5 tok/sRuns great

AGreat

Granite Code 20B20B

Q4_K_M18.7 GB VRAM31.9 tok/sRuns great

AGreat

SQLCoder 7B7B

Q4_K_M9.5 GB VRAM90.6 tok/sRuns great

AGreat

Nemotron Nano 9B v29B

Q4_K_M11.2 GB VRAM70.5 tok/sRuns great

AGreat

DeepSeek Coder V2 16B16B

Q4_K_M16.4 GB VRAM87.8 tok/sRuns great

AGreat

Magistral 7B7B

Q4_K_M9.5 GB VRAM90.6 tok/sRuns great

AGreat

CodeGeeX 4 9B9B

Q4_K_M9.4 GB VRAM71.7 tok/sRuns great

AGreat

Gemma 4 E4B8B

Q4_K_M9.5 GB VRAM60.1 tok/sRuns great

AGreat

OLMo 2 13B13B

Q4_K_M13.7 GB VRAM49.0 tok/sRuns great

AGreat

All MiniLM L6 v20.023000000044703484B

F163.6 GB VRAM2.0 tok/sRuns great

BGood

Qwen 2.5 7B7B

Q4_K_M8.4 GB VRAM91.5 tok/sRuns great

AGreat

Qwen 2.5 Coder 3B3B

Q4_K_M7.3 GB VRAM42.0 tok/sRuns great

AGreat

CodeLlama 13B Instruct13B

Q4_K_M23.4 GB VRAM45.4 tok/sNeeds offload

AGreat

DevStral 7B7B

Q4_K_M9.5 GB VRAM90.6 tok/sRuns great

AGreat

Codestral Mamba 7B7B

Q4_K_M8.1 GB VRAM97.0 tok/sRuns great

AGreat

Granite Code 8B8B

Q4_K_M10.1 GB VRAM79.3 tok/sRuns great

AGreat

DeepSeek R1 Distill 14B14B

Q4_K_M14.8 GB VRAM45.5 tok/sRuns great

AGreat

Granite 4.1 8B8B

Q4_K_M10.6 GB VRAM79.3 tok/sRuns great

AGreat

Pixtral 12B12B

Q4_K_M13.1 GB VRAM52.9 tok/sRuns great

AGreat

StarCoder 7B7B

Q4_K_M14.9 GB VRAM84.3 tok/sRuns great

AGreat

CodeLlama 7B Instruct7B

Q4_K_M15.4 GB VRAM84.3 tok/sRuns great

AGreat

LLaVA 1.6 13B13B

Q4_K_M23.4 GB VRAM45.4 tok/sNeeds offload

AGreat

Gemma 4 E2B5.099999904632568B

Q4_K_M6.9 GB VRAM71.4 tok/sRuns great

AGreat

Ministral 3 3B3B

Q4_K_M5.9 GB VRAM42.0 tok/sRuns great

BGood

Qwen 3.5 2B2B

Q4_K_M6.2 GB VRAM28.0 tok/sRuns great

BGood

GLM-4 9B9B

Q4_K_M9.4 GB VRAM71.7 tok/sRuns great

AGreat

Gemma 3 4B4B

Q4_K_M7.8 GB VRAM56.0 tok/sRuns great

BGood

Llama 3.1 8B8B

Q4_K_M10.1 GB VRAM79.3 tok/sRuns great

AGreat

WizardLM 13B13B

Q4_K_M23.4 GB VRAM45.4 tok/sNeeds offload

AGreat

InternLM 7B7B

Q4_K_M15.4 GB VRAM84.3 tok/sRuns great

AGreat

InternLM Chat 7B7B

Q4_K_M15.4 GB VRAM84.3 tok/sRuns great

AGreat

Solar 7B7B

Q4_K_M10.5 GB VRAM90.6 tok/sRuns great

AGreat

Nous Hermes 1.09B

Q4_K_M21.0 GB VRAM65.6 tok/sTight fit

AGreat

Nous Dolphin 13B13B

Q5_K_M24.9 GB VRAM27.3 tok/sNeeds offload

AGreat

Vicuna 13B13B

Q4_K_M23.4 GB VRAM45.4 tok/sNeeds offload

AGreat

WizardMath 7B7B

Q4_K_M9.5 GB VRAM90.6 tok/sRuns great

AGreat

OLMo 2 7B7B

Q4_K_M9.5 GB VRAM90.6 tok/sRuns great

AGreat

Phi 4 Mini 4B4B

Q4_K_M7.2 GB VRAM56.0 tok/sRuns great

BGood

Qwen 2.5 Coder 7B7B

Q4_K_M8.4 GB VRAM91.5 tok/sRuns great

BGood

Qwen 3 1.7B1.7000000476837158B

Q4_K_M6.0 GB VRAM23.8 tok/sRuns great

BGood

Qwen 2.5 Coder 1.5B1.5B

Q4_K_M4.6 GB VRAM21.0 tok/sRuns great

BGood

Qwen 2.5 3B3B

Q4_K_M7.3 GB VRAM42.0 tok/sRuns great

BGood

LLaVA 1.5 7B7B

Q4_K_M15.4 GB VRAM84.3 tok/sRuns great

AGreat

Granite 4.1 3B3B

Q4_K_M6.4 GB VRAM42.0 tok/sRuns great

BGood

Phi 3 Mini 3.8B3.799999952316284B

Q4_K_M11.5 GB VRAM53.2 tok/sRuns great

BGood

DeepSeek R1 Distill 7B7B

Q4_K_M8.4 GB VRAM91.5 tok/sRuns great

BGood

DeepSeek R1 Distill 8B8B

Q4_K_M10.1 GB VRAM79.3 tok/sRuns great

BGood

Falcon 7B Instruct7B

Q4_K_M7.7 GB VRAM92.6 tok/sRuns great

BGood

MPT-7B-Instruct7B

Q4_K_M15.4 GB VRAM84.3 tok/sRuns great

AGreat

Baichuan 7B7B

Q4_K_M15.4 GB VRAM84.3 tok/sRuns great

AGreat

Baichuan 13B13B

Q5_K_M24.9 GB VRAM27.3 tok/sNeeds offload

BGood

Cerebras-GPT 13B13B

Q5_K_M22.4 GB VRAM39.2 tok/sTight fit

BGood

Samantha 7B7B

Q4_K_M9.5 GB VRAM90.6 tok/sRuns great

BGood

Granite Code 3B3B

Q4_K_M7.6 GB VRAM42.0 tok/sRuns great

BGood

Phi 3.5 Mini 4B4B

Q4_K_M11.6 GB VRAM56.0 tok/sRuns great

BGood

Gemma 2 9B9B

Q4_K_M13.9 GB VRAM52.2 tok/sRuns great

BGood

Llama 3.2 11B Vision11B

Q4_K_M12.0 GB VRAM57.7 tok/sRuns great

BGood

Qwen 2.5 Coder 14B14B

Q4_K_M14.8 GB VRAM45.5 tok/sRuns great

BGood

Llama 3.2 3B3B

Q4_K_M6.8 GB VRAM42.0 tok/sRuns great

BGood

100

Mistral Nemo 12B12B

Q4_K_M13.1 GB VRAM52.9 tok/sRuns great

BGood

101

Mistral 7B Instruct v0.37B

Q4_K_M9.5 GB VRAM90.6 tok/sRuns great

BGood

102

Yi Coder 9B9B

Q4_K_M10.3 GB VRAM71.3 tok/sRuns great

BGood

103

TinyLlama 1.1B1.100000023841858B

Q4_K_M4.3 GB VRAM15.4 tok/sRuns great

CUsable

104

Phi 3 Medium 14B14B

Q4_K_M14.9 GB VRAM45.3 tok/sRuns great

BGood

105

DeepSeek R1 1.5B1.5B

Q4_K_M4.6 GB VRAM21.0 tok/sRuns great

CUsable

106

Codestral 22B22B

Q4_K_M19.2 GB VRAM28.8 tok/sRuns great

BGood

107

Ministral 8B8B

Q4_K_M10.4 GB VRAM79.3 tok/sRuns great

BGood

108

Qwen 2.5 Coder 0.5B0.5B

Q4_K_M3.8 GB VRAM7.0 tok/sRuns great

CUsable

109

Qwen 2.5 1.5B1.5B

Q4_K_M4.6 GB VRAM21.0 tok/sRuns great

CUsable

110

Gemma 3 1B1B

Q4_K_M4.3 GB VRAM14.0 tok/sRuns great

CUsable

111

SmolLM3 3B3B

Q4_K_M7.1 GB VRAM42.0 tok/sRuns great

CUsable

112

Qwen 3 0.6B0.6000000238418579B

Q4_K_M4.5 GB VRAM8.4 tok/sRuns great

CUsable

113

Aya Expanse 32B32B

Q4_K_M25.3 GB VRAM13.5 tok/sNeeds offload

CUsable

114

Gemma 2 2B2B

Q4_K_M6.1 GB VRAM28.0 tok/sRuns great

CUsable

115

Llama 3.2 1B1B

Q4_K_M4.4 GB VRAM14.0 tok/sRuns great

CUsable

116

Granite 3.1 8B8B

Q4_K_M10.1 GB VRAM91.2 tok/sRuns great

BGood

117

Yi 1.5 9B9B

Q4_K_M10.3 GB VRAM71.3 tok/sRuns great

CUsable

118

Qwen 2.5 Math 7B7B

Q4_K_M8.4 GB VRAM91.5 tok/sRuns great

CUsable

119

Qwen 3.5 0.6B0.6000000238418579B

Q4_K_M4.5 GB VRAM8.4 tok/sRuns great

CUsable

120

Qwen 2.5 0.5B0.5B

Q4_K_M3.8 GB VRAM7.0 tok/sRuns great

DPoor

121

OpenChat 7B7B

Q4_K_M9.5 GB VRAM90.6 tok/sRuns great

CUsable

122

Aya Expanse 8B8B

Q4_K_M10.1 GB VRAM79.3 tok/sRuns great

CUsable

123

StarCoder2 15B15B

Q5_K_M15.3 GB VRAM37.1 tok/sRuns great

CUsable

124

Nemotron Mini 4B4B

Q4_K_M7.7 GB VRAM56.0 tok/sRuns great

CUsable

125

OpenHermes 2.5 7B7B

Q4_K_M9.5 GB VRAM90.6 tok/sRuns great

CUsable

126

Dolphin 2.9 8B8B

Q4_K_M10.1 GB VRAM79.3 tok/sRuns great

CUsable

127

Zephyr 7B Beta7B

Q4_K_M9.5 GB VRAM90.6 tok/sRuns great

CUsable

128

Vicuna 7B7B

Q4_K_M15.4 GB VRAM84.3 tok/sRuns great

BGood

129

Qwen3.5 27B27B

Q4_K_M22.9 GB VRAM21.9 tok/sNeeds offload

CUsable

130

Qwen3.5 9B9B

Q4_K_M9.8 GB VRAM65.6 tok/sRuns great

CUsable

131

gemma 2b2B

Q4_K_M4.8 GB VRAM28.0 tok/sRuns great

CUsable

132

Qwen3.5 9B Uncensored HauhauCS Aggressive9B

Q4_K_M9.8 GB VRAM65.6 tok/sRuns great

CUsable

133

gemma 2 2b it2B

Q6_K5.2 GB VRAM28.0 tok/sRuns great

CUsable

134

Meta Llama 3.1 8B Instruct8B

Q4_K_M9.1 GB VRAM73.8 tok/sRuns great

CUsable

135

Llama 3.2 3B Instruct3B

Q5_K_M5.8 GB VRAM42.0 tok/sRuns great

CUsable

136

Qwen3.5 4B4B

Q4_K_M6.2 GB VRAM56.0 tok/sRuns great

CUsable

137

Llama 2 7B Chat7B

Q4_K_M8.4 GB VRAM84.3 tok/sRuns great

CUsable

138

llava llama 3 8b v1 18B

Q4_K_M9.1 GB VRAM73.8 tok/sRuns great

CUsable

139

Llama 3.2 1B Instruct Q8 01B

Q6_K4.2 GB VRAM14.0 tok/sRuns great

DPoor

140

Starling LM 7B7B

Q4_K_M9.5 GB VRAM90.6 tok/sRuns great

CUsable

141

Qwen2.5 3B Instruct3B

Q4_K_M5.5 GB VRAM42.0 tok/sRuns great

CUsable

142

Qwen2.5 1.5B Instruct1.5B

Q4_K_M4.4 GB VRAM21.0 tok/sRuns great

CUsable

143

SmolVLM 500M Instruct0.5B

Q6_K3.8 GB VRAM7.0 tok/sRuns great

DPoor

144

Mistral 7B Instruct v0.27B

Q4_K_M8.4 GB VRAM84.3 tok/sRuns great

CUsable

145

DeepSeek R1 0528 Qwen3 8B8B

Q4_K_M9.1 GB VRAM73.8 tok/sRuns great

CUsable

146

TinyLlama 1.1B Chat v1.01.100000023841858B

Q4_K_M4.1 GB VRAM15.4 tok/sRuns great

CUsable

147

gemma 3 27b it27B

Q4_K_M22.9 GB VRAM21.9 tok/sNeeds offload

CUsable

148

Gemmasutra Mini 2B v12B

Q4_K_M4.8 GB VRAM28.0 tok/sRuns great

CUsable

149

Qwen3.5 9B9B

Q4_K_M9.8 GB VRAM65.6 tok/sRuns great

CUsable

150

Mistral 7B Instruct v0.37B

Q4_K_M8.4 GB VRAM84.3 tok/sRuns great

CUsable

151

gemma 3 4b it4B

Q4_K_M6.2 GB VRAM56.0 tok/sRuns great

CUsable

152

embeddinggemma 300M0.30000001192092896B

Q6_K3.6 GB VRAM4.2 tok/sRuns great

DPoor

153

Meta Llama 3 8B Instruct8B

Q4_K_M9.1 GB VRAM73.8 tok/sRuns great

CUsable

154

DeepSeek R1 Distill Llama 8B8B

Q4_K_M9.1 GB VRAM73.8 tok/sRuns great

CUsable

155

Dolphin3.0 Llama3.1 8B8B

Q4_K_M9.1 GB VRAM73.8 tok/sRuns great

CUsable

156

Llama 3 8B Instruct 32k v0.18B

Q4_K_M9.1 GB VRAM73.8 tok/sRuns great

CUsable

157

DeepSeek R1 Distill Qwen 1.5B1.5B

Q4_K_M4.4 GB VRAM21.0 tok/sRuns great

CUsable

158

Meta Llama 3.1 8B Instruct8B

Q4_K_M9.1 GB VRAM73.8 tok/sRuns great

CUsable

159

vntl llama3 8b v28B

Q4_K_M9.1 GB VRAM73.8 tok/sRuns great

CUsable

160

Mistral Small 3.2 24B Instruct 250624B

Q4_K_M20.8 GB VRAM24.6 tok/sTight fit

CUsable

161

DeepSeek R1 0528 Qwen3 8B8B

Q4_K_M9.1 GB VRAM73.8 tok/sRuns great

CUsable

162

DeepSeek R1 Distill Qwen 14B14B

Q4_K_M13.5 GB VRAM42.2 tok/sRuns great

CUsable

163

gemma 3 4b it4B

Q4_K_M6.2 GB VRAM56.0 tok/sRuns great

CUsable

164

Yi Coder 1.5B Chat1.5B

Q4_K_M4.4 GB VRAM21.0 tok/sRuns great

CUsable

165

gemma 3 12b it12B

Q4_K_M12.0 GB VRAM49.2 tok/sRuns great

CUsable

166

Llama 3.2 1B Instruct1B

Q4_K_M4.0 GB VRAM14.0 tok/sRuns great

DPoor

167

gemma 2 2b it2B

Q4_K_M4.8 GB VRAM28.0 tok/sRuns great

CUsable

168

Llama 3.2 3B Instruct3B

Q4_K_M5.5 GB VRAM42.0 tok/sRuns great

CUsable

169

gemma 3 1b it1B

Q4_K_M4.0 GB VRAM14.0 tok/sRuns great

DPoor

170

cognitivecomputations Dolphin Mistral 24B Venice Edition24B

Q4_K_M20.8 GB VRAM24.6 tok/sTight fit

CUsable

171

DeepSeek R1 0528 Qwen3 8B8B

Q4_K_M9.1 GB VRAM73.8 tok/sRuns great

CUsable

172

Mistral Small 24B Instruct 250124B

Q4_K_M20.8 GB VRAM24.6 tok/sTight fit

CUsable

173

Yi 1.5 6B Chat6B

Q4_K_M7.7 GB VRAM84.0 tok/sRuns great

CUsable

174

Ministral 3 3B Instruct 25123B

Q4_K_M5.5 GB VRAM42.0 tok/sRuns great

CUsable

175

gemma 3 27b it27B

Q4_K_M22.9 GB VRAM21.9 tok/sNeeds offload

CUsable

176

Yi Coder 9B Chat9B

Q4_K_M9.8 GB VRAM65.6 tok/sRuns great

CUsable

177

glm 4 9b chat 1m9B

Q4_K_M9.8 GB VRAM65.6 tok/sRuns great

CUsable

178

Qwen3 8B DeepSeek v3.2 Speciale Distill8B

Q4_K_M9.1 GB VRAM73.8 tok/sRuns great

CUsable

179

Hermes 3 Llama 3.1 8B8B

Q4_K_M9.1 GB VRAM73.8 tok/sRuns great

CUsable

180

Mistral 7B Instruct v0.37B

Q4_K_M8.4 GB VRAM84.3 tok/sRuns great

CUsable

181

stablelm 2 zephyr 1 6b6B

Q4_K_M7.7 GB VRAM84.0 tok/sRuns great

CUsable

182

Hermes 2 Pro Mistral 7B7B

Q4_K_M8.4 GB VRAM84.3 tok/sRuns great

CUsable

183

mistral small 3.1 24b instruct 2503 hf24B

Q4_K_M20.8 GB VRAM24.6 tok/sTight fit

CUsable

184

TinyLlama 1.1B Chat v0.31.100000023841858B

Q4_K_M4.1 GB VRAM15.4 tok/sRuns great

DPoor

185

cognitivecomputations Dolphin3.0 R1 Mistral 24B24B

Q4_K_M20.8 GB VRAM24.6 tok/sTight fit

CUsable

186

TinyLlama 1.1B Chat v0.61.100000023841858B

Q4_K_M4.1 GB VRAM15.4 tok/sRuns great

DPoor

187

zephyr 7B beta7B

Q4_K_M8.4 GB VRAM84.3 tok/sRuns great

CUsable

188

Codestral 22B v0.122B

Q4_K_M19.3 GB VRAM26.8 tok/sRuns great

CUsable

189

SOLAR 10.7B Instruct v1.0 uncensored10.699999809265137B

Q4_K_M11.1 GB VRAM55.2 tok/sRuns great

CUsable

190

Hermes 2 Pro Llama 3 8B8B

Q4_K_M9.1 GB VRAM73.8 tok/sRuns great

CUsable

191

Dolphin3.0 Llama3.1 8B8B

Q4_K_M9.1 GB VRAM73.8 tok/sRuns great

CUsable

192

Nous Hermes 2 Mistral 7B DPO7B

Q4_K_M8.4 GB VRAM84.3 tok/sRuns great

CUsable

193

Codestral 22B v0.122B

Q4_K_M19.3 GB VRAM26.8 tok/sRuns great

CUsable

194

Nous Hermes 2 SOLAR 10.7B10.699999809265137B

Q4_K_M11.1 GB VRAM55.2 tok/sRuns great

CUsable

195

Phi 4 reasoning vision 15B15B

Q4_K_M14.2 GB VRAM39.3 tok/sRuns great

CUsable

196

granite 8b code instruct 4k8B

Q4_K_M9.1 GB VRAM73.8 tok/sRuns great

CUsable

197

Dolphin Mistral GLM 4.7 Flash 24B Venice Edition Thinking Uncensored i124B

Q4_K_M20.8 GB VRAM24.6 tok/sTight fit

CUsable

198

HELVETE 3B3B

Q4_K_M5.5 GB VRAM42.0 tok/sRuns great

CUsable

199

StarCoder2 15B15B

Q4_K_M14.2 GB VRAM39.3 tok/sRuns great

CUsable

200

dolphin 2.9.4 llama3.1 8b8B

Q4_K_M9.1 GB VRAM73.8 tok/sRuns great

CUsable

201

Yi 1.5 9B Chat9B

Q4_K_M9.8 GB VRAM65.6 tok/sRuns great

CUsable

202

Falcon H1R 7B7B

Q4_K_M8.4 GB VRAM84.3 tok/sRuns great

CUsable

203

NousResearch Hermes 4 14B14B

Q4_K_M13.5 GB VRAM42.2 tok/sRuns great

CUsable

204

granite embedding 107m multilingual0.10700000077486038B

Q4_K_M3.5 GB VRAM2.0 tok/sRuns great

DPoor

205

falcon mamba 7b instruct Q4 K M7B

Q4_K_M8.4 GB VRAM97.0 tok/sRuns great

CUsable

206

Hermes 3 Llama 3.2 3B3B

Q4_K_M5.5 GB VRAM42.0 tok/sRuns great

CUsable

207

zephyr 7B alpha7B

Q4_K_M8.4 GB VRAM84.3 tok/sRuns great

CUsable

208

speechless zephyr code functionary 7b7B

Q4_K_M8.4 GB VRAM84.3 tok/sRuns great

CUsable

209

stablelm zephyr 3b3B

Q4_K_M5.5 GB VRAM42.0 tok/sRuns great

CUsable

210

Yi 1.5 6B Chat6B

Q4_K_M7.7 GB VRAM84.0 tok/sRuns great

CUsable

211

openchat 3.6 8b 20240522 IMat8B

Q4_K_M9.1 GB VRAM73.8 tok/sRuns great

CUsable

212

EXAONE 4.0 1.2B1.2000000476837158B

Q4_K_M4.2 GB VRAM16.8 tok/sRuns great

DPoor

213

StarCoder2 3B3B

Q4_K_M5.5 GB VRAM42.0 tok/sRuns great

CUsable

214

stablelm 2 1 6b chat imatrix6B

Q4_K_M7.7 GB VRAM84.0 tok/sRuns great

CUsable

215

EXAONE 3.5 2.4B Instruct2.4000000953674316B

Q4_K_M5.0 GB VRAM33.6 tok/sRuns great

CUsable

216

EXAONE 3.5 7.8B Instruct7.800000190734863B

Q4_K_M9.0 GB VRAM75.7 tok/sRuns great

CUsable

217

StarCoder2 7B7B

Q4_K_M8.4 GB VRAM84.3 tok/sRuns great

CUsable

218

Yi 1.5 6B6B

Q4_K_M7.9 GB VRAM84.0 tok/sRuns great

CUsable

219

aya expanse 8b8B

Q4_K_M9.1 GB VRAM73.8 tok/sRuns great

CUsable

220

japanese stablelm instruct gamma 7B7B

Q4_K_M8.4 GB VRAM84.3 tok/sRuns great

CUsable

221

Yi Coder 1.5B1.5B

Q4_K_M4.4 GB VRAM21.0 tok/sRuns great

CUsable

222

EXAONE 3.5 7.8B Instruct7.800000190734863B

Q4_K_M9.0 GB VRAM75.7 tok/sRuns great

CUsable

223

dolphin v2 8b abliterated i18B

Q4_K_M9.1 GB VRAM73.8 tok/sRuns great

CUsable

224

Codestral 22B v0.122B

Q4_K_M19.3 GB VRAM26.8 tok/sRuns great

CUsable

225

StableLM 2 12B12B

Q5_K_M24.1 GB VRAM28.8 tok/sNeeds offload

CUsable

226

Falcon H1 7B Instruct7B

Q4_K_M8.4 GB VRAM84.3 tok/sRuns great

CUsable

227

Falcon H1 Tiny 90M Instruct0.09000000357627869B

Q4_K_M3.5 GB VRAM2.0 tok/sRuns great

DPoor

228

EXAONE 3.5 7.8B Instruct i17.800000190734863B

Q4_K_M9.0 GB VRAM75.7 tok/sRuns great

CUsable

229

Falcon H1R 7B7B

Q4_K_M8.4 GB VRAM84.3 tok/sRuns great

CUsable

230

AI21 Jamba Reasoning 3B3B

Q4_K_M5.5 GB VRAM42.0 tok/sRuns great

CUsable

231

exaone 3.0 7.8b it7.800000190734863B

Q4_K_M9.0 GB VRAM75.7 tok/sRuns great

CUsable

232

aya expanse 8b orthogonal heretic i18B

Q4_K_M9.1 GB VRAM73.8 tok/sRuns great

CUsable

233

Falcon3 1B Instruct abliterated1B

Q4_K_M4.0 GB VRAM14.0 tok/sRuns great

DPoor

234

starcoder2 15b instruct v0.115B

Q4_K_M14.2 GB VRAM39.3 tok/sRuns great

CUsable

235

starcoder2 15b i115B

Q4_K_M14.2 GB VRAM39.3 tok/sRuns great

CUsable

236

GGUF SOLARized GraniStral 14B 2102 YeAM HCT 32QKV14B

Q4_K_M13.5 GB VRAM42.2 tok/sRuns great

CUsable

237

Yi 9B Coder i19B

Q4_K_M9.8 GB VRAM65.6 tok/sRuns great

CUsable

238

solar finalised finetuned Model 10.7B i110.699999809265137B

Q4_K_M11.1 GB VRAM55.2 tok/sRuns great

CUsable

239

stablelm 3b 4e1t3B

Q4_K_M5.5 GB VRAM42.0 tok/sRuns great

CUsable

240

cognitivecomputations Dolphin Mistral 24B Venice Edition24B

Q4_K_M20.8 GB VRAM24.6 tok/sTight fit

CUsable

241

Mamba Codestral 7B v0.17B

Q4_K_M8.4 GB VRAM97.0 tok/sRuns great

CUsable

242

ai21labs AI21 Jamba Reasoning 3B3B

Q4_K_M5.5 GB VRAM42.0 tok/sRuns great

CUsable

243

starcoder2 15b instruct v0.115B

Q4_K_M14.2 GB VRAM39.3 tok/sRuns great

CUsable

244

stabilityai japanese stablelm base gamma 7b7B

Q4_K_M8.4 GB VRAM84.3 tok/sRuns great

CUsable

245

internlm JanusCoder 14B14B

Q4_K_M13.5 GB VRAM42.2 tok/sRuns great

CUsable

246

stablelm 2 zephyr 1.6b1.600000023841858B

Q4_K_M4.5 GB VRAM22.4 tok/sRuns great

CUsable

247

Falcon H1 1.5B Instruct1.5B

Q4_K_M4.4 GB VRAM21.0 tok/sRuns great

CUsable

248

aya expanse 8b orthogonal heretic8B

Q4_K_M9.1 GB VRAM73.8 tok/sRuns great

CUsable

249

internlm2 5 20b chat20B

Q4_K_M17.8 GB VRAM29.5 tok/sRuns great

CUsable

250

baichuan2 7b chat7B

Q4_K_M8.4 GB VRAM84.3 tok/sRuns great

CUsable

251

Codestral 22B v0.1 IMat22B

Q4_K_M19.3 GB VRAM26.8 tok/sRuns great

CUsable

252

logos16v2 stablelm2 1.6b i11.600000023841858B

Q4_K_M4.5 GB VRAM22.4 tok/sRuns great

CUsable

253

SOLAR 10.7B v1.010.699999809265137B

Q4_K_M11.1 GB VRAM55.2 tok/sRuns great

CUsable

254

GGUF SOLARized GraniStral 14B 1902 YeAM HCT14B

Q4_K_M13.5 GB VRAM42.2 tok/sRuns great

CUsable

255

DiscoPOP zephyr 7b gemma7B

Q4_K_M8.4 GB VRAM84.3 tok/sRuns great

CUsable

256

HelpingAI2 9B9B

Q4_K_M9.8 GB VRAM65.6 tok/sRuns great

CUsable

257

ai21labs AI21 Jamba2 3B3B

Q4_K_M5.5 GB VRAM42.0 tok/sRuns great

CUsable

258

internlm2 limarp chat 20b20B

Q4_K_M17.8 GB VRAM29.5 tok/sRuns great

CUsable

259

TinyLlama 1.1B Chat v1.0 imatrix1.100000023841858B

Q4_K_M4.1 GB VRAM15.4 tok/sRuns great

DPoor

260

starcoder2 7b7B

Q4_K_M8.4 GB VRAM84.3 tok/sRuns great

CUsable

261

zephyr 7b beta Mistral 7B Instruct v0.27B

Q4_K_M8.4 GB VRAM84.3 tok/sRuns great

CUsable

262

OpenChat 3.5 7B Qwen v2.0 i17B

Q4_K_M8.4 GB VRAM84.3 tok/sRuns great

CUsable

263

OpenChat 3.5 7B Starling v2.0 i17B

Q4_K_M8.4 GB VRAM84.3 tok/sRuns great

CUsable

264

HelpingAI2 6B6B

Q4_K_M7.7 GB VRAM84.0 tok/sRuns great

CUsable

265

HelpingAI 15B i115B

Q4_K_M14.2 GB VRAM39.3 tok/sRuns great

CUsable

266

internlm2 math plus 7b IMat7B

Q4_K_M8.4 GB VRAM84.3 tok/sRuns great

CUsable

267

CodeNinja 1.0 OpenChat 7B i17B

Q4_K_M8.4 GB VRAM84.3 tok/sRuns great

CUsable

268

HelpingAI2 9B i19B

Q4_K_M9.8 GB VRAM65.6 tok/sRuns great

CUsable

269

HelpingAI2.5 5B i15B

Q4_K_M6.9 GB VRAM70.0 tok/sRuns great

CUsable

270

Codestral 21B Pruned i121B

Q4_K_M18.6 GB VRAM28.1 tok/sRuns great

CUsable

271

internlm2 5 1 8b chat i18B

Q4_K_M9.1 GB VRAM73.8 tok/sRuns great

CUsable

272

internlm2 math plus 20b i120B

Q4_K_M17.8 GB VRAM29.5 tok/sRuns great

CUsable

273

HelpingAI 9B 200k i19B

Q4_K_M9.8 GB VRAM65.6 tok/sRuns great

CUsable

274

internlm2 5 7b chat i17B

Q4_K_M8.4 GB VRAM84.3 tok/sRuns great

CUsable

275

HelpingAI 3B hindi i13B

Q4_K_M5.5 GB VRAM42.0 tok/sRuns great

CUsable

276

internlm3 8b instruct abliterated i18B

Q4_K_M9.1 GB VRAM73.8 tok/sRuns great

CUsable

277

HelpingAI2 6B i16B

Q4_K_M7.7 GB VRAM84.0 tok/sRuns great

CUsable

278

OpenSafetyLab MD Judge v0 2 internlm2 7b7B

Q4_K_M8.4 GB VRAM84.3 tok/sRuns great

CUsable

279

AI21 Jamba2 3B3B

Q4_K_M5.5 GB VRAM42.0 tok/sRuns great

CUsable

280

MD Judge v0 2 internlm2 7b i17B

Q4_K_M8.4 GB VRAM84.3 tok/sRuns great

CUsable

281

HelpingAI2.5 10B i110B

Q4_K_M10.6 GB VRAM59.0 tok/sRuns great

CUsable

282

HelpingAI 3B hindi3B

Q4_K_M5.5 GB VRAM42.0 tok/sRuns great

CUsable

283

zephyr 7b gemma sft african ultrachat 100k7B

Q4_K_M8.4 GB VRAM84.3 tok/sRuns great

CUsable

284

HelpingAI 9B i19B

Q4_K_M9.8 GB VRAM65.6 tok/sRuns great

CUsable

285

Codestral 22B v0.1 i122B

Q4_K_M19.3 GB VRAM26.8 tok/sRuns great

CUsable

286

jointpreferences mistral 7b sft helpful7B

Q4_K_M8.4 GB VRAM84.3 tok/sRuns great

CUsable

287

zephyr 7b dpo full i17B

Q4_K_M8.4 GB VRAM84.3 tok/sRuns great

CUsable

288

Codestral RAG 19B Pruned i119B

Q4_K_M17.1 GB VRAM31.1 tok/sRuns great

CUsable

289

blossom v3 baichuan2 7b i17B

Q4_K_M8.4 GB VRAM84.3 tok/sRuns great

CUsable

290

Helply 10.2b chat i110.199999809265137B

Q4_K_M10.7 GB VRAM57.9 tok/sRuns great

CUsable

291

AI21 Jamba2 3B i13B

Q4_K_M5.5 GB VRAM42.0 tok/sRuns great

CUsable

292

blossom v1 baichuan 7b i17B

Q4_K_M8.4 GB VRAM84.3 tok/sRuns great

CUsable

293

Neural Chat 7B7B

Q4_K_M9.5 GB VRAM90.6 tok/sRuns great

CUsable

294

StarCoder2 7B7B

Q4_K_M8.1 GB VRAM92.0 tok/sRuns great

CUsable

295

StarCoder2 3B3B

Q4_K_M5.6 GB VRAM42.0 tok/sRuns great

CUsable

296

DeepSeek LLM 7B7B

Q4_K_M14.9 GB VRAM84.3 tok/sRuns great

CUsable

URL: https://willitrunai.com/browse/best-for/24gb

⇱ Best AI Models for 24GB VRAM — Local LLMs | WillItRunAI