Voozh

Qwen 3.5 9B9B

Q4_K_M10.2 GB VRAM70.5 tok/sRuns great

SExcellent

Qwen 3 14B14B

Q4_K_M13.5 GB VRAM45.5 tok/sTight fit

SExcellent

Qwen 3.5 4B4B

Q4_K_M7.1 GB VRAM56.0 tok/sRuns great

SExcellent

Qwen 3 8B8B

Q4_K_M9.6 GB VRAM79.3 tok/sRuns great

SExcellent

Phi-4-reasoning-plus 14B14.699999809265137B

Q4_K_M14.5 GB VRAM43.2 tok/sTight fit

SExcellent

Phi-4 Mini Reasoning 4B3.799999952316284B

Q4_K_M6.3 GB VRAM53.2 tok/sRuns great

SExcellent

Jina Embeddings v30.5720000267028809B

F165.6 GB VRAM8.0 tok/sRuns great

AGreat

BGE M30.5680000185966492B

F164.8 GB VRAM8.0 tok/sRuns great

AGreat

Nemotron Nano 8B8B

Q4_K_M9.3 GB VRAM79.3 tok/sRuns great

SExcellent

Ministral 3 14B14B

Q4_K_M13.5 GB VRAM45.3 tok/sTight fit

SExcellent

mxbai Embed Large0.33500000834465027B

F164.7 GB VRAM4.7 tok/sRuns great

AGreat

Snowflake Arctic Embed L0.33500000834465027B

F164.7 GB VRAM4.7 tok/sRuns great

AGreat

CogVLM2 19B19B

Q4_K_M16.5 GB VRAM23.4 tok/sNeeds offload

AGreat

Nomic Embed Text v1.50.13699999451637268B

F163.3 GB VRAM2.0 tok/sRuns great

BGood

InternVL2 8B8B

Q4_K_M9.3 GB VRAM79.3 tok/sRuns great

SExcellent

Phi-4 14B14B

Q4_K_M14.1 GB VRAM45.3 tok/sTight fit

AGreat

Qwen 3 4B4B

Q4_K_M7.1 GB VRAM56.0 tok/sRuns great

AGreat

BGE Large EN v1.50.33500000834465027B

F164.7 GB VRAM4.7 tok/sRuns great

AGreat

Qwen 2.5 14B14B

Q4_K_M14.0 GB VRAM45.5 tok/sTight fit

AGreat

Ministral 3 8B8B

Q4_K_M9.6 GB VRAM79.3 tok/sRuns great

AGreat

MiniCPM-V 2.6 8B8B

Q4_K_M9.3 GB VRAM79.3 tok/sRuns great

AGreat

Gemma 3 12B12B

Q4_K_M14.7 GB VRAM39.1 tok/sTight fit

AGreat

Qwen 2.5 VL 7B7B

Q4_K_M7.6 GB VRAM91.5 tok/sRuns great

AGreat

SQLCoder 7B7B

Q4_K_M8.7 GB VRAM90.6 tok/sRuns great

AGreat

Nemotron Nano 9B v29B

Q4_K_M10.4 GB VRAM70.5 tok/sRuns great

AGreat

DeepSeek Coder V2 16B16B

Q4_K_M15.6 GB VRAM87.8 tok/sNeeds offload

AGreat

Magistral 7B7B

Q4_K_M8.7 GB VRAM90.6 tok/sRuns great

AGreat

CodeGeeX 4 9B9B

Q4_K_M8.6 GB VRAM71.7 tok/sRuns great

AGreat

Gemma 4 E4B8B

Q4_K_M8.7 GB VRAM60.1 tok/sRuns great

AGreat

OLMo 2 13B13B

Q4_K_M12.9 GB VRAM49.0 tok/sRuns great

AGreat

All MiniLM L6 v20.023000000044703484B

F162.8 GB VRAM2.0 tok/sRuns great

BGood

Qwen 2.5 7B7B

Q4_K_M7.6 GB VRAM91.5 tok/sRuns great

AGreat

Qwen 2.5 Coder 3B3B

Q4_K_M6.5 GB VRAM42.0 tok/sRuns great

AGreat

DevStral 7B7B

Q4_K_M8.7 GB VRAM90.6 tok/sRuns great

AGreat

Codestral Mamba 7B7B

Q4_K_M7.3 GB VRAM97.0 tok/sRuns great

AGreat

Granite Code 8B8B

Q4_K_M9.3 GB VRAM79.3 tok/sRuns great

AGreat

DeepSeek R1 Distill 14B14B

Q4_K_M14.0 GB VRAM45.5 tok/sTight fit

AGreat

Granite 4.1 8B8B

Q4_K_M9.8 GB VRAM79.3 tok/sRuns great

AGreat

Pixtral 12B12B

Q4_K_M12.3 GB VRAM52.9 tok/sRuns great

AGreat

StarCoder 7B7B

Q4_K_M14.1 GB VRAM84.3 tok/sTight fit

AGreat

CodeLlama 7B Instruct7B

Q4_K_M14.6 GB VRAM84.3 tok/sTight fit

AGreat

Gemma 4 E2B5.099999904632568B

Q4_K_M6.1 GB VRAM71.4 tok/sRuns great

AGreat

Ministral 3 3B3B

Q4_K_M5.1 GB VRAM42.0 tok/sRuns great

BGood

Qwen 3.5 2B2B

Q4_K_M5.4 GB VRAM28.0 tok/sRuns great

BGood

GLM-4 9B9B

Q4_K_M8.6 GB VRAM71.7 tok/sRuns great

AGreat

Gemma 3 4B4B

Q4_K_M7.0 GB VRAM56.0 tok/sRuns great

AGreat

Llama 3.1 8B8B

Q4_K_M9.3 GB VRAM79.3 tok/sRuns great

AGreat

InternLM 7B7B

Q4_K_M14.6 GB VRAM84.3 tok/sTight fit

AGreat

InternLM Chat 7B7B

Q4_K_M14.6 GB VRAM84.3 tok/sTight fit

AGreat

Solar 7B7B

Q4_K_M9.7 GB VRAM90.6 tok/sRuns great

AGreat

WizardMath 7B7B

Q4_K_M8.7 GB VRAM90.6 tok/sRuns great

AGreat

OLMo 2 7B7B

Q4_K_M8.7 GB VRAM90.6 tok/sRuns great

AGreat

Phi 4 Mini 4B4B

Q4_K_M6.4 GB VRAM56.0 tok/sRuns great

BGood

Qwen 2.5 Coder 7B7B

Q4_K_M7.6 GB VRAM91.5 tok/sRuns great

AGreat

Qwen 3 1.7B1.7000000476837158B

Q4_K_M5.2 GB VRAM23.8 tok/sRuns great

BGood

Qwen 2.5 Coder 1.5B1.5B

Q4_K_M3.8 GB VRAM21.0 tok/sRuns great

BGood

Qwen 2.5 3B3B

Q4_K_M6.5 GB VRAM42.0 tok/sRuns great

BGood

LLaVA 1.5 7B7B

Q4_K_M14.6 GB VRAM84.3 tok/sTight fit

AGreat

Granite 4.1 3B3B

Q4_K_M5.6 GB VRAM42.0 tok/sRuns great

BGood

Phi 3 Mini 3.8B3.799999952316284B

Q4_K_M10.7 GB VRAM53.2 tok/sRuns great

AGreat

DeepSeek R1 Distill 7B7B

Q4_K_M7.6 GB VRAM91.5 tok/sRuns great

BGood

DeepSeek R1 Distill 8B8B

Q4_K_M9.3 GB VRAM79.3 tok/sRuns great

AGreat

Falcon 7B Instruct7B

Q4_K_M6.9 GB VRAM92.6 tok/sRuns great

BGood

MPT-7B-Instruct7B

Q4_K_M14.6 GB VRAM84.3 tok/sTight fit

BGood

Baichuan 7B7B

Q4_K_M14.6 GB VRAM84.3 tok/sTight fit

BGood

Samantha 7B7B

Q4_K_M8.7 GB VRAM90.6 tok/sRuns great

BGood

Granite Code 3B3B

Q4_K_M6.8 GB VRAM42.0 tok/sRuns great

BGood

Phi 3.5 Mini 4B4B

Q4_K_M10.8 GB VRAM56.0 tok/sRuns great

AGreat

Gemma 2 9B9B

Q4_K_M13.1 GB VRAM52.2 tok/sRuns great

BGood

Llama 3.2 11B Vision11B

Q4_K_M11.2 GB VRAM57.7 tok/sRuns great

BGood

Qwen 2.5 Coder 14B14B

Q4_K_M14.0 GB VRAM45.5 tok/sTight fit

BGood

Llama 3.2 3B3B

Q4_K_M6.0 GB VRAM42.0 tok/sRuns great

BGood

Mistral Nemo 12B12B

Q4_K_M12.3 GB VRAM52.9 tok/sRuns great

BGood

Mistral 7B Instruct v0.37B

Q4_K_M8.7 GB VRAM90.6 tok/sRuns great

BGood

Yi Coder 9B9B

Q4_K_M9.5 GB VRAM71.3 tok/sRuns great

BGood

TinyLlama 1.1B1.100000023841858B

Q4_K_M3.5 GB VRAM15.4 tok/sRuns great

CUsable

Phi 3 Medium 14B14B

Q4_K_M14.1 GB VRAM45.3 tok/sTight fit

BGood

DeepSeek R1 1.5B1.5B

Q4_K_M3.8 GB VRAM21.0 tok/sRuns great

CUsable

Ministral 8B8B

Q4_K_M9.6 GB VRAM79.3 tok/sRuns great

BGood

Qwen 2.5 Coder 0.5B0.5B

Q4_K_M3.0 GB VRAM7.0 tok/sRuns great

CUsable

Qwen 2.5 1.5B1.5B

Q4_K_M3.8 GB VRAM21.0 tok/sRuns great

CUsable

Gemma 3 1B1B

Q4_K_M3.5 GB VRAM14.0 tok/sRuns great

CUsable

SmolLM3 3B3B

Q4_K_M6.3 GB VRAM42.0 tok/sRuns great

BGood

Qwen 3 0.6B0.6000000238418579B

Q4_K_M3.7 GB VRAM8.4 tok/sRuns great

CUsable

Gemma 2 2B2B

Q4_K_M5.3 GB VRAM28.0 tok/sRuns great

CUsable

Llama 3.2 1B1B

Q4_K_M3.6 GB VRAM14.0 tok/sRuns great

CUsable

Granite 3.1 8B8B

Q4_K_M9.3 GB VRAM91.2 tok/sRuns great

BGood

Yi 1.5 9B9B

Q4_K_M9.5 GB VRAM71.3 tok/sRuns great

BGood

Qwen 2.5 Math 7B7B

Q4_K_M7.6 GB VRAM91.5 tok/sRuns great

BGood

Qwen 3.5 0.6B0.6000000238418579B

Q4_K_M3.7 GB VRAM8.4 tok/sRuns great

CUsable

Qwen 2.5 0.5B0.5B

Q4_K_M3.0 GB VRAM7.0 tok/sRuns great

CUsable

OpenChat 7B7B

Q4_K_M8.7 GB VRAM90.6 tok/sRuns great

BGood

Aya Expanse 8B8B

Q4_K_M9.3 GB VRAM79.3 tok/sRuns great

BGood

StarCoder2 15B15B

Q5_K_M14.5 GB VRAM37.1 tok/sTight fit

CUsable

Nemotron Mini 4B4B

Q4_K_M6.9 GB VRAM56.0 tok/sRuns great

CUsable

OpenHermes 2.5 7B7B

Q4_K_M8.7 GB VRAM90.6 tok/sRuns great

CUsable

Dolphin 2.9 8B8B

Q4_K_M9.3 GB VRAM79.3 tok/sRuns great

CUsable

Zephyr 7B Beta7B

Q4_K_M8.7 GB VRAM90.6 tok/sRuns great

CUsable

Vicuna 7B7B

Q4_K_M14.6 GB VRAM84.3 tok/sTight fit

CUsable

100

Qwen3.5 9B9B

Q4_K_M9.0 GB VRAM65.6 tok/sRuns great

CUsable

101

gemma 2b2B

Q4_K_M4.0 GB VRAM28.0 tok/sRuns great

CUsable

102

Qwen3.5 9B Uncensored HauhauCS Aggressive9B

Q4_K_M9.0 GB VRAM65.6 tok/sRuns great

CUsable

103

gemma 2 2b it2B

Q6_K4.4 GB VRAM28.0 tok/sRuns great

CUsable

104

Meta Llama 3.1 8B Instruct8B

Q4_K_M8.3 GB VRAM73.8 tok/sRuns great

CUsable

105

Llama 3.2 3B Instruct3B

Q5_K_M5.0 GB VRAM42.0 tok/sRuns great

CUsable

106

Qwen3.5 4B4B

Q4_K_M5.4 GB VRAM56.0 tok/sRuns great

CUsable

107

Llama 2 7B Chat7B

Q4_K_M7.6 GB VRAM84.3 tok/sRuns great

CUsable

108

llava llama 3 8b v1 18B

Q4_K_M8.3 GB VRAM73.8 tok/sRuns great

CUsable

109

Llama 3.2 1B Instruct Q8 01B

Q6_K3.4 GB VRAM14.0 tok/sRuns great

CUsable

110

Starling LM 7B7B

Q4_K_M8.7 GB VRAM90.6 tok/sRuns great

CUsable

111

Qwen2.5 3B Instruct3B

Q4_K_M4.7 GB VRAM42.0 tok/sRuns great

CUsable

112

Qwen2.5 1.5B Instruct1.5B

Q4_K_M3.6 GB VRAM21.0 tok/sRuns great

CUsable

113

SmolVLM 500M Instruct0.5B

Q6_K3.0 GB VRAM7.0 tok/sRuns great

DPoor

114

Mistral 7B Instruct v0.27B

Q4_K_M7.6 GB VRAM84.3 tok/sRuns great

CUsable

115

DeepSeek R1 0528 Qwen3 8B8B

Q4_K_M8.3 GB VRAM73.8 tok/sRuns great

CUsable

116

TinyLlama 1.1B Chat v1.01.100000023841858B

Q4_K_M3.3 GB VRAM15.4 tok/sRuns great

CUsable

117

Gemmasutra Mini 2B v12B

Q4_K_M4.0 GB VRAM28.0 tok/sRuns great

CUsable

118

Qwen3.5 9B9B

Q4_K_M9.0 GB VRAM65.6 tok/sRuns great

CUsable

119

Mistral 7B Instruct v0.37B

Q4_K_M7.6 GB VRAM84.3 tok/sRuns great

CUsable

120

gemma 3 4b it4B

Q4_K_M5.4 GB VRAM56.0 tok/sRuns great

CUsable

121

embeddinggemma 300M0.30000001192092896B

Q6_K2.8 GB VRAM4.2 tok/sRuns great

DPoor

122

Meta Llama 3 8B Instruct8B

Q4_K_M8.3 GB VRAM73.8 tok/sRuns great

CUsable

123

DeepSeek R1 Distill Llama 8B8B

Q4_K_M8.3 GB VRAM73.8 tok/sRuns great

CUsable

124

Dolphin3.0 Llama3.1 8B8B

Q4_K_M8.3 GB VRAM73.8 tok/sRuns great

CUsable

125

Llama 3 8B Instruct 32k v0.18B

Q4_K_M8.3 GB VRAM73.8 tok/sRuns great

CUsable

126

DeepSeek R1 Distill Qwen 1.5B1.5B

Q4_K_M3.6 GB VRAM21.0 tok/sRuns great

CUsable

127

Meta Llama 3.1 8B Instruct8B

Q4_K_M8.3 GB VRAM73.8 tok/sRuns great

CUsable

128

vntl llama3 8b v28B

Q4_K_M8.3 GB VRAM73.8 tok/sRuns great

CUsable

129

DeepSeek R1 0528 Qwen3 8B8B

Q4_K_M8.3 GB VRAM73.8 tok/sRuns great

CUsable

130

DeepSeek R1 Distill Qwen 14B14B

Q4_K_M12.7 GB VRAM42.2 tok/sRuns great

CUsable

131

gemma 3 4b it4B

Q4_K_M5.4 GB VRAM56.0 tok/sRuns great

CUsable

132

Yi Coder 1.5B Chat1.5B

Q4_K_M3.6 GB VRAM21.0 tok/sRuns great

CUsable

133

gemma 3 12b it12B

Q4_K_M11.2 GB VRAM49.2 tok/sRuns great

CUsable

134

Llama 3.2 1B Instruct1B

Q4_K_M3.2 GB VRAM14.0 tok/sRuns great

CUsable

135

gemma 2 2b it2B

Q4_K_M4.0 GB VRAM28.0 tok/sRuns great

CUsable

136

Llama 3.2 3B Instruct3B

Q4_K_M4.7 GB VRAM42.0 tok/sRuns great

CUsable

137

gemma 3 1b it1B

Q4_K_M3.2 GB VRAM14.0 tok/sRuns great

CUsable

138

DeepSeek R1 0528 Qwen3 8B8B

Q4_K_M8.3 GB VRAM73.8 tok/sRuns great

CUsable

139

Yi 1.5 6B Chat6B

Q4_K_M6.9 GB VRAM84.0 tok/sRuns great

CUsable

140

Ministral 3 3B Instruct 25123B

Q4_K_M4.7 GB VRAM42.0 tok/sRuns great

CUsable

141

Yi Coder 9B Chat9B

Q4_K_M9.0 GB VRAM65.6 tok/sRuns great

CUsable

142

glm 4 9b chat 1m9B

Q4_K_M9.0 GB VRAM65.6 tok/sRuns great

CUsable

143

Qwen3 8B DeepSeek v3.2 Speciale Distill8B

Q4_K_M8.3 GB VRAM73.8 tok/sRuns great

CUsable

144

Hermes 3 Llama 3.1 8B8B

Q4_K_M8.3 GB VRAM73.8 tok/sRuns great

CUsable

145

Mistral 7B Instruct v0.37B

Q4_K_M7.6 GB VRAM84.3 tok/sRuns great

CUsable

146

stablelm 2 zephyr 1 6b6B

Q4_K_M6.9 GB VRAM84.0 tok/sRuns great

CUsable

147

Hermes 2 Pro Mistral 7B7B

Q4_K_M7.6 GB VRAM84.3 tok/sRuns great

CUsable

148

TinyLlama 1.1B Chat v0.31.100000023841858B

Q4_K_M3.3 GB VRAM15.4 tok/sRuns great

CUsable

149

TinyLlama 1.1B Chat v0.61.100000023841858B

Q4_K_M3.3 GB VRAM15.4 tok/sRuns great

CUsable

150

zephyr 7B beta7B

Q4_K_M7.6 GB VRAM84.3 tok/sRuns great

CUsable

151

SOLAR 10.7B Instruct v1.0 uncensored10.699999809265137B

Q4_K_M10.3 GB VRAM55.2 tok/sRuns great

CUsable

152

Hermes 2 Pro Llama 3 8B8B

Q4_K_M8.3 GB VRAM73.8 tok/sRuns great

CUsable

153

Dolphin3.0 Llama3.1 8B8B

Q4_K_M8.3 GB VRAM73.8 tok/sRuns great

CUsable

154

Nous Hermes 2 Mistral 7B DPO7B

Q4_K_M7.6 GB VRAM84.3 tok/sRuns great

CUsable

155

Nous Hermes 2 SOLAR 10.7B10.699999809265137B

Q4_K_M10.3 GB VRAM55.2 tok/sRuns great

CUsable

156

Phi 4 reasoning vision 15B15B

Q4_K_M13.4 GB VRAM39.3 tok/sTight fit

CUsable

157

granite 8b code instruct 4k8B

Q4_K_M8.3 GB VRAM73.8 tok/sRuns great

CUsable

158

HELVETE 3B3B

Q4_K_M4.7 GB VRAM42.0 tok/sRuns great

CUsable

159

StarCoder2 15B15B

Q4_K_M13.4 GB VRAM39.3 tok/sTight fit

CUsable

160

dolphin 2.9.4 llama3.1 8b8B

Q4_K_M8.3 GB VRAM73.8 tok/sRuns great

CUsable

161

Yi 1.5 9B Chat9B

Q4_K_M9.0 GB VRAM65.6 tok/sRuns great

CUsable

162

Falcon H1R 7B7B

Q4_K_M7.6 GB VRAM84.3 tok/sRuns great

CUsable

163

NousResearch Hermes 4 14B14B

Q4_K_M12.7 GB VRAM42.2 tok/sRuns great

CUsable

164

granite embedding 107m multilingual0.10700000077486038B

Q4_K_M2.7 GB VRAM2.0 tok/sRuns great

DPoor

165

falcon mamba 7b instruct Q4 K M7B

Q4_K_M7.6 GB VRAM97.0 tok/sRuns great

CUsable

166

Hermes 3 Llama 3.2 3B3B

Q4_K_M4.7 GB VRAM42.0 tok/sRuns great

CUsable

167

zephyr 7B alpha7B

Q4_K_M7.6 GB VRAM84.3 tok/sRuns great

CUsable

168

speechless zephyr code functionary 7b7B

Q4_K_M7.6 GB VRAM84.3 tok/sRuns great

CUsable

169

stablelm zephyr 3b3B

Q4_K_M4.7 GB VRAM42.0 tok/sRuns great

CUsable

170

Yi 1.5 6B Chat6B

Q4_K_M6.9 GB VRAM84.0 tok/sRuns great

CUsable

171

openchat 3.6 8b 20240522 IMat8B

Q4_K_M8.3 GB VRAM73.8 tok/sRuns great

CUsable

172

EXAONE 4.0 1.2B1.2000000476837158B

Q4_K_M3.4 GB VRAM16.8 tok/sRuns great

CUsable

173

StarCoder2 3B3B

Q4_K_M4.7 GB VRAM42.0 tok/sRuns great

CUsable

174

stablelm 2 1 6b chat imatrix6B

Q4_K_M6.9 GB VRAM84.0 tok/sRuns great

CUsable

175

EXAONE 3.5 2.4B Instruct2.4000000953674316B

Q4_K_M4.2 GB VRAM33.6 tok/sRuns great

CUsable

176

EXAONE 3.5 7.8B Instruct7.800000190734863B

Q4_K_M8.2 GB VRAM75.7 tok/sRuns great

CUsable

177

StarCoder2 7B7B

Q4_K_M7.6 GB VRAM84.3 tok/sRuns great

CUsable

178

Yi 1.5 6B6B

Q4_K_M7.1 GB VRAM84.0 tok/sRuns great

CUsable

179

aya expanse 8b8B

Q4_K_M8.3 GB VRAM73.8 tok/sRuns great

CUsable

180

japanese stablelm instruct gamma 7B7B

Q4_K_M7.6 GB VRAM84.3 tok/sRuns great

CUsable

181

Yi Coder 1.5B1.5B

Q4_K_M3.6 GB VRAM21.0 tok/sRuns great

CUsable

182

EXAONE 3.5 7.8B Instruct7.800000190734863B

Q4_K_M8.2 GB VRAM75.7 tok/sRuns great

CUsable

183

dolphin v2 8b abliterated i18B

Q4_K_M8.3 GB VRAM73.8 tok/sRuns great

CUsable

184

Falcon H1 7B Instruct7B

Q4_K_M7.6 GB VRAM84.3 tok/sRuns great

CUsable

185

Falcon H1 Tiny 90M Instruct0.09000000357627869B

Q4_K_M2.7 GB VRAM2.0 tok/sRuns great

DPoor

186

EXAONE 3.5 7.8B Instruct i17.800000190734863B

Q4_K_M8.2 GB VRAM75.7 tok/sRuns great

CUsable

187

Falcon H1R 7B7B

Q4_K_M7.6 GB VRAM84.3 tok/sRuns great

CUsable

188

AI21 Jamba Reasoning 3B3B

Q4_K_M4.7 GB VRAM42.0 tok/sRuns great

CUsable

189

exaone 3.0 7.8b it7.800000190734863B

Q4_K_M8.2 GB VRAM75.7 tok/sRuns great

CUsable

190

aya expanse 8b orthogonal heretic i18B

Q4_K_M8.3 GB VRAM73.8 tok/sRuns great

CUsable

191

Falcon3 1B Instruct abliterated1B

Q4_K_M3.2 GB VRAM14.0 tok/sRuns great

DPoor

192

starcoder2 15b instruct v0.115B

Q4_K_M13.4 GB VRAM39.3 tok/sTight fit

CUsable

193

starcoder2 15b i115B

Q4_K_M13.4 GB VRAM39.3 tok/sTight fit

CUsable

194

GGUF SOLARized GraniStral 14B 2102 YeAM HCT 32QKV14B

Q4_K_M12.7 GB VRAM42.2 tok/sRuns great

CUsable

195

Yi 9B Coder i19B

Q4_K_M9.0 GB VRAM65.6 tok/sRuns great

CUsable

196

solar finalised finetuned Model 10.7B i110.699999809265137B

Q4_K_M10.3 GB VRAM55.2 tok/sRuns great

CUsable

197

stablelm 3b 4e1t3B

Q4_K_M4.7 GB VRAM42.0 tok/sRuns great

CUsable

198

Mamba Codestral 7B v0.17B

Q4_K_M7.6 GB VRAM97.0 tok/sRuns great

CUsable

199

ai21labs AI21 Jamba Reasoning 3B3B

Q4_K_M4.7 GB VRAM42.0 tok/sRuns great

CUsable

200

starcoder2 15b instruct v0.115B

Q4_K_M13.4 GB VRAM39.3 tok/sTight fit

CUsable

201

stabilityai japanese stablelm base gamma 7b7B

Q4_K_M7.6 GB VRAM84.3 tok/sRuns great

CUsable

202

internlm JanusCoder 14B14B

Q4_K_M12.7 GB VRAM42.2 tok/sRuns great

CUsable

203

stablelm 2 zephyr 1.6b1.600000023841858B

Q4_K_M3.7 GB VRAM22.4 tok/sRuns great

CUsable

204

Falcon H1 1.5B Instruct1.5B

Q4_K_M3.6 GB VRAM21.0 tok/sRuns great

CUsable

205

aya expanse 8b orthogonal heretic8B

Q4_K_M8.3 GB VRAM73.8 tok/sRuns great

CUsable

206

internlm2 5 20b chat20B

Q4_K_M17.0 GB VRAM19.4 tok/sNeeds offload

DPoor

207

baichuan2 7b chat7B

Q4_K_M7.6 GB VRAM84.3 tok/sRuns great

CUsable

208

logos16v2 stablelm2 1.6b i11.600000023841858B

Q4_K_M3.7 GB VRAM22.4 tok/sRuns great

CUsable

209

SOLAR 10.7B v1.010.699999809265137B

Q4_K_M10.3 GB VRAM55.2 tok/sRuns great

CUsable

210

GGUF SOLARized GraniStral 14B 1902 YeAM HCT14B

Q4_K_M12.7 GB VRAM42.2 tok/sRuns great

CUsable

211

DiscoPOP zephyr 7b gemma7B

Q4_K_M7.6 GB VRAM84.3 tok/sRuns great

CUsable

212

HelpingAI2 9B9B

Q4_K_M9.0 GB VRAM65.6 tok/sRuns great

CUsable

213

ai21labs AI21 Jamba2 3B3B

Q4_K_M4.7 GB VRAM42.0 tok/sRuns great

CUsable

214

internlm2 limarp chat 20b20B

Q4_K_M17.0 GB VRAM19.4 tok/sNeeds offload

DPoor

215

TinyLlama 1.1B Chat v1.0 imatrix1.100000023841858B

Q4_K_M3.3 GB VRAM15.4 tok/sRuns great

CUsable

216

starcoder2 7b7B

Q4_K_M7.6 GB VRAM84.3 tok/sRuns great

CUsable

217

zephyr 7b beta Mistral 7B Instruct v0.27B

Q4_K_M7.6 GB VRAM84.3 tok/sRuns great

CUsable

218

OpenChat 3.5 7B Qwen v2.0 i17B

Q4_K_M7.6 GB VRAM84.3 tok/sRuns great

CUsable

219

OpenChat 3.5 7B Starling v2.0 i17B

Q4_K_M7.6 GB VRAM84.3 tok/sRuns great

CUsable

220

HelpingAI2 6B6B

Q4_K_M6.9 GB VRAM84.0 tok/sRuns great

CUsable

221

HelpingAI 15B i115B

Q4_K_M13.4 GB VRAM39.3 tok/sTight fit

CUsable

222

internlm2 math plus 7b IMat7B

Q4_K_M7.6 GB VRAM84.3 tok/sRuns great

CUsable

223

CodeNinja 1.0 OpenChat 7B i17B

Q4_K_M7.6 GB VRAM84.3 tok/sRuns great

CUsable

224

HelpingAI2 9B i19B

Q4_K_M9.0 GB VRAM65.6 tok/sRuns great

CUsable

225

HelpingAI2.5 5B i15B

Q4_K_M6.1 GB VRAM70.0 tok/sRuns great

CUsable

226

internlm2 5 1 8b chat i18B

Q4_K_M8.3 GB VRAM73.8 tok/sRuns great

CUsable

227

internlm2 math plus 20b i120B

Q4_K_M17.0 GB VRAM19.4 tok/sNeeds offload

DPoor

228

HelpingAI 9B 200k i19B

Q4_K_M9.0 GB VRAM65.6 tok/sRuns great

CUsable

229

internlm2 5 7b chat i17B

Q4_K_M7.6 GB VRAM84.3 tok/sRuns great

CUsable

230

HelpingAI 3B hindi i13B

Q4_K_M4.7 GB VRAM42.0 tok/sRuns great

CUsable

231

internlm3 8b instruct abliterated i18B

Q4_K_M8.3 GB VRAM73.8 tok/sRuns great

CUsable

232

HelpingAI2 6B i16B

Q4_K_M6.9 GB VRAM84.0 tok/sRuns great

CUsable

233

OpenSafetyLab MD Judge v0 2 internlm2 7b7B

Q4_K_M7.6 GB VRAM84.3 tok/sRuns great

CUsable

234

AI21 Jamba2 3B3B

Q4_K_M4.7 GB VRAM42.0 tok/sRuns great

CUsable

235

MD Judge v0 2 internlm2 7b i17B

Q4_K_M7.6 GB VRAM84.3 tok/sRuns great

CUsable

236

HelpingAI2.5 10B i110B

Q4_K_M9.8 GB VRAM59.0 tok/sRuns great

CUsable

237

HelpingAI 3B hindi3B

Q4_K_M4.7 GB VRAM42.0 tok/sRuns great

CUsable

238

zephyr 7b gemma sft african ultrachat 100k7B

Q4_K_M7.6 GB VRAM84.3 tok/sRuns great

CUsable

239

HelpingAI 9B i19B

Q4_K_M9.0 GB VRAM65.6 tok/sRuns great

CUsable

240

jointpreferences mistral 7b sft helpful7B

Q4_K_M7.6 GB VRAM84.3 tok/sRuns great

CUsable

241

zephyr 7b dpo full i17B

Q4_K_M7.6 GB VRAM84.3 tok/sRuns great

CUsable

242

Codestral RAG 19B Pruned i119B

Q4_K_M16.3 GB VRAM22.4 tok/sNeeds offload

CUsable

243

blossom v3 baichuan2 7b i17B

Q4_K_M7.6 GB VRAM84.3 tok/sRuns great

CUsable

244

Helply 10.2b chat i110.199999809265137B

Q4_K_M9.9 GB VRAM57.9 tok/sRuns great

CUsable

245

AI21 Jamba2 3B i13B

Q4_K_M4.7 GB VRAM42.0 tok/sRuns great

CUsable

246

blossom v1 baichuan 7b i17B

Q4_K_M7.6 GB VRAM84.3 tok/sRuns great

CUsable

247

Neural Chat 7B7B

Q4_K_M8.7 GB VRAM90.6 tok/sRuns great

CUsable

248

StarCoder2 7B7B

Q4_K_M7.3 GB VRAM92.0 tok/sRuns great

CUsable

249

StarCoder2 3B3B

Q4_K_M4.8 GB VRAM42.0 tok/sRuns great

CUsable

250

DeepSeek LLM 7B7B

Q4_K_M14.1 GB VRAM84.3 tok/sTight fit

CUsable

URL: https://willitrunai.com/browse/best-for/16gb

⇱ Best AI Models for 16GB VRAM — Local LLMs | WillItRunAI