Large language model (LLM) inference in C/C++.

llama.cpp is available in the AUR:

• Install llama.cpp^AUR for CPU inference.
• Install llama.cpp-vulkan^AUR for inference with Vulkan.
• Install llama.cpp-cuda^AUR for inference with CUDA.
• Install llama.cpp-hip^AUR for inference with ROCm.

Alternatively, follow the instructions on llama.app.

The primary executor is llama.

llama cli returns an interactive prompt in command-line:

$ llama cli -m model.gguf

llama serve launches an API server with a built-in WebUI:

$ llama serve --host address --port port -m model.gguf

llama help returns a list available commands:

$ llama help

llama.cpp uses models in the GGUF format.

Download models from Hugging Face using the -hf flag:

$ llama cli -hf org/model

Models can be downloaded manually using a full URL and a downloader such as wget or curl.

Quantization lowers model precision to reduce memory usage.

GGUF models use suffixes to indicate quantization level. Generally, lower numbers (e.g. Q4) use less memory but may reduce quality compared to higher numbers (e.g. Q8).

Knowledge distillation compresses a larger model into a smaller model by training the smaller model to follow the behaviors of the larger model.

Typically, GGUF models indicate knowledge distillation using the student-teacher-distill denotation, where:

• student represents the smaller model.
• teacher represents the larger model.

llama.cpp loads the context size from the model by default, and it allocates memory for the whole context window.

Specify a lower context size in case you run out of memory.

$ llama cli -c 32000 -m model.gguf

For further memory efficiency, you can quantize the key-value cache.

$ llama cli -ctk q8_0 -ctv q8_0 -m model.gguf

This, combined with a lower context size, can significantly reduce memory usage.

While the API server runs a WebUI, the same endpoint also operates as an OpenAI-compatible server. It can be configured to use with a coding agent like opencode.

Alternatively, the WebUI has introduced built-in agent capabilities.

To enable built-in tools for filesystem operations and shell access, start llama-server with:

$ llama serve --tools all -m model.gguf

This, combined with a reasonably strong reasoning model, can be considered as a minimal coding agent running in web browser.

Other tools (e.g. web_search, fetch) can be integrated to the WebUI, given that the tools are served as MCP endpoints.

See Graphics processing unit#Monitoring.

To use the WebUI with an MCP endpoint hosted online, enable MCP CORS proxy:

$ llama serve --ui-mcp-proxy -m model.gguf

• Upstream GitHub repository
• Upstream guide: using the new WebUI of llama.cpp
• Upstream guide: running gpt-oss with llama.cpp

Retrieved from "https://wiki.archlinux.org/index.php?title=Llama.cpp&oldid=878654"