rocm/vllm-dev repository overview

⁠vLLM-dev

vLLM-dev is a weekly dev stream toolkit and library for large language model (LLM) inference and serving. These containers are a quick way to run or try vLLM on AMD Instinct™ accelerators. For building or extensions for vLLM-dev, please see the Build Dockerfiles⁠.

Running Containers

docker run -it --rm \
 --ipc=host \
 --network=host \
 --privileged \
 --cap-add=CAP_SYS_ADMIN \
 --device=/dev/kfd \
 --device=/dev/dri \
 --device=/dev/mem \
 --group-add render \
 --cap-add=SYS_PTRACE \
 --security-opt seccomp=unconfined \
 rocm/vllm-dev:main

Documentation

Please see here⁠ for full documentation on getting started, setting up the environment, and more.

Tag summary