Run gpt-oss 120B on vLLM with an AMD Instinct MI300X GPU Droplet

One of the greatest challenges any new user of large scale LLM technology needs to consider is always going to be computation. From the VRAM to the throughput to the underlying technology and software, there are so many differences between different machines that it can be genuinely dizzying. When deploying LLMs, this can be even more apparent. At the end of the day, we want to get the best quality at a low cost, and it’s striking the balance where we find the true source of the challenge.

Today we are going to examine this more closely with a look at AMD Instinct’s MI300X GPU running gpt-oss 120b. This powerful machine is one of the flagship processing units from AMD, and it is truly beefy and fast. With a whopping 192 GB of HBM3 memory, it is capable of processing 653.7 TFLOPs to create an overall, max, theoretical throughput of 5.3 TB/s. This awesome power makes it an ideal machine for testing LLMs, and we are going to use OpenAI’s gpt-oss 120b for the example. This powerful language model has made big waves recently for its robust agentic and coding capabilities, making it perfect for demonstrating the awesome power of the machine.

Follow along in this tutorial for a deep dive into using vLLM with AMD GPUs. Readers can expect to leave with a full understanding of vLLM, gpt-oss, and each step required to run gpt-oss 120b using vLLM on a Gradient AMD powered GPU Droplet.