Running AI models locally provides enhanced privacy, reduced latency, and complete control over your infrastructure. In this guide, we’ll walk through setting up DeepSeek models (like DeepSeek-R1) inside a Proxmox virtual machine with GPU passthrough. This configuration allows the AI model to access your NVIDIA GPU directly for significantly improved performance.

In this tutorial you will learn:

• How to configure IOMMU and GPU passthrough on your Proxmox host
• How to create a properly configured VM for GPU computing
• How to install NVIDIA drivers in the VM
• How to install and run Ollama with DeepSeek models
• How to choose between different DeepSeek model sizes based on your hardware
• How to troubleshoot common GPU passthrough issues

Setting Up GPU Passthrough on Proxmox

Before running DeepSeek-R1 on Ollama, we need to properly configure GPU passthrough so that our virtual machine can directly access the GPU. This involves enabling IOMMU, configuring the system to use VFIO drivers, and setting up the VM correctly.

1. Enable IOMMU and configure GRUB: First, we need to enable IOMMU support on the host system

   # nano /etc/default/grub

   Find the line with GRUB_CMDLINE_LINUX_DEFAULT and add the IOMMU parameters:

   GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"

   After saving the file, update GRUB and reboot:

   # update-grub
   # reboot

2. Verify IOMMU is working: After the system reboots, check if IOMMU is properly enabled

   # dmesg | grep -e IOMMU -e DMAR

   You should see output indicating that IOMMU is enabled and functioning.

   👁 Check if IOMMU is Enabled on Proxmox
   

3. Configure VFIO drivers for your GPU: Configure the system to use VFIO drivers for your NVIDIA GPU
   First, find your GPU’s vendor and device IDs:

   # lspci -nn | grep -i nvidia

   The output will look something like:

   01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA102 [GeForce RTX 3080] [10de:2206] (rev a1)
   01:00.1 Audio device [0403]: NVIDIA Corporation GA102 High Definition Audio Controller [10de:1aef] (rev a1)

   In this example, the vendor and device IDs are `10de:2206` (for the GPU) and `10de:1aef` (for its audio component). You need both for proper passthrough.

   Method 1: Automatic configuration with modprobe:

   # echo "options vfio-pci ids=10de:2206,10de:1aef" > /etc/modprobe.d/vfio.conf
   # update-initramfs -u
   # reboot

   Use the actual vendor:device IDs from your system as shown in the lspci output. After running update-initramfs, the changes will be applied on the next boot.

   Method 2: Manual binding (if automatic method doesn’t work):
   If the automatic method doesn’t work after rebooting, you can manually bind the GPU to VFIO drivers using a systemd service:
   First, create a script that will handle the binding process:

   # nano /usr/local/bin/vfio-bind-gpu.sh

   Add the following content:

   #!/bin/bash
   
   # The full PCI address needs the domain part "0000:" added before the addresses from lspci
   # For example, if lspci shows "01:00.0", use "0000:01:00.0" here
   GPU_IDS="0000:01:00.0 0000:01:00.1"
   
   modprobe vfio-pci
   
   for dev in $GPU_IDS; do
    if [ -e /sys/bus/pci/devices/$dev/driver/unbind ]; then
    echo -n "$dev" > /sys/bus/pci/devices/$dev/driver/unbind
    fi
    if [ -e /sys/bus/pci/drivers/vfio-pci/bind ]; then
    echo -n "$dev" > /sys/bus/pci/drivers/vfio-pci/bind
    fi
   done

   Important: The lspci command shows addresses like “01:00.0”, but the sysfs filesystem requires the domain prefix “0000:”. Always add “0000:” before each PCI address from lspci when using them in this script.

   Make the script executable:

   # chmod +x /usr/local/bin/vfio-bind-gpu.sh

   Let’s create a systemd service to run this script at boot:

   # nano /etc/systemd/system/vfio-bind-gpu.service

   Add the following content:

   [Unit]
   Description=Bind NVIDIA GPU to VFIO at Boot
   After=network.target
   
   [Service]
   Type=oneshot
   ExecStart=/usr/local/bin/vfio-bind-gpu.sh
   RemainAfterExit=yes
   
   [Install]
   WantedBy=multi-user.target

   Enable and start the service:

   # systemctl daemon-reload
   # systemctl enable vfio-bind-gpu.service
   # systemctl start vfio-bind-gpu.service

Creating a VM for GPU Computing

Now that we have GPU passthrough set up on the host, we need to create a VM that’s properly configured to use the passed-through GPU. We’ll use the Proxmox web interface for this process.

1. Access the Proxmox web interface: Open your web browser and navigate to your Proxmox host (https://your-proxmox-ip:8006) and log in with your credentials.
2. Create a new VM: In the Proxmox web interface:
   1. Select your Proxmox node in the server view (left panel)
   2. Click the “Create VM” button at the top right
   3. In the “General” tab:
      • VM ID: Choose a unique ID (e.g., 9100)
      • Name: Enter a descriptive name (e.g., “ollama-vm”)
      • Click Next
3. Configure OS settings: In the “OS” tab:
   1. Select “Use CD/DVD disc image file (iso)”
   2. Storage: Choose your ISO storage (e.g., “local-disks”)
   3. ISO Image: Select your Linux distribution ISO (e.g., “debian-12.9.0-amd64-netinst.iso”)
   4. Type: Linux
   5. Version: 6.x – 2.6 Kernel
   6. Click Next
4. Configure system settings: In the “System” tab:
   1. Graphics card: Set to “Default”
   2. Machine: Select “q35” (this is crucial for PCI passthrough)
   3. BIOS: Select “OVMF (UEFI)”
   4. Add EFI Disk: Check this option
   5. EFI Storage: Select your ZFS storage pool (e.g., “zfs_raid1_storage”)
   6. Pre-Enrolled Keys: Uncheck this option (disables secure boot)
   7. Click Next
5. Configure disk: In the “Disks” tab:
   1. Storage: Select your ZFS storage pool (e.g., “zfs_raid1_storage”)
   2. Disk size: Set to “100” GB
   3. Format: qcow2
   4. Click Next
6. Configure CPU: In the “CPU” tab:
   1. Sockets: 1
   2. Cores: 8 (adjust based on your available resources)
   3. Type: host (for best performance)
   4. Click Next
7. Configure memory: In the “Memory” tab:
   1. Memory: 32768 MB (32GB) for 7B parameter models, or 65536 MB (64GB) for 14B parameter models
   2. Click Next
8. Configure network: In the “Network” tab:
   1. Bridge: vmbr0
   2. Firewall: Checked (if you want firewall protection)
   3. Click Next
9. Confirm settings: Review your settings and click “Finish” to create the VM.
10. Add the GPU to the VM: After creating the VM:
    1. Select your newly created VM in the left panel
    2. Go to the “Hardware” tab
    3. Click “Add” → “PCI Device”
    4. Select your NVIDIA GPU from the dropdown
    5. Check “All Functions” (to include both GPU and audio components)
    6. Check “PCI-Express” (required for modern GPUs)
    7. Click “Add”
11. Verify hardware configuration: Your hardware tab should now show:
    • Memory: Your allocated memory (e.g., 32 or 64 GiB)
    • Processors: Your configured CPU cores
    • BIOS: OVMF (UEFI)
    • Machine: q35
    • SCSI Controller: VirtIO SCSI
    • CD/DVD Drive: Your Debian/Ubuntu ISO
    • Hard Disk: Your ZFS storage disk (e.g., 100G)
    • Network Device: virtio with your bridge
    • EFI Disk: Your ZFS storage EFI disk (1MB)
    • PCI Device: Your GPU (e.g., 0000:01:00,pcie=1)

    👁 Verify VM setting for GPU Computing and Passthrough
    

12. Start VM and install Linux: Click the “Start” button to boot the VM and install your Linux distribution.

Hardware Recommendations:

• For 7B parameter models: 8+ CPU cores, 32GB RAM, NVIDIA GPU with 8GB+ VRAM (RTX 3060 or better)
• For 14B parameter models: 12+ CPU cores, 64GB RAM, NVIDIA GPU with 16GB+ VRAM (RTX 3080 or better)

Installing NVIDIA Drivers in the VM

Once your VM is running, you need to install the NVIDIA drivers to access the GPU.

1. Add non-free and contrib repositories: Enable the necessary repositories for NVIDIA drivers

   # apt update
   # apt install -y software-properties-common
   # add-apt-repository contrib
   # add-apt-repository non-free
   # apt update

2. Install the NVIDIA driver and CUDA toolkit: Install the required drivers and CUDA on your VM

   # apt install -y nvidia-driver nvidia-cuda-toolkit firmware-misc-nonfree
   # reboot

   After the VM reboots, verify that the GPU is recognized:

   $ nvidia-smi

   You should see output displaying your GPU model, driver version, and other information.

   👁 Verify that the GPU is recognized
   

Installing and Running Ollama with DeepSeek Models

Now that our VM can access the GPU, we can install Ollama and run DeepSeek models.

1. Install Ollama: Install Ollama on your VM

   $ curl -fsSL https://ollama.com/install.sh | sh

   Verify the installation:

   $ ollama --version

2. Choose and run a DeepSeek model: Download and run your preferred model
   For the 7B parameter model (faster, less resource-intensive):

   $ ollama pull deepseek-r1:7b
   $ ollama run deepseek-r1:7b

   For the 14B parameter model (higher quality, more resource-intensive):

   $ ollama pull deepseek-r1:14b
   $ ollama run deepseek-r1:14b

   This will download the model (which will take some time depending on your internet connection) and then start an interactive chat session. The 7B model offers a good balance of speed and quality, while the 14B model provides better responses but requires more GPU memory and runs slower.

   👁 Test the DeepSeek. Have it for example to write poem about GNU/Linux
   

3. Monitor GPU usage: You can monitor GPU usage during inference

   $ nvidia-smi -l 1

   This will show GPU usage, memory consumption, and temperature in real-time, refreshing every second. This can help you determine if your hardware is sufficient for your chosen model size.

   👁 Monitor GPU usage
   

Troubleshooting Common Issues

1. GPU not detected in VM: If your GPU isn’t being detected

   $ lspci | grep NVIDIA

   If you don’t see your GPU listed, check:
   1. IOMMU group isolation
   2. Whether the GPU was properly unbound from the host
   3. VM configuration (should be q35 machine type with UEFI)

2. NVIDIA driver issues: If the driver doesn’t load properly

   $ dmesg | grep -i nvidia

   Look for any error messages. Common issues include:
   1. Secure boot interference (ensure it’s disabled)
   2. Incompatible driver version
   3. GPU not fully passed through (both video and audio functions need passthrough)

Conclusion

Setting up DeepSeek models with GPU passthrough in Proxmox allows you to run powerful AI models locally with optimal performance. This configuration provides better response times, privacy, and control over your AI infrastructure. While the initial setup process may seem complex, the benefits of having a locally-running LLM with full GPU acceleration are substantial.

By following this guide, you’ve created a dedicated AI virtual machine that can be backed up, cloned, and migrated just like any other Proxmox VM, while still maintaining high-performance GPU access. The flexibility to choose between different model sizes (7B or 14B) lets you balance performance and quality based on your hardware capabilities and needs.

Our testing shows that on a system with an RTX 3080 GPU, the 7B model provides snappy responses with minimal latency, while the 14B model offers higher quality outputs at the cost of slightly slower response times. You can now explore other models supported by Ollama or customize DeepSeek for your specific use cases.