Ask any home lab enthusiast about the ideal way to build your server, and they’d always recommend bare-metal virtualization platforms over Type-2 hypervisors. For starters, passing through devices, especially components relying on the PCIe interface, is a lot easier on Type-1 hypervisors installed directly on top of your hardware. In a similar vein, bare-metal setups are always superior in the performance and security departments, as you don't have to contend with an underlying OS.

But what if you wanted to live on the edge and configure a home lab distro on top of your existing operating system? As someone who has worked on wackier projects before, running a Proxmox virtual environment inside a VM on Windows 11 seemed like the perfect experiment to satisfy the tinkerer residing in my brain. So, here’s a log of all my observations after building an unholy Proxmox home lab using a Type-2 hypervisor.

Choosing the hypervisor

And creating the Proxmox VM

Since I attempted the experiment on Windows 11, I was low-key tempted to use Hyper-V, as it’s possible to enable it even on the Home versions of the OS. Unfortunately, Hyper-V is more of a Type-1 hypervisor and choosing it would go against the spirit of this article. In the end, I picked VirtualBox because it’s a lot simpler to navigate than VMware Workstation Pro. It's also the Type-2 hypervisor I'm most familiar with, so I figured I wouldn't encounter too many issues with running Proxmox on it. Or at least, that's what I thought initially.

Hardware-wise, I decided to ditch my dual-CPU Xeon build and went with my daily driver, which houses a 6-core, 12-thread Ryzen 5 5600X processor and 32GB of DDR4 memory. All-in-all, it's a standard - if somewhat outdated - gaming system that can run a couple of VMs inside VirtualBox.

After launching VirtualBox, I created a new virtual machine, added the Proxmox ISO file, and allocated six cores and 24GB of memory. Since I was attempting to run a virtual machine inside another VM, the entire project hinged on nested virtualization. Technically, VirtualBox supports this facility, but much to my chagrin, the Enable Nested VT-x/AMD-V option was grayed out inside the Type-2 hypervisor, meaning it was time to put on my tinkering hat.

Enabling nested virtualization

And booting my first VM inside the virtualized Proxmox environment

When I began surfing the Internet for a fix, I chanced upon the terminal command to turn on nested virtualization for a VirtualBox VM. After using the cd keyword to switch to my VirtualBox folder, I ran the VBoxManage modifyvm Proxmox --nested-hw-virt on command in the Windows terminal to forcibly enable nested virtualization for the Proxmox instance. I also switched the Ethernet driver to Bridged Adapter to access the Proxmox web UI from a browser.

With that, I deployed the VM and followed the procedure to install Proxmox on a virtual drive. Although the installation succeeded, I noticed an error message citing that the virtual machine had failed to detect hardware-accelerated KVM virtualization, which I decided to tackle later. Once the installation was complete, I headed to the Storage settings and removed the Proxmox ISO attachment under the Controller: IDE section to force the virtual machine to boot into the newly-installed Proxmox environment.

Soon, the VM restarted and the IP address of the PVE web UI appeared inside the VirtualBox Console. After transferring the Ubuntu ISO to Proxmox, I used it to deploy a new virtual machine inside the web UI. Unfortunately, the virtual machine had failed to leverage KVM-based virtualization provisions and was stuck in a boot loop.

One potential fix involved enabling KVM inside the Acceleration tab of the System settings. It also turned out that the Windows Hypervisor Platform feature conflicted with the hardcore virtualization options of VirtualBox. So, I disabled it using the Control Panel, and voilà, my Ubuntu virtual machine worked without issues. Installing Ubuntu was a cinch, though I had to set the Promiscuous mode to Allow all inside the Network tab to connect my Proxmox-based Ubuntu virtual machine to the Internet.

Running multiple virtual guests inside the virtualized Proxmox instance

Including another Proxmox VM

Performance-wise, the Ubuntu VM wasn’t as terrible as I’d initially expected, though it was definitely slower and less responsive than a bare-metal Proxmox setup. I also bumped the number of allocated cores (or rather logical cores, if you want to go technical with the details) to 10 and deployed Debian, OpenMediaVault, and Elementary OS VMs. Running all of them in tandem took some toll on the Proxmox virtual machine, though I didn’t have to resort to overprovisioning the cores… at least, not yet.

I also spun up some containers using the Proxmox VE-Helper Scripts. But since neither the VMs nor the containers were utilizing the cores (or even RAM) to their full extent, I could run them without crashing the virtualized Proxmox environment. For the final round of tests, I gave in to my intrusive thoughts and deployed another Proxmox VM inside the existing PVE server (after stopping the other virtual guests, of course). Despite my best efforts, I was unable to get the virtual Proxmox guest working. With that, it was time to look back and assess the feasibility of this project.

Should you put together a Proxmox home lab inside Windows 11?

If you’re planning to get into home server projects, running Proxmox inside Windows 11 isn’t a very bright idea because of the massive performance overhead caused by Microsoft’s flagship OS. That said, you could try replicating a Proxmox environment inside a Type-2 hypervisor if you wish to test it out without purchasing another system. Just make sure your PC has enough memory, disk space, and CPU cores and threads before you go around building complex server projects.