Monitoring your home lab and keeping tabs on your log files can be draining, but there are multiple automated solutions aimed at making it easier and to help you visualize what's going on. These tools include Grafana Alloy, the successor to Promtail, and Prometheus, a system monitoring tool. Armed with Grafana, Alloy, and Loki, along with Prometheus for collecting my server system data, I built the ultimate system monitor that can keep tabs on all of my machines.
Setting this up was fairly simple and took five minutes following the official Grafana documentation on deploying this kind of configuration. I then took things a step further, to demonstrate how one could collect data from their Proxmox host.
Why go to all the trouble of monitoring your system?
Sometimes, it's easier to have it all in one place
For home labbers with a lot of different virtual machines, servers, and containers, it can be unwieldy to keep tabs on all of them and ensure everything is still running smoothly. For example, I have a Frigate container that occasionally just starts using 100% of the memory assigned to it, 100% of the CPU assigned to it, and 100% of the swap. I only realize when I either see that my Frigate card is no longer working in Home Assistant, or when I walk through my hallway and hear the fans of my mini PC at full blast.
However, with monitoring, it'd be super easy to see from a handful of dashboards in Grafana, and I wouldn't need to check individual containers, VMs, or even the Proxmox host UI. We'll get to that last part as it's a bit different, but monitoring any individual container or virtual machine is incredibly easy. And, once you've done it, you can then add more devices, and treat Grafana as a centralized resource to keep tabs on all of your software.
For a person who's only getting started, something like this isn't needed. However, once you find that keeping tabs on your software that's running is getting a bit out of hand, that's where an external service comes in. There's a bit of setup to do it right, but the end result is wonderful.
How to monitor any Linux system in Grafana
There's a premade Docker Compose file with everything you need
These steps are taken from the Grafana documentation, and you can modify the configuration files to suit your exact needs. The end result with a dashboard will use all of the data exported, though, so be aware that changing things will require you to create your own dashboard or modify the existing one. To get started, first ensure that you have Docker and Git installed, and run the following commands in your terminal:
git clone https://github.com/grafana/alloy-scenarios.git
cd alloy-scenarios/linux
docker compose up -dThat's all you have to do! The Docker container will start, launching Grafana, Prometheus, Loki, and Alloy. You can then go to your Grafana instance at (VM_address):3000 and see the data coming in. There's a pre-configured JSON you can import as a dashboard which will visualize of the data collected, and this is enough to deploy across any Linux environment capable of running a container.
But what about Proxmox? There's a great, easy to use tool that utilizes the Proxmox API to pull data from your own nodes.
Monitoring Proxmox in Grafana
Just one more Docker container
We'll again be utilizing Docker, alongside the Prometheus PVE Exporter project. We can also use the same Grafana stack that we used for monitoring a Linux VM, so if you have that up and running, leave it up and running for now. We'll feed data into that particular instance and visualize it there, too.
To deploy this using Docker, first create a folder that you want to save your Proxmox credentials in, and in there, create a file called "pve.yml". You'll then want to create a new user on your Proxmox host with the PVEAuditor role, as this will allow it to only read, not write, your system metrics. After that, you'll either want to create a token for that user to log in, or use a username and password in PVE Exporer. Either works, but a token is more secure as you can revoke it at any time.
Once you have those configured, add this to your pve.yml:
default:
user: prometheus@pve
password: sEcr3T!
# Optional: set to false to skip SSL/TLS verification
verify_ssl: true
Ensure that you use proper indentation. Run the following command to download the Docker image:
docker pull prompve/prometheus-pve-exporter
Then run your container, where we'll point it to our pve.yml file for connecting to Proxmox:
docker run --init --name prometheus-pve-exporter -d -p 0.0.0.0:9221 -v /path/to/pve.yml:/etc/prometheus/pve.yml prompve/prometheus-pve-exporter
In your browser, if you navigate to "(VM IP):9221/pve?target=(proxmox IP)" you should see a bunch of metrics collected from Prometheus. If you see an internal server error, use the "docker logs" command to see what failed. But how do we get the data into Prometheus?
Navigate back to the folder you created the Grafana testing instance in, and modify the "prom-config.yaml" file. Mine looks like the above, but the GitHub for the PVE Exporter has a sample config you can add. Once you've added it, type the following:
docker compose down
docker compose up -d
This will start your Docker container, pulling in the new Prometheus configuration which scrapes the data from PVE Exporter. Finally, we can represent our data using the "Proxmox via Prometheus" Grafana dashboard, which you can import to your existing Grafana instance.
That's it! You can track multiple Proxmox nodes from PVE Exporter to keep tabs on them, and it supports clusters as well if you have any deployed. It's a quick and easy way to keep tabs on your self-hosted servers, and it's easy to look at, too.
There are many ways to expand on this further, and if you use Home Assistant, you can set up automatic screenshots in Grafana to pull them into a Home Assistant dashboard. There's a lot of room for improvement and testing here, and it's worth it for a one-stop shop to track everything you self-host.