Like most households, mine is no stranger to smart assistants. They're great for completing basic commands to help take the burden of carrying out mundane tasks such as turning on and off appliances, bulbs, and even configuring stuff on schedules. I wasn't always a fan, and even when purchasing our first Amazon Echo device, I still wasn't entirely convinced. They're smart, sure, and when paired with hubs such as the Philips Hue Bridge, you can have some fun with lighting controls.

But I quickly discovered how limited these assistants are without the right cloud-based support. Then there's the fact that without the active connection, they're almost useless. Couple that with all of our recorded data being shared with and stored on various servers owned by big tech giants, and it's a recipe for a privacy disaster. Even when disabled and attempting to give Alexa a command, the assistant will still respond, proving that the microphone is never "off".

That's why I eventually migrated away from these proprietary assistants and built my own using a large language model (LLM), Proxmox server, Home Assistant, and Home Assistant Voice Preview Edition.

What's wrong with other smart assistants

They're not that smart

Aside from not really understanding my wife's ever-so-slight Caribbean accent, Alexa would flat out not understand our commands. Asking Alexa to turn the living room lights off would end up blinding us both before heading to both with enough lumens to land an Airbus A380. This ended up forcing us to either use our phones to fire up the Philips Hue and other apps to control bulbs individually, or use Home Assistant.

Then there's the fact that we found Alexa (and other assistants) to be more of a miss than a hit. They couldn't really do all that much. Sure, we would be reminded about all the optional extra services we could pay for on top of what we already subscribed to, but aside from asking for a joke every so often, we found that we didn't really utilize the smart assistants. This was largely due to their unreliability with handling vocal input, but Alexa didn't play well with Jellyfin, for instance.

There's also a lack of freedom when it comes to these devices since you're likely going to want to use them over something dedicated, such as a pair of high-quality speakers. The Sonos Era 100 is an excellent speaker and can be used to create some epic sounds throughout the home. The best part? You can fully integrate them with Home Assistant using the Voice Preview Edition or some other connected device.

Lastly, I no longer trust the big tech giants. I used to store everything in the cloud, have all my personal details saved and backed up, and even loaded up my storage with personal photos and videos. I've since migrated all my memories from cloud storage, and now self-host much more than before. Leaping from smart assistants to my own voice controls is a smaller pill to swallow.

Adding voice to Home Assistant

Expanding the scope of my assistant

Depending on what you're using to host and run Home Assistant, which can be anything from an AMD EPYC server to a Raspberry Pi single-board computer (SBC), getting everything up and running is largely painless. It's important to consider what you plan to automate through Home Assistant and all the available integrations. Using HA alone isn't going to provide much to the "smart home," but once you've kitted the house out with IoT devices, sensors, and have a running LLM, you're well on your way to creating the ultimate smart assistant.

For voice controls, the easiest way to add this to Home Assistant is through the official Home Assistant Voice Preview Edition. It's a compact device featuring several high-quality microphones and advanced internal components to accurately capture voice commands and relay them through the software. But really, all you need is a free USB port on the device running Home Assistant and a spare USB microphone. If you don't have one at hand, they can be bought for less than $10. After enabling advanced mode within HA, Whisper, Piper, openWakeWord, Duck DNS, Dnsmasq, Studio Code Server, and Assist Microphone can all be installed.

From there, using your own microphone or some other input device is incredibly easy, once everything is configured and HTTPS is active. Our in-depth guide can quickly walk you through all the necessary steps to get everything working. If you're not fancying all the steps to get voice commands added to Home Assistant, I highly suggest using some integration or the official Voice Preview Edition for a more convenient setup process. Using a locally-hosted LLM can unlock additional commands and options for using smart assistants.

Instead of simple, basic commands like "Turn off all the lights," you can now specify devices to leave on, such as "Turn off all the lights except for the TV blacklight." The local LLM will understand the request and process it, supplying Home Assistant with data it can then interpret as commands. As for actually running the LLM, you can use plenty of available models depending on your hardware, though the actual results will vary between configurations and models used. I'm still learning the ropes with getting everything under one roof, but it's all working and can be used as a base for adding more to our HA install.