As we move into an era of artificial intelligence, we're going to see computers that come with a performance measurement called "TOPS." This stands for "Trillion Operations Per Second," and it's a measure of how well a PC can handle artificial intelligence-based tasks. You'll likely see this crop up more often in the future, as we move away from sending requests to an AI model running on an outside server, and more toward running those models on our own hardware.

You'll see the term "TOPS" pop up when discussing AI PCs, which are computers that pack an NPU in it that can process AI-based processes. But as we were occupied looking at the TOPS war between the major NPU chip manufacturers, BeagleBoard released a single-board computer (SBC) that could output 4 TOPS. While 4 TOPS isn't a huge number (Intel's Meteor Lake can output 34 TOPS, for example), this is still an exciting venture, as SPCs with an AI processor are huge for the hobbyist community.

Bringing AI processes to local hardware helps with privacy

No more beaming your data to an external server

So why are we bothering to bring AI processes to local hardware? There are a few advantages, but one of the biggest is that running the AI on your own hardware is a lot better for your privacy than sending it to a server. Privacy-minded users might wince at the idea of having all their data sent to an external server to be processed, and that goes double if their replies will be used to train and enhance an AI model to respond to others better. Talking to an AI always feels opaque; where is the data you're sending going, and what's it being used for?

The easiest way to solve this problem is to own the hardware doing all the processing. Now you know exactly what it's being used for; after all, you set up the hardware. And you don't have to share the service with anyone else, meaning there are no slowdowns or outages.

You can create an AI shell that runs independently from your PC

No need to keep your PC running all the time

Okay, so running an LLM on an SBC helps with privacy. However, the same could be said for running LLMs on any local hardware. Regardless of whether it's on a Raspberry Pi or your main PC, it still respects your privacy and gives you control over how the data is used.

The key with LLMs on an SBC, however, is that you can make the SBC your LLM's "home" on your network. If you run a chatbot on your PC, that bot will only handle your requests as long as your computer is turned on. That might be alright for some, but for those who turn their PC off for the night, it's not ideal.

With an SBC, you don't need to turn it off. SBCs don't require a lot of power to run, so you can have it running on your desk 24/7 without issue. Connect it to your network, and you can give it requests no matter what device you're on. You could even have it respond to requests from outside the network so you can ask it things while you're out and about.

The SBC community loves sharing its projects with others

Follow the instructions to make your very own AI

The best part about AIs on SBCs is how open-source the community is. Chances are if you see a cool Raspberry Pi project, the designer has not only written out a full guide on how to make it, but will also give you all the code if you ask nicely (and if they haven't already uploaded it somewhere for everyone to use). As such, this allows people to develop an AI, make a project, and then offer up both the build materials and the AI model for other people to download and try.

As such, if AIs on SBCs take off, you could essentially go shopping for an LLM that does the job you want it to. Once you find one, you can slap it on your SBC and set it running as you like it. In a way, it'd be like "homebrew AIs" designed to achieve specific goals that people can use for themselves.

AI on SBCs: it won't be powerful, but it will be fun

There isn't much chance that SBCs will take over as the AI hardware of choice. Right now, raw processing power is king, and computers do that best. But for the hobbyist who wants to tinker with their own AI and make a fun little project, its inclusion on SBCs will be huge.