Did you ever have a PlayStation 3? A fun bit of trivia about that particular console is that in 2010, the US Air Force built the 33rd most powerful supercomputer in the world by building a PlayStation 3 cluster out of nearly two thousand consoles working together in order to process high-definition satellite imagery. While PlayStation 3 clusters in the 2000s were somewhat common, it's not likely something that will ever happen again.

How PlayStation 3 clusters were built

A marvel of computing

Back in the 2000s, Sony allowed users to install other operating systems on their PlayStation 3 through the "OtherOS" feature. This feature was an official one made by Sony and was used to install the likes of Yellow Dog Linux, Fedora, and Ubuntu, though it was then removed by Sony in 2010, apparently thanks to security reasons. Hackers still utilized custom firmware to continue using Linux on their PS3s, though it killed any mainstream adoption of the feature and also saw Sony face a class-action lawsuit where it eventually paid out $10.07 to each member of the class nearly a decade later.

According to Phys, the so-called "Condor Cluster" was capable of performing 500 trillion floating point operations per second (TFLOPS) and cost only five to ten percent of a "real" supercomputer that would have been capable of the same performance. Phys also says that for each PlayStation console that cost $400 for the USAF to procure, comparable technology would have cost $10,000 per unit, with the entire project costing $2 million. Not only that, but PS3s were incredibly power efficient, with the Condor Cluster apparently using only 10% of the energy of a supercomputer of comparable performance.

As for managing connections between the cluster of PlayStation 3s, Message Passing Interface (MPI) tools like OpenMPI or MPICH can be used for that purpose, with cluster software like TORQUE (Terascale Open-source Resource and Queue Manager), Slurm, or HTCondor being able to manage job scheduling and resource allocation. Finally, shared storage is set up so that all devices in the cluster can access the same data in parallel.

The PS3 used an IBM Cell processor, a processor that Valve's Gabe Newell once famously said was a "waste of everyone's time." While it was undoubtedly a powerful processor, the PS3's hardware was exceptionally different from what was in the Xbox 360 at the time, and arguably, different from what had been in basically any other computing hardware before it. With its central Power Processing Element (PPE) and seven Synergistic Processing Elements (SPEs) (technically eight, though one was disabled on the PS3 for improved yield), developers had to learn to build games for an entirely new architecture, which meant that advancements were being made years into the console's lifespan.

Famously, Sony Computer Entertainment CEO Kaz Hirai once said in an issue of the Official PlayStation Magazine that it was intentionally designed this way.

We don't provide the 'easy to program for' console that (developers) want, because 'easy to program for' means that anybody will be able to take advantage of pretty much what the hardware can do, so then the question is, what do you do for the rest of the nine-and-a-half years?

While the explanation makes a modicum of sense, it's not exactly a great justification for the difficulty that it caused developers. This same architecture that meant that the PS3 was different from the rest of the console and computing landscape also posed difficulties in clusters, though it also meant that it was a very capable machine for parallel processing.

As for why it made sense to use PS3s in a supercomputer cluster, the eight SPEs in the PS3 somewhat resemble modern GPU cores in the sense that they're specifically designed to handle a lot of simple math very quickly. It was essentially a way to cheaply get hardware that could process a lot of data in parallel, and with multiple machines in a cluster all joined together with an MPI bridge, made them a very attractive offering to would-be supercomputer builders.

Why it can never happen again

A mixture of locking down software and costs

The problem for the PS3's ability to be used as a supercomputer was two-fold, at least for Sony. For starters, it seems that the first successful jailbreak on the PlayStation 3 (which allowed consumers to enjoy pirated games and homebrew) made use of the system's OtherOS feature to disable security on the console. From a forum post more than 13 years ago, I found apparent chat logs with George Hotz, or geohot, where he says the following:

In OtherOS, all 7 SPUs are idle. You can command an SPU (which I'll leave as an exercise to the reader) to load metldr, from that load the loader of your choice, and from that decrypt what you choose, everything from pkgs to selfs. Including those from future versions.

What this essentially means is that Hotz was able to decrypt anything on the machine using one of the SPEs. An SPU is the core of the SPE, where the SPE also has its own local store of 256KB and a Memory Flow Controller, or MFC. In other words, geohot is alluding to using OtherOS as a way to derive the keys on the PS3, which had been teased by fail0verflow at Chaos Communication Congress 27C3. Hotz then released the private encryption key, though he was later sued by Sony. That case was later settled out of court.

Given that it seems like OtherOS was used as an attack vector for the PS3, it makes a lot of sense that Sony would want to lock it down and remove the feature. I also think that it did make Sony's job significantly harder to prevent its encryption keys from leaking, given that with OtherOS, you had full system access, enough that the USAF felt that it could build an entire supercomputer out of nearly two thousand of these machines. While Sony's hypervisor was supposed to prevent access to the RSX "Reality Synthesizer" GPU, the hypervisor enabling OtherOS was later exploited to gain full system access.

However, piracy and security aren't the only reasons that it's unlikely we'll ever see something like this again. Consoles, for the most part, are commonly sold at a loss in order to position them at an attractive price point. Companies such as Sony will often make significant amounts of money on game licensing, which means that the cost of the console can be lower than the cost to produce it because it's assumed that the person who purchases the console will also buy games. This puts more money back into Sony's pocket, so even if a console is sold at a loss, the games that will be sold should make up for it.

The problem with that is that when you have groups like the USAF building its supercomputer out of PS3 consoles, Sony is making a loss on each console sold without any way to recuperate that loss. Those consoles are never going to be used for gaming, which means that Sony is never going to get the money back that it lost on each console being sold. It's not that Sony is a small indie company starving for funds, but it means that there's no attraction to selling a console with a feature like OtherOS again because the console can't be sold at a loss to position it better in the market and it might even open the door to piracy in the future.

Nowadays, the console landscape is a bit different, but the practice of selling consoles at a loss is still somewhat prevalent. According to Sony chief financial officer Hiroki Totoki in 2021, the PS5 with a disc reader, costing $500, had sold at a loss initially but had eventually turned profitable, whereas the digital-only PS5 was still sold at a loss. Likewise, the Xbox One S and One X at the time were also sold at a loss, which is still apparently the same today. Nintendo, however, isn't known for the practice, with only the 3DS after its price cut and the Wii U known to have sold at a loss.