Apple has been ahead of the curve when it comes to AI thanks to its Apple Silicon chips, with many image generators, large language models, and more all running natively on even an M1 Mac for a long time. That's thanks to a combination of the NPU and the powerful graphics inside, but Apple's lead looks to be waning. With its M4 chip launching to claims of 38 TOPS, there's a lot to unpack about what those numbers refer to, and how Qualcomm's Snapdragon X Elite stacks up.

I believe that Apple's AI lead is beginning to drop off, and companies are starting to catch up. With Intel and AMD offerings around the corner and Qualcomm's Snapdragon X Elite mere weeks away, Apple might need to start watching its back.

"38 TOPS" doesn't meet anything by itself

AI has multiple measurement criteria

When discussing AI performance, TOPS (trillion operations per second) is a critical metric, and understanding how different data types like INT4, INT8, and FP16 contribute to this is important.

For context, INT4 and INT8 are integer formats, with INT4 being a 4-bit integer and INT8 being an 8-bit integer. These are typically used in neural network computations where precision can be sacrificed for speed and power efficiency. The smaller bit-width in INT4 allows for more operations per cycle but at the cost of lower precision, making it ideal for applications where high throughput is more crucial than high accuracy. On the other hand, INT8 strikes a balance, offering more accuracy than INT4 while still maintaining a higher computational speed compared to more precise formats.

FP16, or half-precision floating-point, is another beast altogether. This 16-bit floating-point format provides a much broader range of values than INT4 or INT8, as it includes both significant digits and an exponent. This allows FP16 to handle much more complex calculations and algorithms, which require a high degree of numerical precision. This is crucial for tasks involving detailed graphics or scientific computations where accuracy cannot be compromised. Comparing the TOPS values among these formats is not just about raw speed; it’s about the suitability for specific applications. For instance, while INT4 and INT8 may provide higher TOPS, FP16’s ability to deliver more detailed and nuanced computations makes it indispensable for more precision-critical applications.

With all of that out of the way, we can make a number of assumptions about what Apple is talking about here. We assume that the TOPS number Apple is advertising is calculated using INT8 precision. Apple's A17 Pro chip, said to be capable of 35 TOPS, was measured using FP16, but the M3, using a lot of the same parts (including cores) as the A17 Pro, was said to "only" support 18 TOPS. This would be explained by a change in precision and would be closer in line with what the wider industry is using nowadays. For example, AMD's Ryzen 7 8700G with its neural engine supports 18 TOPS too, and that uses INT8 as well.

In that same regard, even a jump from 18 TOPS to 38 TOPS is pretty substantial, but the only way it surpasses Qualcomm is if this was measured using FP16. I can't imagine that Apple would change their measuring convention back to FP16 to buck the trend of the industry measuring using INT8 in order to have a worse number, and given that Qualcomm's figures are calculated at INT8 precision, it's easy to connect the dots and see that Apple is almost certainly falling behind.

There's more to AI than just TOPS

Most AI apps don't even use Apple's Neural Engine

It's important to be clear about something when it comes to Apple and AI, though: most of the AI applications currently available that run on Mac don't even use Apple's neural engine. LM Studio runs on the CPU and GPU, and so do pretty much all of the open implementations of Stable Diffusion. In that sense, Apple's Neural Engine TOPS numbers aren't even comparable given that they're not representative of the actual AI workloads these devices are capable of.

However, with Intel and AMD, both of those companies actively support the development and usage of their neural processing units. Intel natively supports over 500 models already running on its NPUs and is actively helping developers support theirs, and AMD is in the same boat. Apple, meanwhile, only supports CoreML, which is an API that can use the neural engine in Apple Silicon but does not fully support most of the technology that makes LLMs and other models possible, such as quantization.

As well, because of unified memory, Apple's GPUs can still access the same memory space at the same speed as the neural engine can, meaning that the same storage and memory-related bottlenecks remain no matter how AI models are executed. Because of all of this, comparing Apple's NPU to other NPUs doesn't really mean much even if the company still insists on sharing TOPS numbers that are comparable to others in the industry.

Apple has fallen behind in AI

Not just in hardware, but in software too

Apple has been behind the AI curve when it comes to software for arguably a long time, particularly with Siri, arguably one of the worst voice assistants out there. It's fine and gets the job done, but there's no AI smarts behind it. Likewise, Google has been arguably running rings around Apple when it comes to on-device AI, such as Google's Now Playing, Magic Editor, and much more.

Of course, Apple still has AI in its laptops and smartphones, but it's nowhere near what rivals are doing currently. The company has been working on LLMs in the background with R&D continuing in that direction, though if that will end up coming to iPhones or MacBooks isn't clear. Google has reportedly been in talks with Apple to bring Gemini to the company's products, so it seems that things are up in the air. Plus, Apple may end up building its own server infrastructure to support cloud-based AI features, just like Samsung and Google have done.

Apple has a lot to learn if it wants to catch up in the AI space, but the company tends to be quiet in its approach to new technologies. It doesn't need AI to succeed (clearly), but it looks weirder and weirder as time passes without so much as a whisper of anything about it. Apple even started the NPU race in its MacBooks, but there's a chance by the end of this year that a Mac is no longer the best place to run AI applications on the go.