In the continuously growing world of artificial intelligence, Neural Processing Units (NPUs) have emerged as a critical aspect, allowing for advancements in on-device AI processing and technology. These specialized chips are made to handle intense computational demands specific to neural networks, offering performance and efficiency that simply can't be matched by your conventional consumer-grade CPU. They're also required for Copilot+, but what exactly are NPUs, and how do they fit into the broader landscape of AI?

The architecture of an NPU

Breaking down the design

At the heart of the design of an NPU lies an architecture centered around parallelism and data locality. Unlike general-purpose CPUs and GPUs, which are designed to handle a broad range of tasks (though GPUs handle a significant amount in parallel as well), NPUs are specifically tailored for the unique demands of neural network computations.

With a focus on parallelism, NPUs feature a large array of processing cores. These cores are capable of executing multiple instructions concurrently, which benefits the inherently parallel nature of neural network operations. Neural networks often involve performing the same operation on different data points simultaneously, which makes sense to execute in parallel. On top of that, each core in an NPU can handle vector and matrix operations, which are everywhere in neural network calculations. By distributing these operations across many cores, NPUs can achieve significant performance gains.

NPUs may employ Single Instruction, Multiple Data (SIMD), or Multiple Instruction, Multiple Data (MIMD) architectures. SIMD allows a single instruction to be applied to multiple data points concurrently, while MIMD enables different cores to execute different instructions simultaneously. This gives flexibility for optimizing operations in neural network layers.

On top of parallelism, NPUs incorporate specialized memory hierarchies. These hierarchies are designed to ensure that data is as close as possible to the processing units, thereby reducing the time and energy required to fetch data. This may mean that there is on-chip SRAM, which is faster, low-latency memory for storing intermediate data, weights, and activations during neural network processing. This has benefits as keeping frequently accessed data on-chip means that NPUs minimize the need to access slower, off-chip DRAM, significantly reducing latency and power consumption.

With that, NPUs may also feature multi-level cache systems, with L1 and L2 caches providing additional layers of fast-access memory. These caches store recently accessed data and instructions, further enhancing data locality and reducing access times. Intel's Meteor Lake Neural Compute Engines (like cores in an NPU) do not have an L1 or L2 cache but do have access to 2MB of SRAM per NCE.

When it comes to computation, dedicated Tensor Processing Units (TPUs), which are specifically designed to handle the mathematical operations fundamental to neural networks, handle the bulk of the workload. Neural networks rely heavily on matrix multiplications and convolution operations, and the TPUs within NPUs are optimized for these operations, featuring hardware accelerators that can perform large matrix multiplications and convolutions. These accelerators make use of data parallelism by executing operations across multiple data points simultaneously.

NPUs can process various layers of a neural network at a time, from dense fully-connected layers to convolutional layers. Parallelism allows NPUs to process vast amounts of data, accelerating both the training and inference phases. Many NPUs utilize systolic arrays, a parallel computing architecture that's built to handle the repetitive and regular data flow patterns of neural network operations. Systolic arrays consist of a grid of Processing Elements (PEs) that pass data through the array, enabling high-throughput and low-latency computations and speeding up matrix multiplications.

Finally, NPUs often include Direct Memory Access (DMA) controllers that make it so that transferring data between memory and processing units can happen without requiring the computational cores to do anything. This allows cores to focus on computation, which in turn improves overall performance.

An NPU is going to become increasingly important

At least for on-device AI

Your next laptop will probably have an NPU, at least if it's powered by a recent Intel or Ryzen chip. Qualcomm also has its own NPU, and Apple's own neural engine is an NPU as well. They're becoming increasingly important for managing on-device AI workloads in a more power-efficient manner, though every NPU is different. As it stands, we have a ton of different pieces of NPU hardware that developers are also looking to use, and they have different capabilities.

As time goes on, we expect that NPUs will be standardized a little bit more, but right now, it's a bit like the wild west. There are a lot of new standards and a lot of new hardware being tested, so everyone's experience is different.