PC graphics have hit a standstill. Of course, we're still getting beautiful games that push the technical envelope on PC, from Doom: The Dark Ages with its real-time path tracing to Nvidia's Multi-Frame Generation (MFG) that's available in dozens of games. We've hit somewhat of a wall this generation, though. Virtualized geometry like Unreal Engine's Nanite is available in dozens of games, and ray tracing has progressed so far that it's a required feature in titles like Doom: The Dark Ages.

Although real-time rendering has reached solid footing this generation, there's a new feature that's been bubbling under the surface for a couple of years. It's not something you'll read about on a Steam page or hear about in a keynote presentation, but a small set of enthusiasts have been salivating over its potential since the concept was introduced by Nvidia two years ago. I'm talking about Neural Texture Compression (NTC), which has been a nut graphics researchers have tried to crack for over two years. And it just had a major breakthrough.

Taking a look at Neural Texture Compression

It's been around for a few years

Neural Texture Compression is a big deal, and you can see that just by understanding what it does. When you play a game, textures are loaded from compressed files stored alongside the game. Those files are sent over to your GPU, stored in VRAM, and decompressed. There are two sources of performance and/or quality loss there. There's the time it takes to read and decompress the texture from your hard drive, as well as the demand on VRAM once that texture is ready for presentation.

That's led to issues as textures get larger and more elaborate. There's the issue with too little VRAM, where level-of-detail swapping can't keep up with the game logic and causes clear pop-in. There's performance loss, as large textures take longer send across your system and decompress on the GPU. Neural Texture Compression throws AI at the problem. Rather than a compressed texture file that needs to go through this chain, Neural Texture Compression looks to represent textures with a format that an AI model can read and perform inference on.

Instead of some massive texture that takes up a ton of your VRAM, Neural Texture Compression uses a very lightweight "texture" that an AI model sees. Then, the Tensor cores are fired up on your GPU to perform the inference, and the texture comes out looking like you'd expect it to. In fact, Nvidia's original paper showed that Neural Texture Compression could lead to higher quality textures. Game developers would not only be able to use larger textures, they also wouldn't have to deal with compression artifacts that show up with the block compression of textures.

It's hard to argue with a technique like Neural Texture Compression. It offers better performance, less overhead, and potentially high-quality textures. But it's been a few years since Nvidia introduced the idea, and we've yet to see it materialize in an application rendering in real time. Nvidia has shown a 272MB texture shrinking down to just 11.37MB with NTC. Who could argue with that? As impressive as NTC is, it hasn't been practical in a shipping game up to this point. That might be changing, however.

A major breakthrough just happened

Dealing with a filtering hurdle

In the original Nvidia research paper for NTC, the researchers left this note under the "Limitations" section: "Unrolled filtering is computationally expensive, and stochastic filtering can introduce flickering by increasing the burden on spatiotemporal reconstruction. Literature shows that it is possible to create filterable neural representations directly, but we leave this for the future work." That future work referenced in the original research paper is what we're seeing now.

In a new paper from Nvidia titled "Collaborative Texture Filtering," researchers talk about the issues with stochastic texture filtering when using a neural representation of a texture, such as with NTC; you can read Nvidia's paper on stochastic texture filtering if you're interested in learning more about that technique. In the context of NTC, the important thing to know is that texture filtering leads to noise and flickering when the texture is magnified, basically making NTC a no-go for any developer working on a shipping game.

The technique Nvidia calls Collaborative Texture Filtering (CTF) relies on previous work on stochastic texture filtering, where researchers have shown that it's possible to share decompressed texels (the unit of a texture) with nearby pixels. CTF takes that a step further. Using GPU intrinsics available in DirectX 12 and Vulkan, Nvidia says it's able to share texels across pixels without any repeated texel decompression, and in the process, produce zero-error filtering with NTC without a massive performance overhead.

If all that sounds like gibberish, here's the TL;DR — NTC would cause noise and flickering when a texture was magnified, making it unusable in a real game. Now, Nvidia's researchers have found a way to get rid of that flickering with proper texture filtering. And, in their paper, the researchers specifically showed CTF working with an NTC texture.

Where do we go from here?

Cooperative Vector support is very new

I won't pretend to know about some sort of timeline for NTC or similar techniques demonstrated by researchers at AMD or Intel. The NTC SDK is freely available on GitHub, so, in theory, it could be included in games right now. As for when we start seeing NTC in games, that's completely in the hands of game developers. Up to this point, there have been plenty of reasons not to use neural representations for texture compression, not only due to the aforementioned issues with filtering those textures, but also due to hardware support.

NTC runs a neural network on the GPU, so it needs access to AI hardware, and that's the area where the stars are finally aligning. DirectX and Vulkan now support Cooperative Vectors, which basically allows developers to tap into the AI hardware available on your graphics card through the API — developers can add AI rendering into their shaders without targeting a specific vendor. The problem is that Cooperative Vector support is relatively new; it was just introduced at the beginning of the year in DirectX. GPU support is still very much up in the air.

AMD, Nvidia, and Intel all have AI accelerators on their GPUs, but to varying degrees. RDNA 4 GPUs like the RX 9070 XT support Cooperative Vectors, for instance, but RDNA 3 support seems lofty. In the NTC SDK, Nvidia notes that everything down to the GTX 10-series should work, but it specifically disabled inference with Cooperative Vectors on Turing and Ampere GPUs. Before something like NTC can show up in a shipping game, it needs to be cleanly supported by a wide swath of hardware that gamers are actually using. That work is underway now, but it's far from finished.

There's also the issue with fallbacks to contend with. If support is as spotty as it is now, developers would need to include the regular compressed textures alongside those neural representations for NTC, which is a lot of legwork for what is a niche capability right now. It's going to be a few years until we see anything like NTC regardless, and I suspect it'll be implemented silently. We'll likely see it applied to some textures and not others at first, offering an invisible performance and/or quality benefit on GPUs that support Cooperative Vectors.

Real-time rendering ironically takes time

Rendering techniques take a long time. It's often years after we hear about something in a research paper before it shows up in some real media, and sometimes it even takes decades before a technique provides enough benefit to warrant its usage broadly. NTC is no different. It's a technique that has limitations and certain applications, and it'll be up to game developers to decide if it warrants using or not.

The main difference with NTC is that it has some very clear benefits and applications, so it's probably something developers have toyed around with — or at least thought about — up to this point. It's possible there are games in development now that will use neural representations for textures, and we just don't know about them yet. My guess is that we'll probably have at least some examples of neural texture compression in a handful of games over the next couple of years, but we'll just have to wait and see for now.