Revisiting DDR5-6400 vs. MRDIMM-8800 Performance With Intel Xeon 6 "Granite Rapids"
HPCG is a great benchmark for demonstrating system memory performance. With HPCG is a very healthy ~24% improvement replacing the DDR5-6400 memory with MRDIMM-8800 modules.
Though now being on a retail OEM/ODM server for Granite Rapids testing, I was monitoring the total system power consumption as reported by the BMC during testing... The difference in the total server power consumption was more significant with MRDIMMs than what I was originally anticipating up to this point. The Giga Computing server with one Xeon 6980P and 12 memory modules went from an 800 Watt average with DDR5-6400 64GB DIMMs to 1237 Watt average in HPCG with the 8800MT/s MRDIMMs. There was much higher power consumption when changing out the DDR5-6400 memory for MRDIMM-8800 modules.
While the power consumption shot up quite a bit, the MRDIMM modules were only running around 5 degrees warmer than the DDR5-6400 modules as going off the DIMM_G temperature sensor reported by the BMC.
At first when seeing the significant server power consumption difference between the two sets of DIMMs, I was wondering if something was going awry in the BIOS or elsewhere. But when shifting away to the less memory intensive benchmarks, that delta receded. With NPB for example the power difference was just a 508 vs. 522 Watt average difference with MRDIMMs only consuming slightly more in this instance. This was observed in other workloads too but when running very memory bandwidth intensive workloads, the significant increase reappeared.
The HPC benchmarks continue to love the MRDIMMs for enhancing the Granite Rapids performance, albeit now we have an idea for the power consumption difference.
There is also the cost difference to consider with roughly +$100 more per 64GB MRDIMM than going for DDR5-6400 ECC server memory.
But if the cost and power consumption difference aren't a large concern, MRDIMM-8800 modules can deliver some great results with the Xeon 6900P Granite Rapids servers.
