Intel Xeon 6 Granite Rapids Memory Scaling Performance From 6 To 12 MRDIMMs
With HPCG being a very memory bandwidth bound benchmark, it showed the benefit of using up to the 12 MRDIMMs for feeding the 128 cores with the Xeon 6980P. One item worth noting is the very small benefit of using 10 MRDIMMs in relation to 8 MRDIMMs. This likely comes down to the default SNC3 clustering mode with how the ten memory modules are distributed among the compute dies.
When looking at the total server power consumption use we also see the impact on power consumption from the additional MRDIMMs populating the server. The ten MRDIMM configuration also reaffirms the oddity observed in the performance.
For some HPC workloads you can get by without going for twelve MRDIMMs with a top-end 128-core Xeon 6980P server albeit highly dependent upon the workload.
Initially going for a lower quantity of MRDIMMs could also be a short-term solution and upgrading later to address the performance gap. The ten MRDIMM configuration continued to prove quite awkward and likely should be avoided.