[OmniOS-discuss] [smartos-discuss] expected memory bandwidth of Supermicro X9DRi-F and X9SRI-F

[Paul B. Henson]

On 7/9/2013 8:53 AM, Keith Wesolowski wrote:

> Get the documents from Intel describing expected memory bandwidth and
> optimal configurations.  There are significant -- nay, dramatic --
> differences depending on how you populate your controllers and channels.
> It's not surprising that having only one CPU would significantly degrade
> performance, since the MCs are part of the CPUs.  There are also
> population rules based on DIMM speed, rank count, power levels, and so
> on, and if you do not follow them you will have reduced performance.

Under memtest86+, the *dual* processor board was actually reporting less 
bandwidth than the single processor board.

I asked supermicro support what the expected bandwidth would be using 
their supported memory, and they were kind enough to run some benchmarks 
and send me the results.

They recommended using http://www.cs.virginia.edu/stream/ for benchmarking.

On the X9SRI with 32G (4x8G) they were getting roughly 30GB/s. On the 
same board with 48G (6x8G) they were only getting 18GB/s. They said 6 
DIMMs isn't balanced, resulting in the lower performance.

On the X9DRi with 64G (8x8G), they were getting roughly 40GB/s, pretty 
close to the theoretical maximum.

I'm going to try the same benchmark on my equipment. I originally had 
32G in the X9SRi board, I recently picked up an additional 16G as to 
increase the ARC, I didn't realize that would result in decreased memory 
performance. The motherboard manual doesn't mention balancing the DIMMs, 
only which order to install them in depending on how many you have. I 
don't really want to add more memory to that box, so I guess I'll have 
to decide between the trade-off of increased ARC vs decreased memory 
bandwidth.

I'm going to have 64G in the X9DRi before before I deploy it, 
unfortunately one of the kits was defective and right now it only has 
48G pending RMA.