[OmniOS-discuss] Slow NFS speeds at rsize > 128k

[Stephan Budach]

Hi Dan,

Am 07.01.15 um 16:12 schrieb Dan McDonald:
>> On Jan 7, 2015, at 5:28 AM, Stephan Budach <stephan.budach at JVM.DE> wrote:
>>
>> Hello everyone,
>>
>> I am sharing my zfs via NFS to a couple of OVM nodes. I noticed really bad NFS read performance, when rsize goes beyond 128k, whereas the performance is just fine at 32k. The issue is, that the ovs-agent, which is performing the actual mount, doesn't accept or pass any NFS mount options to the NFS server. To give some numbers, a rsize of 1mb results in a read throughput of approx. 2Mb/s, whereas a rsize of 32k gives me 110Mb/s. Mounting a NFS export from a OEL 6u4 box has no issues with this, as the read speeds from this export are 108+MB/s regardles of the rsize of the NFS mount.
>>
>> The OmniOS box is currently connected to a 10GbE port at our core 6509, but the NFS client is connected through a 1GbE port only. MTU is at 1500 and can currently not be upped.
>> Anyone having a tip, why a rsize of 64k+ will result in such a performance drop?
> Assuming you're running over TCP, perhaps you need to increase the receive window?
>
> 	ndd -set /dev/tcp tcp_xmit_hiwat 1048576
>
> Dan
>
unfortuanetly…no. Strange thing to note is, that on my 2nd OmniBox, 
which is at r012, the read speads from the received zfs are in the 60's 
MB/s. The actual box is still at r006.
I tried the settings you suggested on both, and both did not change.

budy