[OmniOS-discuss] My NFS low performance test.

[Richard Elling]

On Sep 12, 2013, at 9:38 AM, Ian Kaufman <ikaufman at eng.ucsd.edu> wrote:

> Hi Muhammad,
> 
> What is the output of "sharectl get nfs"? Have you done any NFS tuning?
> 
> Even though the system says you have 7GB of RAM free, you are running
> a very slim system for ZFS. Do you have compression enabled?
> Deduplication? How big is your filesystem? Here are some RAM rules of
> thumb for ZFS (anfd other advice):
> 
> For decent ZFS performance, you need at least 1GB of RAM for every TB
> of data, plus at least 1 extra GB for OS

Disagree. We've done many, many studies of this and the RAM size is a 
function of the working set size (WSS), not total space. For example, in the 
copying case here, there is no expectation of reuse, so most of the data will 
efficiently flow through the MRU side of the ARC. The rules of thumb you're 
reaching for are for frequently used data on the MFU side of the ARC. For 
OSes like OmniOS, we see the MFU WSS peaking around 200 MB, as an 
example.

> 
> If deduplication is turned on, you need about 4GB of RAM per TB of
> data in addition to normal system and ZFS RAM if using 64K blocks on
> average, if using smaller 8K blocks on average, then you need about
> 32GB of RAM per TB of data.
> 
> RAIDZ2 vdevs should have at least 5 disks and no more than 10. Some
> believe 6 or 8 disks per vdev for RAIDZ2 is optimal.

It depends entirely on your workload. For itty-bitty files, fewer disks is faster
but less space efficient. For big files, more disks is faster and more space
efficient. If you don't know, then the rule is to protect your data.

NB, a 4-disk raidz2 offers better slightly data dependability, the same available
space, but slightly lower streaming write performance than 4 disks in a 2-way
mirror configuration.

> 
> Improve read performance with L2ARC and SSDs - improve write
> performance with either RAMDISK SSDs or SLC Flash Memory SSDs, and
> mirror the devices.

This is more likely the issue. NFS for many clients is a synchronous workload
where performance is improved by using a separate log device (SSD usually).
The quick test is to "zfs set sync=disabled file/system" and see if the perf
improves. If it does, then this well-known behavior is no mystery and the 
usual methods to solve it apply. I'll bet a steak dinner that this is the case, since
the ssh/scp case is an asynchronous write workload and performs much better :-)
 -- richard

> 
> Ian
> 
> 
> 
> 
> On Wed, Sep 11, 2013 at 11:42 PM, Muhammad Yousuf Khan <sirtcp at gmail.com> wrote:
>> Just notice and would like to share my new finding.
>> 
>> as i mentioned earlier i am using RAIDz2 and hosted a VM on NFS. so for
>> testing i "SCP"  the whole HD image of VM to some other Linux system so that
>> i can learn if the issue is with Drive speed or the issue is with NFS
>> protocol, (of course not a bug with NFS but i have done something wrong in
>> configuring)
>> 
>> here is the SCP result. the  virtual harddisk image size is 32 GB.
>> i am running this command from OmniOS SSH.
>> 
>> 
>> vm-100001-disk-1.qco  50% |**************               | 16461 MB    09:42
>> ETA
>> 
>> 
>> you can see it copy 16 GB and i confirm that it took 3 to 4 minutes to copy
>> that huge data.
>> 
>> it seems fine to me. same RAIDz2 but  protocol change (SCP on SSH) so i
>> think the issue is with NFS. not with the Hardware speed/specs limitation.
>> 
>> any idea please.
>> 
>> Thanks,
>> Myk
>> 
>> 
>> 
>> On Thu, Sep 12, 2013 at 2:40 AM, Muhammad Yousuf Khan <sirtcp at gmail.com>
>> wrote:
>>> 
>>> first of  sorry for starting a new thread for same issue explained earlier
>>> because. i
>>> mistakenly deleted the email. so my apologizes.
>>> 
>>> my omniOS server hardware is as under.
>>> dual 3.2GHz Processors, 12 GB RAM. dell 490 workstation
>>> 4 sata drives set with RAIDz2
>>> 1 40GB very old IDE drive for OmniOS root system
>>> 
>>> i have 3 servers for testing.
>>> 1. KVM virtualization server (Proxmox)
>>> 2. OmniOS (NFS)
>>> 3. Desktop machine
>>> 
>>> all three are connected via 1GB LAN broadcast domain.
>>> 
>>> i have hosted a test VM (windows 2003 server) on NFS and when i am trying
>>> to copy the data (2gb File) from VM via Microsoft sharing to a desktop
>>> machine. i doing something very abnormal which i can not understand why
>>> 
>>> just FYI : all the virtual machines hosted on KVM (proxmox server) local
>>> harddrives are on mdadm linux raid. and working great. and i see no issue in
>>> copy and pasting data from or to the local VM machines.
>>> 
>>> In attached graphics you will see 2 test of sending 2GB file.
>>> in "snap a.1"  for few seconds file transfer stuck at 2 percent and then
>>> due to unknown reasons bumps to 15%. and it continues for few seconds and
>>> when file reaches the end it again goes down to 3 Percent  as you can see in
>>> "snap a.2"
>>> 
>>> in "snap b.1" you can see it starts at 3% max and for 2 minutes it
>>> continues and speed only  changes btw 1% to 3%. see in "snap b.2"
>>> 
>>> some says disable sync for testing and see if performance increases, sorry
>>> it didnt help me. same thing happens, whether i disable or enable the sync
>>> it makes no difference for me
>>> 
>>> and Some says to upgrade the RAM which i am already looking but just a
>>> part of my confusion when i see vmstat it shows me i have 7GB free RAM
>>> 
>>> is there anyone can please explain me why and what is happening behind all
>>> this, the workstation hosting OmniOS is a good machine. why it is doing some
>>> thing very unexpected.
>>> 
>>> Any advice, suggestion, tip etc. will be highly appreciated.
>>> 
>>> Thanks
>>> Myk
>>> 
>> 
>> 
>> _______________________________________________
>> OmniOS-discuss mailing list
>> OmniOS-discuss at lists.omniti.com
>> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>> 
> 
> 
> 
> -- 
> Ian Kaufman
> Research Systems Administrator
> UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss

--

Richard.Elling at RichardElling.com
+1-760-896-4422



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://omniosce.org/ml-archive/attachments/20130912/962e9053/attachment-0001.html>