[OmniOS-discuss] Slow CIFS Writes when using Moca 2.0 Adapter

[Bob Friesenhahn]

On Wed, 27 Jan 2016, Mini Trader wrote:

> Slow CIFS Writes when using Moca 2.0 Adapter.
> 
> I am experiencing this only under OmniOS.  I do not see this in Windows or Linux.
> 
> I have a ZFS CIFS share setup which can easily do writes that would saturate a 1GBe connection.
> 
> My problem appears to be related somehow to the interaction between OmniOS and ECB6200 Moca 2.0 adapters.
> 
> 1. If I write to my OmniOS CIFS share using ethernet my speeds up/down are around 110 mb/sec - good
> 
> 2. If I write to my share using the same source but over the adapter my speeds are around 35mb/sec - problem

MoCA has a 3.0+ millisecond latency (I typically see 3.5ms when using 
ping).  This latency is fairly large compared with typical hard drive 
latencies and vastly higher than Ethernet.  There is nothing which can 
be done about this latency.

Unbonded MoCA 2.0 throughput for streaming data is typically 
500Mbit/second, and bonded (two channels) MoCA 2.0 doubles that (the 
claimed specs are of course higher than this and higher speeds can be 
measured under ideal conditions).  This means that typical MoCA 2.0 
(not bonded) achieves a bit less than half of what gigabit Ethernet 
achieves when streaming data over TCP.

> 3. If I read from the share using the same device over the adapter my speeds are around 110mb/sec - good

Reading is normally more of a streaming operation so the TCP will 
stream rather well.

> 4. If I setup a share on a Windows machine and write to it from the same source using the  adapter the speeds are
> around 110mb/sec.  The Windows machine is actually a VM whos disks are backed by a ZFS NFS share on the same
> machine

This seems rather good. Quite a lot depends on what the server side 
does.  If it commits each write to disk before accepting more, then 
the write speed would suffer.

> So basically the issue only takes place when writing to the OmniOS CIFS share using the adapter, if the adapter is
> not used than the write speed is perfect.

If the MoCA adaptor supports bonded mode, then it is useful to know 
that usually bonded mode needs to be enabled.  Is it possible that the 
Windows driver is enabling bonded mode but the OmniOS driver does not?

Try running a TCP streaming benchmark (program to program) to see what 
the peak network throughput is in each case.

> Any ideas why/how a Moca 2.0 adapter which is just designed to convert an ethernet  signal to a coax and back to
> ethernet  would cause issues with writes on OmniOS when the exact same share has no issues when using an actual
> ethernet connection?  More importantly, why is this happening with OmniOS CIFS and not anything else?

Latency, synchronous writes, and possibly bonding not enabled.  Also, 
OmniOS r151016 or later is need to get the latest CIFS implementation 
(based on Nexenta changes), which has been reported on this list to be 
quite a lot faster than the older one.

Bob
-- 
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/