[OmniOS-discuss] Emulex 10Gbit adapter speeds

[Ales]

Hey guys (and girls),

I've got a few servers with and Emulex OneConnect 10Gb (I think it's 
OCe11102 chipset based), running on OmniOS (08 and 010) an Linux (OpenSuSE).

Out of the box the performance is pretty shitty but once you tweak the 
windows sizes and buffers it takes off.

But here's what's bothering me, I can reach 9,2Gbit/s on linux but only 
around 6.3Gbit/s on OmniOS. The machines are identical dual xeons 
E5-2630 so the CPU power isn't an issue (loaded with SSDs).

I'm testing my through put with iperf and netperf both show similar speeds.

Below are my tweaks to the netstack an I'd really like your input on 
10Gbit tweaks to reach the full potential.


ipadm set-prop -p recv_buf=87380  tcp
ipadm set-prop -p send_buf=87380  tcp
ipadm set-prop -p send_buf=87380  udp
ipadm set-prop -p recv_buf=87380  udp

(both enabled by default but it never hurts)

ndd -set /dev/tcp tcp_wscale_always 1
ndd -set /dev/tcp tcp_tstamp_if_wscale 1

ndd -set /dev/tcp tcp_recv_hiwat 1048576
ndd -set /dev/tcp tcp_xmit_hiwat 1048576
ndd -set /dev/udp udp_max_buf 16777216
ndd -set /dev/tcp tcp_max_buf 16777216
ndd -set /dev/tcp tcp_cwnd_max 8388608
ndd -set /dev/tcp tcp_time_wait_interval 10000
ndd -set /dev/tcp tcp_conn_req_max_q 8000
ndd -set /dev/tcp tcp_conn_req_max_q0 300000

P.S.
The MTU is 1500, I can't use jumbo frames due to support (or lack of) on 
endclients. But doesn't seem to bother linux.
The speeds reached are TCP, UDP floats around 4,5-5Gbit/s