[OmniOS-discuss] infiniband

[Rune Tipsmark]

What network throughput were you looking at before the tweaking?

Br,
Rune

-----Original Message-----
From: OmniOS-discuss [mailto:omnios-discuss-bounces at lists.omniti.com] On Behalf Of Michael Rasmussen
Sent: Sunday, November 09, 2014 5:21 PM
To: omnios-discuss at lists.omniti.com
Subject: Re: [OmniOS-discuss] infiniband

On Mon, 10 Nov 2014 00:06:26 +0100
Michael Rasmussen <mir at miras.org> wrote:

> 
> This explains the difference in speed since 2.5GT/s x2 is roughly 3.5 
> gbps over IBoIP.
> 
> Can anybody explain why Omnios only uses 2 lains when PCIe slot and 
> HCA is capable of 8 lains?
> 
Found the "bug". All required was to change the slot;-)

01:00.0 InfiniBand: Mellanox Technologies MT25408 [ConnectX VPI - IB SDR / 10GigE] (rev a0) Subsystem: Mellanox Technologies Device 0003
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr+ Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF-
FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 11 Region 0: Memory at fea00000 (64-bit, non-prefetchable)
        Region 2: Memory at fc000000 (64-bit, prefetchable)
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst- PME-Enable-
DSel=0 DScale=0 PME- Capabilities: [48] Vital Product Data
                Not readable
        Capabilities: [9c] MSI-X: Enable+ Count=256 Masked-
                Vector table: BAR=0 offset=0007c000
                PBA: BAR=0 offset=0007d000
        Capabilities: [60] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
                DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+
Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq-
AuxPwr- TransPend- LnkCap: Port #8, Speed 2.5GT/s, Width x8, ASPM L0s, Exit Latency L0s unlimited, L1 unlimited ClockPM- Surprise- LLActRep-
BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk-
DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABCD,
TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us
to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed:
2.5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis:
-6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-,
EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-

And some network performance tuning:
* network performance tuning
set ndd:tcp_wscale_always=1
set ndd:tcp_tstamp_if_wscale=1
set ndd:tcp_max_buf=16777216
set ndd:tcp_cwnd_max=8388608
set ndd:tcp_xmit_hiwat=1048576
set ndd:tcp_recv_hiwat=1048576

/var/web-gui/data/tools/iperf/iperf -s -B 10.0.2.10
------------------------------------------------------------
Server listening on TCP port 5001
Binding to local address 10.0.2.10
TCP window size: 1.00 MByte (default)
------------------------------------------------------------
[  4] local 10.0.2.10 port 5001 connected with 10.0.2.1 port 35745
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-30.0 sec  27.4 GBytes  7.84 Gbits/sec


--
Hilsen/Regards
Michael Rasmussen

Get my public GnuPG keys:
michael <at> rasmussen <dot> cc
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xD3C9A00E
mir <at> datanom <dot> net
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE501F51C
mir <at> miras <dot> org
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE3E80917
--------------------------------------------------------------
/usr/games/fortune -es says:
ROMEO:		Courage, man; the hurt cannot be much.
MERCUTIO:	No, 'tis not so deep as a well, nor so wide
			as a church-door; but 'tis enough, 'twill serve.