[OmniOS-discuss] Infiniband In Omni

[Matthew Lagoe]

We talked to mellanox when we were buying  our gear, iirc there was an
upgrade key you needed to go with the base switch for L2 functionality.

 

If you are dealing with mellanox directly they are good, just if you are
buying used or w/e be careful ^.^

 

From: Matthew Mabis [mailto:mmabis at vmware.com] 
Sent: Monday, September 29, 2014 11:32 AM
To: Matthew Lagoe; 'Narayan Desai'
Cc: omnios-discuss at lists.omniti.com
Subject: RE: [OmniOS-discuss] Infiniband In Omni

 

Thanks Narayan for the info that will really help!  I assumed using IPoIB
realistically for all the things i wanted.  but i might want to test out
RDSH and ISER as well.  I just wanted to know what things i should expect
using this type of card... also what firmwares are you using on your
CX1/CX2s?  i think all of mine are programmed to 2.7.0 as that was the most
stable of firmwares for ESXi using Raphael's (Hypervisor.fr) OpenSM manager
:) 

 

Matthew the SX6012 Switch that i have has an OpenSM manager on it so i don't
have to deal with that specific issue.   Here is a link to the switch i got.

 

http://www.mellanox.com/related-docs/prod_ib_switch_systems/SX6012.pdf

 

Matt Mabis

 

  _____  

From: Matthew Lagoe <matthew.lagoe at subrigo.net>
Sent: Monday, September 29, 2014 11:12 AM
To: 'Narayan Desai'; Matthew Mabis
Cc: omnios-discuss at lists.omniti.com
Subject: RE: [OmniOS-discuss] Infiniband In Omni 

 

Also for infiniband you need a subnet manager, if your switch doesn't have a
built in one you have to run like a linux box on the same network just for
traffic to work, as there wasn't any I was able to find for openindiana back
in the day, that's why we went with 10gbe.

 

From: OmniOS-discuss [mailto:omnios-discuss-bounces at lists.omniti.com] On
Behalf Of Narayan Desai
Sent: Monday, September 29, 2014 11:03 AM
To: Matthew Mabis
Cc: omnios-discuss at lists.omniti.com
Subject: Re: [OmniOS-discuss] Infiniband In Omni

 

I hate to sound negative, but you're in for a bad time. The current drivers
were written by sun/oracle, and only just happen to still work. Parts (like
IPoIB) work, but others (EoIB) will randomly panic. Hardware support is
limited to connectx 1-2. And if anything goes wrong, there isn't anyone that
can really help you. 

 

That all said, we had decent success with iscsi/iSER over connectX 1 cards.
Fujita Syoyo forward ported the last version of the OFED stack that
Sun/Oracle modified up to more recent illumos. Those bits are here:

https://github.com/syoyo/solaris-infiniband-tools
<https://urldefense.proofpoint.com/v1/url?u=https://github.com/syoyo/solaris
-infiniband-tools&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=yqgQ6LhGnfWMd79QvLrm
Wsnr%2FlpWj5c0oy4MpT8%2Bgik%3D%0A&m=29y6hvGPnnTjl4Fs2WqbUajUK7K9QkqTmDUzOclS
lGQ%3D%0A&s=5b477f1f1ecb13efc47c34d99b50976160a61dcdc518a7799db1ba2bb2445e05
> 

 -nld

 

On Mon, Sep 29, 2014 at 1:20 PM, Matthew Mabis <mmabis at vmware.com> wrote:

Hey All

 

I was wondering if someone who has been using IB in their lab has any good
advice for someone just starting to integrate IB into their ZFS environment.
I have Connect-X (HCA) Card put into the host ( i do have a CX3 but i have
heard its not natively supported and there were issues with the drivers)
what i am looking for is this

 

    Any Best Practices for using IPoIB with OmniOS?

    Should i use the Stock drivers build into solaris (are there different
ones that work better more stable?)

    Are there Management tools i can install to run further tests?

    Any other Guidelines you might recommend.

 

What i am going to attempt is my 3 ESXI hosts have CX-3 (VPI Pro Cards)
connected to a SX6012 Switch i have a CX4 to QSFP cable coming to hook it
into the switch and wanted to see if there would be any problems (yes the CX
card is 20Gb/s) but this is the best i can offer it based on the drivers.

 

Anything someone might see as an issue (Gonna try iSCSI/NFS with This method
direct lines to the hosts) uses CIFS on the 1Gb Ether. 

Thanks

Matt Mabis


_______________________________________________
OmniOS-discuss mailing list
OmniOS-discuss at lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss
<https://urldefense.proofpoint.com/v1/url?u=http://lists.omniti.com/mailman/
listinfo/omnios-discuss&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=yqgQ6LhGnfWMd7
9QvLrmWsnr%2FlpWj5c0oy4MpT8%2Bgik%3D%0A&m=29y6hvGPnnTjl4Fs2WqbUajUK7K9QkqTmD
UzOclSlGQ%3D%0A&s=f8ffddc7c785147583518256e10cdd31bf7a7f7da0e2b5094c83f3260b
713aaa> 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://omniosce.org/ml-archive/attachments/20140929/b8206e1b/attachment.html>