[OmniOS-discuss] Infiniband In Omni

[Matthew Mabis]

Thanks Narayan for the info that will really help!  I assumed using IPoIB realistically for all the things i wanted.  but i might want to test out RDSH and ISER as well.  I just wanted to know what things i should expect using this type of card... also what firmwares are you using on your CX1/CX2s?  i think all of mine are programmed to 2.7.0 as that was the most stable of firmwares for ESXi using Raphael's (Hypervisor.fr) OpenSM manager :)

Matthew the SX6012 Switch that i have has an OpenSM manager on it so i don't have to deal with that specific issue.   Here is a link to the switch i got.

http://www.mellanox.com/related-docs/prod_ib_switch_systems/SX6012.pdf

Matt Mabis

________________________________
From: Matthew Lagoe <matthew.lagoe at subrigo.net>
Sent: Monday, September 29, 2014 11:12 AM
To: 'Narayan Desai'; Matthew Mabis
Cc: omnios-discuss at lists.omniti.com
Subject: RE: [OmniOS-discuss] Infiniband In Omni

Also for infiniband you need a subnet manager, if your switch doesn't have a built in one you have to run like a linux box on the same network just for traffic to work, as there wasn't any I was able to find for openindiana back in the day, that's why we went with 10gbe.

From: OmniOS-discuss [mailto:omnios-discuss-bounces at lists.omniti.com] On Behalf Of Narayan Desai
Sent: Monday, September 29, 2014 11:03 AM
To: Matthew Mabis
Cc: omnios-discuss at lists.omniti.com
Subject: Re: [OmniOS-discuss] Infiniband In Omni

I hate to sound negative, but you're in for a bad time. The current drivers were written by sun/oracle, and only just happen to still work. Parts (like IPoIB) work, but others (EoIB) will randomly panic. Hardware support is limited to connectx 1-2. And if anything goes wrong, there isn't anyone that can really help you.

That all said, we had decent success with iscsi/iSER over connectX 1 cards. Fujita Syoyo forward ported the last version of the OFED stack that Sun/Oracle modified up to more recent illumos. Those bits are here:
https://github.com/syoyo/solaris-infiniband-tools<https://urldefense.proofpoint.com/v1/url?u=https://github.com/syoyo/solaris-infiniband-tools&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=yqgQ6LhGnfWMd79QvLrmWsnr%2FlpWj5c0oy4MpT8%2Bgik%3D%0A&m=29y6hvGPnnTjl4Fs2WqbUajUK7K9QkqTmDUzOclSlGQ%3D%0A&s=5b477f1f1ecb13efc47c34d99b50976160a61dcdc518a7799db1ba2bb2445e05>
 -nld

On Mon, Sep 29, 2014 at 1:20 PM, Matthew Mabis <mmabis at vmware.com<mailto:mmabis at vmware.com>> wrote:
Hey All

I was wondering if someone who has been using IB in their lab has any good advice for someone just starting to integrate IB into their ZFS environment.  I have Connect-X (HCA) Card put into the host ( i do have a CX3 but i have heard its not natively supported and there were issues with the drivers) what i am looking for is this

    Any Best Practices for using IPoIB with OmniOS?
    Should i use the Stock drivers build into solaris (are there different ones that work better more stable?)
    Are there Management tools i can install to run further tests?
    Any other Guidelines you might recommend.

What i am going to attempt is my 3 ESXI hosts have CX-3 (VPI Pro Cards) connected to a SX6012 Switch i have a CX4 to QSFP cable coming to hook it into the switch and wanted to see if there would be any problems (yes the CX card is 20Gb/s) but this is the best i can offer it based on the drivers.

Anything someone might see as an issue (Gonna try iSCSI/NFS with This method direct lines to the hosts) uses CIFS on the 1Gb Ether.
Thanks
Matt Mabis

_______________________________________________
OmniOS-discuss mailing list
OmniOS-discuss at lists.omniti.com<mailto:OmniOS-discuss at lists.omniti.com>
http://lists.omniti.com/mailman/listinfo/omnios-discuss<https://urldefense.proofpoint.com/v1/url?u=http://lists.omniti.com/mailman/listinfo/omnios-discuss&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=yqgQ6LhGnfWMd79QvLrmWsnr%2FlpWj5c0oy4MpT8%2Bgik%3D%0A&m=29y6hvGPnnTjl4Fs2WqbUajUK7K9QkqTmDUzOclSlGQ%3D%0A&s=f8ffddc7c785147583518256e10cdd31bf7a7f7da0e2b5094c83f3260b713aaa>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://omniosce.org/ml-archive/attachments/20140929/882037d1/attachment-0001.html>