[OmniOS-discuss] Infiniband In Omni

[Narayan Desai]

IIRC, we never worried much about firmware revisions; they were 2.7.0 or
newer.

I'd run a subnet manager on a regular linux box, just because it is easier
to work with linux daemons than processes on a slow embedded linux on a
switch.
 -nld

On Mon, Sep 29, 2014 at 2:31 PM, Matthew Mabis <mmabis at vmware.com> wrote:

>  Thanks Narayan for the info that will really help!  I assumed using
> IPoIB realistically for all the things i wanted.  but i might want to test
> out RDSH and ISER as well.  I just wanted to know what things i should
> expect using this type of card... also what firmwares are you using on your
> CX1/CX2s?  i think all of mine are programmed to 2.7.0 as that was the most
> stable of firmwares for ESXi using Raphael's (Hypervisor.fr) OpenSM manager
> :)
>
>  Matthew the SX6012 Switch that i have has an OpenSM manager on it so i
> don't have to deal with that specific issue.   Here is a link to the switch
> i got.
>
>  http://www.mellanox.com/related-docs/prod_ib_switch_systems/SX6012.pdf
>
>   *Matt Mabis*
>
>  ------------------------------
> *From:* Matthew Lagoe <matthew.lagoe at subrigo.net>
> *Sent:* Monday, September 29, 2014 11:12 AM
> *To:* 'Narayan Desai'; Matthew Mabis
> *Cc:* omnios-discuss at lists.omniti.com
> *Subject:* RE: [OmniOS-discuss] Infiniband In Omni
>
>
> Also for infiniband you need a subnet manager, if your switch doesn’t have
> a built in one you have to run like a linux box on the same network just
> for traffic to work, as there wasn’t any I was able to find for openindiana
> back in the day, that’s why we went with 10gbe.
>
>
>
> *From:* OmniOS-discuss [mailto:omnios-discuss-bounces at lists.omniti.com] *On
> Behalf Of *Narayan Desai
> *Sent:* Monday, September 29, 2014 11:03 AM
> *To:* Matthew Mabis
> *Cc:* omnios-discuss at lists.omniti.com
> *Subject:* Re: [OmniOS-discuss] Infiniband In Omni
>
>
>
> I hate to sound negative, but you're in for a bad time. The current
> drivers were written by sun/oracle, and only just happen to still work.
> Parts (like IPoIB) work, but others (EoIB) will randomly panic. Hardware
> support is limited to connectx 1-2. And if anything goes wrong, there isn't
> anyone that can really help you.
>
>
>
> That all said, we had decent success with iscsi/iSER over connectX 1
> cards. Fujita Syoyo forward ported the last version of the OFED stack that
> Sun/Oracle modified up to more recent illumos. Those bits are here:
>
> https://github.com/syoyo/solaris-infiniband-tools
> <https://urldefense.proofpoint.com/v1/url?u=https://github.com/syoyo/solaris-infiniband-tools&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=yqgQ6LhGnfWMd79QvLrmWsnr%2FlpWj5c0oy4MpT8%2Bgik%3D%0A&m=29y6hvGPnnTjl4Fs2WqbUajUK7K9QkqTmDUzOclSlGQ%3D%0A&s=5b477f1f1ecb13efc47c34d99b50976160a61dcdc518a7799db1ba2bb2445e05>
>
>  -nld
>
>
>
> On Mon, Sep 29, 2014 at 1:20 PM, Matthew Mabis <mmabis at vmware.com> wrote:
>
> Hey All
>
>
>
> I was wondering if someone who has been using IB in their lab has any good
> advice for someone just starting to integrate IB into their ZFS
> environment.  I have Connect-X (HCA) Card put into the host ( i do have a
> CX3 but i have heard its not natively supported and there were issues with
> the drivers) what i am looking for is this
>
>
>
>     Any Best Practices for using IPoIB with OmniOS?
>
>     Should i use the Stock drivers build into solaris (are there different
> ones that work better more stable?)
>
>     Are there Management tools i can install to run further tests?
>
>     Any other Guidelines you might recommend.
>
>
>
> What i am going to attempt is my 3 ESXI hosts have CX-3 (VPI Pro Cards)
> connected to a SX6012 Switch i have a CX4 to QSFP cable coming to hook it
> into the switch and wanted to see if there would be any problems (yes the
> CX card is 20Gb/s) but this is the best i can offer it based on the drivers.
>
>
>
> Anything someone might see as an issue (Gonna try iSCSI/NFS with This
> method direct lines to the hosts) uses CIFS on the 1Gb Ether.
>
> Thanks
>
> *Matt Mabis*
>
>
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
> <https://urldefense.proofpoint.com/v1/url?u=http://lists.omniti.com/mailman/listinfo/omnios-discuss&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=yqgQ6LhGnfWMd79QvLrmWsnr%2FlpWj5c0oy4MpT8%2Bgik%3D%0A&m=29y6hvGPnnTjl4Fs2WqbUajUK7K9QkqTmDUzOclSlGQ%3D%0A&s=f8ffddc7c785147583518256e10cdd31bf7a7f7da0e2b5094c83f3260b713aaa>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://omniosce.org/ml-archive/attachments/20140929/5436a575/attachment-0001.html>