[OmniOS-discuss] Illumos and Infiniband

[Narayan Desai]

We're running a mixed system with connectX 1 and 2, and IB only cards work
great with the hermon driver. We haven't had any luck with CX3, but hear
that it is possible. Performance has been quite good, and we haven't needed
to worry about it so far (knock on wood)

I did a bunch of digging to figure out the OFED front. Basically:
 - Sun did the port themselves
 - they wrote native drivers that provide a different API than the linux
drivers
 - they took their ball and went home

Mellanox has only had indirect involvement in this, and afaict, there has
never been any open activity on this.

Syoyo Fujita did a forward port of the last public release of the userland
source bits (from sunfreeware.com, iirc), and there is a binary package of
the userland bits that works fine for omnios (at least for us). I haven't
managed to get a full build working myself. This bits are here:
 - github.com/syoyo/solaris-infiniband-tools

Unfortunately, this doesn't do you much good on the CX3 front; the drivers
need to be done/reworked for illumos directly.

We're interested in this in order to build high bandwidth servers; we're
going to need 60-80 gbit in the next year or so, which means RDMA is the
only option.
 -nld


On Wed, Jan 29, 2014 at 12:38 PM, Chris Zembower <zembower at criterion.com>wrote:

>
> Hello,
>
> I'm trying to dig up some information on Illumos (or Solaris)
> compatibility with the Mellanox ConnectX-3 VPI HCA. From what I can gather,
> the Hermon driver (for 1st-gen ConnectX) may work with this hardware with
> tweaking, but I can't seem to confirm that.
>
> OmniOS is my server OS of choice (I have several systems running), but the
> necessity of a working IB fabric here has led me to implement a ZFS on
> Linux box for this particular purpose. I'm currently using LIO/targetcli to
> export block SRP targets, and I'm not happy with it at all. Really wishing
> I had Comstar for this.
>
> Anyone here using IB with OmniOS? Anyone aware of a roadmap or active
> development of new IB drivers? Why isn't there an Illumos OFED port
> happening? No interest? It's thriving on the Linux side of the world...
>
> Please don't lecture me on the merits of 10GbE/FC over Infiniband, I've
> heard it all. :-)
>
> Any insights or suggestions are very much appreciated!
>
> Best,
> Chris
>
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://omniosce.org/ml-archive/attachments/20140129/7aaf974a/attachment.html>