[OmniOS-discuss] infiniband

[Narayan Desai]

On Thu, Oct 23, 2014 at 10:36 PM, Michael Rasmussen <mir at miras.org> wrote:

> On Fri, 24 Oct 2014 12:08:14 +1100
> David Bomba <turbo124 at gmail.com> wrote:
>
> > Correct, they are 40Gb/s cards.
> I have heard same stories of problems with 40Gb/s cards on Linux and
> FreeBSD. What it boils down to is that at this speed software subnet
> manager is causing problems so always use a switch with hardware subnet
> manager.
>

This sounds apocryphal and wrong to me. I know of a several large
infiniband networks (1k+ host ports, non-blocking) that run with subnet
managers running on linux machines and generally work pretty well. A few
remarks:
 - subnet managers are a part of the control plane, they don't participate
directly in the data plane after initial connection setup. This means that
the footprint of a QDR port is the same as an SDR port; data packets aren't
processed through the subnet manager.
 - subnet managers aren't bug-free -- the versions integrated into switch
firmware can't be upgraded a la carte, so you are dependent on the vendor
to release a new version of the switch firmware to get bugfixes.
 - the processors included in switches are embedded and hence slow. this
could easily make a difference to subnet performance, if you need to talk
to the subnet manager frequently to get new path records.

I'd strongly recommend against running switch provided subnet managers.
 -nld
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://omniosce.org/ml-archive/attachments/20141024/c8bb9b4b/attachment-0001.html>