[OmniOS-discuss] ixgbe: breaking aggr on 10GbE X540-T2

[Dale Ghent]

> On Jan 17, 2017, at 11:31 AM, Stephan Budach <stephan.budach at JVM.DE> wrote:
> 
> Hi Dale,
> 
> Am 17.01.17 um 17:22 schrieb Dale Ghent:
>>> On Jan 17, 2017, at 11:12 AM, Stephan Budach <stephan.budach at JVM.DE>
>>>  wrote:
>>> 
>>> Hi guys,
>>> 
>>> I am sorry, but I do have to undig this old topic, since I do now have three hosts running omniOS 018/020, which show these pesky  issues with flapping their ixgbeN links on my Nexus FEXes…
>>> 
>>> Does anyone know, if there has any change been made to the ixgbe drivers since 06/2016?
>>> 
>> Since June 2016? Yes! A large update to the ixgbe driver happened in August. This added X550 support, and also brought the Intel Shared Code it uses from its 2012 vintage up to current. The updated driver is available in 014 and later.
>> 
>> /dale
>> 
> 
> do you know of any option to get to know, why three of my boxes are flapping their 10GbE ports? It's actually not only when in aggr mode, but on single use as well. Last week I presumeably had one of my RSF-1 nodes panic, since it couldn't get to it's iSCSI LUNs anymore. The thing ist, that somewhere doen the line, the ixgbe driver seems to be fine, to configure one port to 1GbE instead of 10GbE, which will stop the flapping, but wich will break the VPC on my Nexus nevertheless.
> 
> In syslog, this looks like this:
> 
> ...
> Jan 17 14:46:07 zfsha02gh79 mac: [ID 435574 kern.info] NOTICE: ixgbe1 link up, 1000 Mbps, full duplex
> Jan 17 14:46:21 zfsha02gh79 mac: [ID 486395 kern.info] NOTICE: ixgbe3 link down
> Jan 17 14:46:22 zfsha02gh79 mac: [ID 435574 kern.info] NOTICE: ixgbe3 link up, 10000 Mbps, full duplex
> Jan 17 14:46:22 zfsha02gh79 mac: [ID 486395 kern.info] NOTICE: ixgbe3 link down
> Jan 17 14:46:26 zfsha02gh79 mac: [ID 435574 kern.info] NOTICE: ixgbe3 link up, 10000 Mbps, full duplex
> Jan 17 14:52:22 zfsha02gh79 mac: [ID 486395 kern.info] NOTICE: ixgbe3 link down
> Jan 17 14:52:22 zfsha02gh79 mac: [ID 435574 kern.info] NOTICE: ixgbe3 link up, 10000 Mbps, full duplex
> Jan 17 14:52:22 zfsha02gh79 mac: [ID 486395 kern.info] NOTICE: ixgbe3 link down
> Jan 17 14:52:32 zfsha02gh79 mac: [ID 435574 kern.info] NOTICE: ixgbe3 link up, 10000 Mbps, full duplex
> Jan 17 14:54:50 zfsha02gh79 mac: [ID 486395 kern.info] NOTICE: ixgbe3 link down
> Jan 17 14:54:55 zfsha02gh79 mac: [ID 435574 kern.info] NOTICE: ixgbe3 link up, 10000 Mbps, full duplex
> Jan 17 14:58:12 zfsha02gh79 mac: [ID 486395 kern.info] NOTICE: ixgbe3 link down
> Jan 17 14:58:16 zfsha02gh79 mac: [ID 435574 kern.info] NOTICE: ixgbe3 link up, 10000 Mbps, full duplex
> Jan 17 14:59:46 zfsha02gh79 mac: [ID 486395 kern.info] NOTICE: ixgbe3 link down
> 
> Note on 14:46:07, where the system settles on a 1GbE connection…

Sounds like a cabling issue? Are the runs too long or are you not using CAT6a? It flapping at 10Gb and then settling at 1Gb would indicate a cabling issue to me. The driver will always try to link at the fastest speed that the local controller and the remote peer will negotiate at... it will not proactively downgrade the link speed. If that happens, it is because that is what the controller managed to negotiate with the remote peer at.

Are you using jumbo frames or anything outside of a normal 1500mtu link?

/dale