[OmniOS-discuss] ixgbe: breaking aggr on 10GbE X540-T2

[Stephan Budach]

Am 17.01.17 um 17:37 schrieb Dale Ghent:
>> On Jan 17, 2017, at 11:31 AM, Stephan Budach <stephan.budach at JVM.DE> wrote:
>>
>> Hi Dale,
>>
>> Am 17.01.17 um 17:22 schrieb Dale Ghent:
>>>> On Jan 17, 2017, at 11:12 AM, Stephan Budach <stephan.budach at JVM.DE>
>>>>   wrote:
>>>>
>>>> Hi guys,
>>>>
>>>> I am sorry, but I do have to undig this old topic, since I do now have three hosts running omniOS 018/020, which show these pesky  issues with flapping their ixgbeN links on my Nexus FEXes…
>>>>
>>>> Does anyone know, if there has any change been made to the ixgbe drivers since 06/2016?
>>>>
>>> Since June 2016? Yes! A large update to the ixgbe driver happened in August. This added X550 support, and also brought the Intel Shared Code it uses from its 2012 vintage up to current. The updated driver is available in 014 and later.
>>>
>>> /dale
>>>
>> do you know of any option to get to know, why three of my boxes are flapping their 10GbE ports? It's actually not only when in aggr mode, but on single use as well. Last week I presumeably had one of my RSF-1 nodes panic, since it couldn't get to it's iSCSI LUNs anymore. The thing ist, that somewhere doen the line, the ixgbe driver seems to be fine, to configure one port to 1GbE instead of 10GbE, which will stop the flapping, but wich will break the VPC on my Nexus nevertheless.
>>
>> In syslog, this looks like this:
>>
>> ...
>> Jan 17 14:46:07 zfsha02gh79 mac: [ID 435574 kern.info] NOTICE: ixgbe1 link up, 1000 Mbps, full duplex
>> Jan 17 14:46:21 zfsha02gh79 mac: [ID 486395 kern.info] NOTICE: ixgbe3 link down
>> Jan 17 14:46:22 zfsha02gh79 mac: [ID 435574 kern.info] NOTICE: ixgbe3 link up, 10000 Mbps, full duplex
>> Jan 17 14:46:22 zfsha02gh79 mac: [ID 486395 kern.info] NOTICE: ixgbe3 link down
>> Jan 17 14:46:26 zfsha02gh79 mac: [ID 435574 kern.info] NOTICE: ixgbe3 link up, 10000 Mbps, full duplex
>> Jan 17 14:52:22 zfsha02gh79 mac: [ID 486395 kern.info] NOTICE: ixgbe3 link down
>> Jan 17 14:52:22 zfsha02gh79 mac: [ID 435574 kern.info] NOTICE: ixgbe3 link up, 10000 Mbps, full duplex
>> Jan 17 14:52:22 zfsha02gh79 mac: [ID 486395 kern.info] NOTICE: ixgbe3 link down
>> Jan 17 14:52:32 zfsha02gh79 mac: [ID 435574 kern.info] NOTICE: ixgbe3 link up, 10000 Mbps, full duplex
>> Jan 17 14:54:50 zfsha02gh79 mac: [ID 486395 kern.info] NOTICE: ixgbe3 link down
>> Jan 17 14:54:55 zfsha02gh79 mac: [ID 435574 kern.info] NOTICE: ixgbe3 link up, 10000 Mbps, full duplex
>> Jan 17 14:58:12 zfsha02gh79 mac: [ID 486395 kern.info] NOTICE: ixgbe3 link down
>> Jan 17 14:58:16 zfsha02gh79 mac: [ID 435574 kern.info] NOTICE: ixgbe3 link up, 10000 Mbps, full duplex
>> Jan 17 14:59:46 zfsha02gh79 mac: [ID 486395 kern.info] NOTICE: ixgbe3 link down
>>
>> Note on 14:46:07, where the system settles on a 1GbE connection…
> Sounds like a cabling issue? Are the runs too long or are you not using CAT6a? It flapping at 10Gb and then settling at 1Gb would indicate a cabling issue to me. The driver will always try to link at the fastest speed that the local controller and the remote peer will negotiate at... it will not proactively downgrade the link speed. If that happens, it is because that is what the controller managed to negotiate with the remote peer at.
>
> Are you using jumbo frames or anything outside of a normal 1500mtu link?
>
> /dale
The cables are actually specifically purchased cat6 cables. They run 
about 2m, not more. It could be tna cables, but I am running a couple of 
those and afaik, I only get these issues on these three nodes. I can try 
some other cables, but I hoped to be able to get maybe some kind of 
debug messages from the driver.

Thanks,
Stephan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://omniosce.org/ml-archive/attachments/20170117/dd578ada/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5546 bytes
Desc: not available
URL: <https://omniosce.org/ml-archive/attachments/20170117/dd578ada/attachment-0001.bin>