[OmniOS-discuss] ipadm hangs on interface creation

[Stephan Budach]

Am 10.12.14 um 17:38 schrieb Stephan Budach:
> Am 10.12.14 um 17:20 schrieb Dan McDonald:
>>> On Dec 10, 2014, at 11:12 AM, Stephan Budach <stephan.budach at jvm.de> 
>>> wrote:
>>>
>>> Hi Dan,
>>>
>>> I actually don't know the term "incantation" yet, but I assume, that 
>>> you wanted to know the release of OmniOS I am running?
>> I meant what exact command-line arguments you typed for ipadm(1M).  
>> But that's okay, because it's good to know the other things you're 
>> about to show me.
>>
>>> What is really funny, is that running pstack against the ipadm 
>>> process, somehow brought things back in line:
>>>
>>> root at nfsvmpool01:~# pkg info entire
>>>           Name: entire
>>>        Summary: Incorporation to constrain core system packages to 
>>> same build
>>>    Description: This package constrains core system package versions 
>>> to the same
>>>                 build and provides a minimal set of additional 
>>> packages.
>>>          State: Installed
>>>      Publisher: omnios
>>>        Version: 11
>>> Build Release: 5.11
>>>         Branch: 0.151006
>>> Packaging Date: Mon Oct 27 19:15:00 2014
>>>           Size: 0.00 B
>>>           FMRI: pkg://omnios/entire@11,5.11-0.151006:20141027T191500Z
>>>
>>> I was running ipadm create-if ixgbe3 in another session when I ran 
>>> pstack as suggested:
>>>
>>> root at nfsvmpool01:~# pstack `pgrep ipadm`
>>> 12397:    ipadm create-if ixgbe3
>>> fef07723 open     (8047190, 2, fec9404c)
>>> feef24f8 open     (8047190, 2, fec9404c, 8068db8, 11, feffb0a4) + 105
>>> fec935f8 i_dlpi_open (8068db8, 80475dc, 10, 1, 8047f00, 5) + ed
>>> fec93768 i_dlpi_style1_open (8068db0, 804761c, 20, fec9386d, 3, 3) + 2a
>>> fec939a8 dlpi_open (8047f00, 8047cd4, 10, fedf1541) + 149
>>> fedf17fb i_ipadm_plumb_if (8068968, 8047f00, 2, 3) + 2cb
>>> fedf0ed4 i_ipadm_create_if (8068968, 8047f00, 2, 3, 0, 3) + 9a
>>> fedf0fc2 ipadm_create_if (8068968, 8047f00, 0, 3) + e3
>>> 0805553e do_create_if (2) + 8f
>>> 08052e72 main     (80555ca, feffb0a4, 8047e30, 80525c7, 3, 8047e3c) 
>>> + df
>>> 080525c7 _start   (3, 8047ef0, 8047ef6, 8047f00, 0, 8047f07) + 83
>>>
>>> ipadm returned and created the interface, just as if nothing ever 
>>> happened. Afterwards I was able to create and delete interfaces 
>>> without any issue on the ixgbe adaptors.
>> That's VERY odd, albeit with pleasant results.  I wouldn't think 
>> pstack would unblock something like what you showed me.  You're 
>> running 006 (long-term support), but I'm not seeing any major changes 
>> in the plumbing of interfaces between then and now.
> Yes, it is… ;)
>>
>> You mentioned a desire to not reboot your system.  Did you plug in 
>> these ixgbes while booted (i.e. hotplugging)?  Or were they always 
>> there, and you just hadn't configured them until now?
> The two cards had been in the system right from the start and I 
> already configured one port (ixgbe0) to connect to a 10GbE port on our 
> 6509, which went without issues. I wanted to resolve this issue 
> without rebooting, since rebooting would have killed 53 VMs that are 
> residing on my NFS export that is running over ixgbe0.
>> It seems like you tickled a race condition or some other odd 
>> combination of circumstances that inspecting the ipadm(1M) process 
>> jostled free.  I don't have any ixgbe handy at the moment, but had I, 
>> I'd like to know if this is a reproducible problem.  The only theory 
>> I can think of is that something weird happened while ipadm was 
>> communicating with ipmgmtd, and inspecting the process with pstack 
>> allowed a context switch to occur.
> I can maybe help with that as well… I tried to set the MTU to the 
> maximum, that the driver would accept and I ended up with a mtu of 
> 15500 (what a odd number anyway) which I set using dladm on ixgbe3. 
> When i tried to create the interface using ipadm, that was the moment, 
> when things went quirky. Although I trimmed the mtu down to a 
> reasonable 9216 ipadm wouldn't create any longer and would hang at 
> creating interfaces on any of the remaining 3 ports of my T540-X2's.
>>
>> Thank you, and while it's nice to see things working, I'm sorry I 
>> don't have a better explanation about what happened, or the ability 
>> to immediately reproduce this bug.
>>
>> Dan
>>
> Ha ha, no prob - you already provided a great deal of help!
>
> Thanks a lot,
> budy
Ahh… here we go again…

root at nfsvmpool01:~#  dladm show-linkprop -p mtu
LINK         PROPERTY        PERM VALUE          DEFAULT POSSIBLE
igb0         mtu             rw   1500           1500 60-9000
igb1         mtu             rw   1500           1500 60-9000
igb2         mtu             rw   1500           1500 60-9000
igb3         mtu             rw   1500           1500 60-9000
ixgbe2       mtu             rw   1500           1500 1500-15500
ixgbe0       mtu             rw   1500           1500 1500-15500


root at nfsvmpool01:~#  pstack `pgrep dladm`
23948:  dladm show-linkprop -p mtu

What I did was to remove the ixgbe3 interface and try to set the mtu to 
9216 like this:

dladm set-linkprop -p mtu=9216 ixgbe3

I seem only to be able to kill dladm by killing it using kill -9 <PID>

Cheers,
budy