[OmniOS-discuss] [developer] How to release reserved TCP port binding in BOUND state

[Marcel Telka]

Hi Youzhong,

On Mon, Nov 03, 2014 at 04:14:55PM -0500, Youzhong Yang via illumos-developer wrote:
> Hello,
> 
> We are having a very strange issue on one of servers. The issue is that
> fcntl locking over NFS returns 'no locks available' immediately.
> 
> dtrace shows that bindresvport() returns error code 125 (EADDRINUSE):
> 
> # dtrace -n 'fbt:rpcmod:bindresvport:return /arg1 != 0/ {stack();
> printf("ret = %d", arg1);}'
>   9  52692              bindresvport:return
>               rpcmod`connmgr_get+0x560
>               rpcmod`connmgr_wrapget+0x63
>               rpcmod`clnt_cots_kcallit+0x198
>               rpcmod`rpcbind_getaddr+0x245
>               klmmod`update_host_rpcbinding+0x4f
>               klmmod`nlm_host_get_rpc+0x6d
>               klmmod`nlm_do_lock+0x10d
>               klmmod`nlm4_lock_4_svc+0x2a
>               klmmod`nlm_dispatch+0xe6
>               klmmod`nlm_prog_4+0x34
>               rpcmod`svc_getreq+0x1c1
>               rpcmod`svc_run+0x146
>               rpcmod`svc_do_run+0x8e
>               nfs`nfssys+0xf1
>               unix`_sys_sysenter_post_swapgs+0x149
> ret = 125
> 
> netstat shows that 501 reserved ports are in BOUND state:
> 
> # netstat -an | grep BOUND
>       *.935                *.*                0      0 1049740      0 BOUND
>       *.801                *.*                0      0 1049740      0 BOUND
>       *.798                *.*                0      0 1049740      0 BOUND
>       *.561                *.*                0      0 1049740      0 BOUND
>       *.613                *.*                0      0 1049740      0 BOUND
>       ....
> # netstat -an | grep BOUND | wc -l
>      501
> 
> Has anyone seen this similar issue? is it possible to unbind those reserved
> ports? Rebooting the server is our last resort.
> 
> Any advice would be very much appreciated.

I faced similar issue in connmgr_get().  It is filed as #1616 and the problem
is that the dead connection is not properly closed (there seems to be missing
connmgr_cancelconn() call somewhere), so the client could properly reconnect.
Unfortunately, I had no time to finish the analysis of this bug.


HTH

-- 
+-------------------------------------------+
| Marcel Telka   e-mail:   marcel at telka.sk  |
|                homepage: http://telka.sk/ |
|                jabber:   marcel at jabber.sk |
+-------------------------------------------+