[OmniOS-discuss] rpcbind: t_bind failed
Schweiss, Chip
chip at innovates.com
Wed Jan 3 16:02:43 UTC 2018
The problem occurred again starting last night. I have another clue, but I
still don't know how it is occurring or how to fix it.
It looks like all the TCP ports are in "bound" state, but not being
released.
How can I isolate the cause of this?
# netstat -an
UDP: IPv4
Local Address Remote Address State
-------------------- -------------------- ----------
*.* Unbound
*.57921 Idle
*.42045 Idle
*.* Unbound
*.* Unbound
*.* Unbound
*.* Unbound
*.33757 Idle
*.37883 Idle
*.40744 Idle
*.4045 Idle
*.111 Idle
*.4045 Idle
*.* Unbound
*.60955 Idle
*.42908 Idle
*.56487 Idle
*.111 Idle
*.* Unbound
*.60994 Idle
*.* Unbound
*.2049 Idle
*.520 Idle
*.2049 Idle
*.46876 Idle
*.50309 Idle
*.123 Idle
*.123 Idle
127.0.0.1.123 Idle
10.28.125.29.123 Idle
*.57929 Idle
*.64351 Idle
*.63145 Idle
*.39674 Idle
*.65280 Idle
*.52013 Idle
*.47989 Idle
*.* Unbound
UDP: IPv6
Local Address Remote Address
State If
--------------------------------- ---------------------------------
---------- -----
*.57921 Idle
*.* Unbound
*.* Unbound
*.40744 Idle
*.111 Idle
*.4045 Idle
*.* Unbound
*.60955 Idle
*.42908 Idle
*.* Unbound
*.2049 Idle
*.46876 Idle
*.123 Idle
::1.123 Idle
TCP: IPv4
Local Address Remote Address Swind Send-Q Rwind Recv-Q State
-------------------- -------------------- ----- ------ ----- ------
-----------
*.60571 *.* 0 0 1057280 0 BOUND
*.46344 *.* 0 0 1057280 0 BOUND
*.39729 *.* 0 0 1057280 0 BOUND
*.43531 *.* 0 0 1057280 0 BOUND
*.49051 *.* 0 0 1057280 0 BOUND
*.44876 *.* 0 0 1057280 0 BOUND
*.65416 *.* 0 0 1057280 0 BOUND
*.47714 *.* 0 0 1057280 0 BOUND
*.59055 *.* 0 0 1057280 0 BOUND
*.45033 *.* 0 0 1057280 0 BOUND
*.63321 *.* 0 0 1057280 0 BOUND
*.43896 *.* 0 0 1057280 0 BOUND
*.46627 *.* 0 0 1057280 0 BOUND
*.35555 *.* 0 0 1057280 0 BOUND
*.36115 *.* 0 0 1057280 0 BOUND
*.51969 *.* 0 0 1057280 0 BOUND
*.63741 *.* 0 0 1057280 0 BOUND
*.45747 *.* 0 0 1057280 0 BOUND
*.33245 *.* 0 0 1057280 0 BOUND
*.49925 *.* 0 0 1057280 0 BOUND
*.63503 *.* 0 0 1057280 0 BOUND
*.45319 *.* 0 0 1057280 0 BOUND
*.39977 *.* 0 0 1057280 0 BOUND
....lots of lines deleted...
*.37396 *.* 0 0 1057280 0 BOUND
*.33735 *.* 0 0 1057280 0 BOUND
*.35695 *.* 0 0 1057280 0 BOUND
*.36589 *.* 0 0 1057280 0 BOUND
*.41484 *.* 0 0 1057280 0 BOUND
*.63428 *.* 0 0 1057280 0 BOUND
*.54891 *.* 0 0 1057280 0 BOUND
*.60222 *.* 0 0 1057280 0 BOUND
*.40494 *.* 0 0 1057280 0 BOUND
TCP: IPv6
Local Address Remote Address Swind
Send-Q Rwind Recv-Q State If
--------------------------------- --------------------------------- -----
------ ----- ------ ----------- -----
*.54749 *.* 0
0 128000 0 LISTEN
::1.5999 *.* 0
0 128000 0 LISTEN
*.4045 *.* 0
0 1049200 0 LISTEN
*.111 *.* 0
0 128000 0 LISTEN
*.* *.* 0
0 128000 0 IDLE
*.45543 *.* 0
0 128000 0 LISTEN
*.2049 *.* 0
0 1049200 0 LISTEN
*.53926 *.* 0
0 128000 0 LISTEN
*.50379 *.* 0
0 128000 0 LISTEN
Active UNIX domain sockets
Address Type Vnode Conn Local Addr Remote Addr
ffffd063569210e8 stream-ord 0000000 0000000
ffffd06356921498 stream-ord 0000000 0000000
ffffd06356921bf8 stream-ord 0000000 0000000
ffffd063569260e0 stream-ord 0000000 0000000
ffffd06356926490 stream-ord ffffd0635646d300 0000000 private/scache
ffffd06356926840 stream-ord 0000000 0000000
ffffd06356926bf0 stream-ord 0000000 0000000
ffffd063569290d8 stream-ord ffffd0635646d200 0000000 private/anvil
ffffd06356929488 stream-ord 0000000 0000000
ffffd06356929838 stream-ord 0000000 0000000
ffffd06356929be8 stream-ord ffffd06356810480 0000000 private/lmtp
ffffd0635692d0d0 stream-ord 0000000 0000000
ffffd0635692d480 stream-ord 0000000 0000000
ffffd0635692d830 stream-ord ffffd06356810280 0000000 private/virtual
ffffd0635692dbe0 stream-ord 0000000 0000000
ffffd063569320c8 stream-ord 0000000 0000000
ffffd06356932478 stream-ord ffffd0635685aa00 0000000 private/local
ffffd06356932828 stream-ord 0000000 0000000
ffffd06356932bd8 stream-ord 0000000 0000000
ffffd063569360c0 stream-ord ffffd0635685ad00 0000000 private/discard
ffffd06356936470 stream-ord 0000000 0000000
ffffd06356936820 stream-ord 0000000 0000000
ffffd06356936bd0 stream-ord ffffd0635685ab00 0000000 private/retry
ffffd0635693b0b8 stream-ord 0000000 0000000
ffffd0635693b468 stream-ord 0000000 0000000
ffffd0635693b818 stream-ord ffffd0635685ae00 0000000 private/error
ffffd0635693bbc8 stream-ord 0000000 0000000
ffffd063568e10b0 stream-ord 0000000 0000000
ffffd063568e1460 stream-ord ffffd0635685a400 0000000 public/showq
ffffd063568e1810 stream-ord 0000000 0000000
ffffd063568e1bc0 stream-ord 0000000 0000000
ffffd063568e60a8 stream-ord ffffd0635685a600 0000000 private/relay
ffffd063568e6458 stream-ord 0000000 0000000
ffffd063568e6808 stream-ord 0000000 0000000
ffffd063568e6bb8 stream-ord ffffd0635685a900 0000000 private/smtp
ffffd063568ea0a0 stream-ord 0000000 0000000
ffffd063568ea450 stream-ord 0000000 0000000
ffffd063568ea800 stream-ord ffffd0635646d100 0000000 private/proxywrite
ffffd063568eabb0 stream-ord 0000000 0000000
ffffd063568f0098 stream-ord 0000000 0000000
ffffd063568f0448 stream-ord ffffd0635685ac00 0000000 private/proxymap
ffffd063568f07f8 stream-ord 0000000 0000000
ffffd063568f0ba8 stream-ord 0000000 0000000
ffffd063568f2090 stream-ord ffffd0635685a200 0000000 public/flush
ffffd063568f2440 stream-ord 0000000 0000000
ffffd063568f27f0 stream-ord 0000000 0000000
ffffd063568f2ba0 stream-ord ffffd0635685a500 0000000 private/verify
ffffd063568f9088 stream-ord 0000000 0000000
ffffd063568f9438 stream-ord 0000000 0000000
ffffd063568f97e8 stream-ord ffffd06356810080 0000000 private/trace
ffffd063568f9b98 stream-ord 0000000 0000000
ffffd063568fd080 stream-ord 0000000 0000000
ffffd063568fd430 stream-ord ffffd06356810180 0000000 private/defer
ffffd063568fd7e0 stream-ord 0000000 0000000
ffffd063568fdb90 stream-ord 0000000 0000000
ffffd06356840078 stream-ord ffffd0635685a700 0000000 private/bounce
ffffd06356840428 stream-ord 0000000 0000000
ffffd063568407d8 stream-ord 0000000 0000000
ffffd06356840b88 stream-ord ffffd0635685a800 0000000 private/rewrite
ffffd06356843070 stream-ord ffffd06356810380 0000000 private/tlsmgr
ffffd06356843420 stream-ord 0000000 0000000
ffffd063568437d0 stream-ord 0000000 0000000
ffffd06356849068 stream-ord 0000000 0000000
ffffd06356849418 stream-ord 0000000 0000000
ffffd063568497c8 stream-ord ffffd0635685a000 0000000 public/qmgr
ffffd06356849b78 stream-ord ffffd0635685a100 0000000 public/cleanup
ffffd0635684d060 stream-ord 0000000 0000000
ffffd0635684d410 stream-ord 0000000 0000000
ffffd0635684db70 stream-ord 0000000 0000000
ffffd06355646058 stream-ord 0000000 0000000
ffffd06355646b68 stream-ord ffffd0635685a300 0000000 public/pickup
ffffd063551bf3f8 stream-ord ffffd063193fe900 0000000 /var/run/.inetd.uds
ffffd063550e7b50 dgram ffffd063550eb380 0000000 /var/run/in.rdisc_mib
ffffd06355031798 dgram ffffd063536c8800 0000000 /var/run/in.ndpd_mib
ffffd06355031b48 stream-ord ffffd063536c8c00 0000000 /var/run/in.ndpd_ipadm
ffffd0635265a028 stream-ord 0000000 ffffd0634e4acd00
/var/run/dbus/system_bus_socket
ffffd0635265a788 stream-ord 0000000 ffffd063500ffc80
/var/run/hald/dbus-y1Me9kLIpf
ffffd0635265ab38 stream-ord 0000000 0000000 /var/run/hald/dbus-y1Me9kLIpf
ffffd06351d553d0 stream-ord 0000000 0000000 /var/run/hald/dbus-y1Me9kLIpf
ffffd06351d55780 stream-ord 0000000 0000000 /var/run/hald/dbus-y1Me9kLIpf
ffffd06351d55b30 stream-ord 0000000 ffffd063500ffc80
/var/run/hald/dbus-y1Me9kLIpf
ffffd06351996018 stream-ord 0000000 ffffd063500ffc80
/var/run/hald/dbus-y1Me9kLIpf
ffffd063519963c8 stream-ord 0000000 0000000 /var/run/hald/dbus-y1Me9kLIpf
ffffd06351996778 stream-ord 0000000 ffffd063500ffc80
/var/run/hald/dbus-y1Me9kLIpf
ffffd063500fe010 stream-ord 0000000 0000000 /var/run/hald/dbus-5Qrha0Wmu3
ffffd063500fe3c0 stream-ord 0000000 ffffd063500ffa80
/var/run/hald/dbus-5Qrha0Wmu3
ffffd063500fe770 stream-ord ffffd063500ffa80 0000000
/var/run/hald/dbus-5Qrha0Wmu3
ffffd063500feb20 stream-ord ffffd063500ffc80 0000000
/var/run/hald/dbus-y1Me9kLIpf
ffffd0634e4ad008 stream-ord 0000000 0000000
ffffd0634e4ad3b8 stream-ord 0000000 0000000
ffffd0634e4ad768 stream-ord 0000000 0000000 /var/run/dbus/system_bus_socket
ffffd0634e4adb18 stream-ord ffffd0634e4acd00 0000000
/var/run/dbus/system_bus_socket
A sorted output shows nearly all 64K ports in bound state.
On Tue, Jan 2, 2018 at 8:40 AM, Schweiss, Chip <chip at innovates.com> wrote:
> About once every week or two I'm having NFS connections start to collapse
> to one of my servers. Clients will lose thier connections of the the
> course of several hours. The logs fill with these messages:
>
> Dec 25 16:21:14 mir-zfs03 rpcbind: [ID 452059 daemon.error] do_accept :
> t_bind failed : Couldn't allocate address
> Dec 25 16:21:14 mir-zfs03 /usr/lib/nfs/nfsd[27689]: [ID 396295
> daemon.error] t_bind(file descriptor 188/transport tcp) TLI error 5
> Dec 25 16:21:31 mir-zfs03 last message repeated 85 times
> Dec 25 16:21:31 mir-zfs03 rpcbind: [ID 452059 daemon.error] do_accept :
> t_bind failed : Couldn't allocate address
> Dec 25 16:21:32 mir-zfs03 /usr/lib/nfs/nfsd[27689]: [ID 396295
> daemon.error] t_bind(file descriptor 188/transport tcp) TLI error 5
> Dec 25 16:21:34 mir-zfs03 last message repeated 19 times
> Dec 25 16:21:37 mir-zfs03 /usr/lib/nfs/nfsd[27689]: [ID 396295
> daemon.error] t_bind(file descriptor 200/transport tcp) TLI error 5
> Dec 25 16:22:17 mir-zfs03 last message repeated 116 times
> Dec 25 16:22:21 mir-zfs03 /usr/lib/nfs/nfsd[27689]: [ID 396295
> daemon.error] t_bind(file descriptor 206/transport tcp) TLI error 5
> Dec 25 16:23:04 mir-zfs03 last message repeated 81 times
>
> This is a fully updated OmniOS CE r151022.
>
> I've tried restarting NFS services, but the only thing that has been
> successful in restoring services has been rebooting.
>
> I'm not finding anything useful via Google except the source code that
> spits out this message. HP-UX appears to have had the same issue that
> they patched years ago. I'm guessing shared NFS/RPC code.
>
> Any clue as to the cause of this and how to fix?
>
> -Chip
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://omniosce.org/ml-archive/attachments/20180103/d9b516ea/attachment-0001.html>
More information about the OmniOS-discuss
mailing list