[OmniOS-discuss] ILB memory leak?

[Dan McDonald]

> On Oct 21, 2015, at 6:08 AM, Al Slater <al.slater at scluk.com> wrote:
> 
> Hi,
> 
> I am running omnios r151014 on a couple of machines with a couple of zones each.  1 zone runs apache as an SSL reverse proxy, the other runs ILB for load balancing web to app tier connections.
> 
> I noticed that in the ILB zone, the ilbd process memory grows to about 2Gb.   Restarting ILB releases the memory, and then the memory usage gradually increases again, with each memory increase approximately 2 * the size of the previous one.  I run a cronjob twice a day ( 8am and 8pm) which restarts the ilb service and releases the memory.
> 
> A graph of memory usage is available at https://www.dropbox.com/s/zaz51apxslnivlq/ILB_Memory_2_days.png?dl=0
> 
> There are currently 62 rules in the load balancer, with a total of 664 server/port pairs.
> 
> Is there anything I can provide that would help track this down?

You can use svccfg(1M) to enable user-level memory debugging on ilb.  It may cause the ilb daemon to dump core.  (And you're just noticing this in the process, not kernel memory consumption, correct?)

As root:

	svcadm disable -t ilb
	svccfg -s ilb setenv LD_PRELOAD libumem.so
	svccfg -s ilb setenv UMEM_DEBUG default
	svccfg -s ilb refresh
	svcadm enable ilb

That should enable user-level memory debugging.  If you get a coredump, save it and share it.  If you don't and the ilb daemon keeps running, eventually please:

	gcore `pgrep ilbd`

and share THAT corefile.  You can also do this by youself:

	mdb <ilbd-core>
	> ::findleaks

and share ::findleaks.

Once you're done generating corefiles, repeat the steps above, but use "unsetenv LD_PRELOAD" and "unsetenv UMEM_DEBUG" instead of the setenv lines.

Hope this helps,
Dan