[OmniOS-discuss] ILB memory leak?

[Al Slater]

On 21/10/2015 17:35, Dan McDonald wrote:
>
>> On Oct 21, 2015, at 6:08 AM, Al Slater <al.slater at scluk.com>
>> wrote:
>>
>> Hi,
>>
>> I am running omnios r151014 on a couple of machines with a couple
>> of zones each.  1 zone runs apache as an SSL reverse proxy, the
>> other runs ILB for load balancing web to app tier connections.
>>
>> I noticed that in the ILB zone, the ilbd process memory grows to
>> about 2Gb.   Restarting ILB releases the memory, and then the
>> memory usage gradually increases again, with each memory increase
>> approximately 2 * the size of the previous one.  I run a cronjob
>> twice a day ( 8am and 8pm) which restarts the ilb service and
>> releases the memory.
>>
>> A graph of memory usage is available at
>> https://www.dropbox.com/s/zaz51apxslnivlq/ILB_Memory_2_days.png?dl=0
>>
 >> There are currently 62 rules in the load balancer, with a
 >> total
>> of 664 server/port pairs.
>>
>> Is there anything I can provide that would help track this down?
>
> You can use svccfg(1M) to enable user-level memory debugging on ilb.
>  It may cause the ilb daemon to dump core.  (And you're just noticing
>  this in the process, not kernel memory consumption, correct?)

I am seeing kernel memory consumption increasing as well, but that may 
be a different issue.  The ilbd process memory is definitely growing.

> As root:
>
> svcadm disable -t ilb svccfg -s ilb setenv LD_PRELOAD libumem.so
> svccfg -s ilb setenv UMEM_DEBUG default svccfg -s ilb refresh svcadm
>  enable ilb
>
> That should enable user-level memory debugging.  If you get a
> coredump, save it and share it.  If you don't and the ilb daemon
> keeps running, eventually please:
>
> gcore `pgrep ilbd`
>
> and share THAT corefile.  You can also do this by youself:
>
> mdb <ilbd-core> > ::findleaks
>
> and share ::findleaks.
>
> Once you're done generating corefiles, repeat the steps above, but
> use "unsetenv LD_PRELOAD" and "unsetenv UMEM_DEBUG" instead of the
> setenv lines.

Thanks Dan.  As we are talking about production boxes here, I will have 
to try and reproduce on another box and then I will give the process 
above a go and see what we come up with.

-- 
Al Slater

Technical Director
SCL

Phone : +44 (0)1273 666607
Fax   : +44 (0)1273 666601
email : al.slater at scluk.com

Stanton Consultancy Ltd

Park Gate, 161 Preston Road, Brighton, East Sussex, BN1 6AU

Registered in England Company number: 1957652 VAT number: GB 760 2433 55