[OmniOS-discuss] Clues for tracking down why kernel memory isn't being released?

[Chris Siebenmann]

> > It turns out that the explanation for this is relatively simple, as
> > is the work around. Put simply: the OmniOS kernel does not actually
> > free up these deallocated cache objects until the system is put
> > under relatively strong memory pressure. Crucially, *the ZFS ARC
> > does not create this memory pressure*; I think that you pretty much
> > need a user level program allocating enough memory in order to
> > trigger it, and I think the memory growth needs to happen relatively
> > rapidly fast so that the kernel doesn't reclaim enough memory
> > through lesser means (such as shrinking the ZFS ARC).
>
> I don't think we will get much traction for ZFS pushing applications
> out of RAM.  There is a nuance here, that can be difficult to resolve.

 I don't think ZFS should push applications out of RAM, but that's not
the issue here. The RAM in question is in the kernel and is known to
be unused; it simply isn't being freed when ZFS would like more ARC
space.  The kmem code even maintains information on the 'working set' of
allocated objects for caches, it just doesn't do anything with it until
a kmem reap triggers.

(Which as Richard mentions can be (very) painful.)

 The ideal situation would be that caches with significant numbers of
fully unused slab objects would have those unused slabs slowly released
back to the kernel's free memory pool. This might happen either just
generally or specifically in response to continued desire from the ARC
for more memory.

> > 	- cks
> > [*: interested parties can see how often cache reaping has been triggered
> >    with the following 'mdb -k' command:
> > 	::walk kmem_cache | ::printf "%4d %s\n" kmem_cache_t cache_reap cache_name
> 
> ugh. How about:
> kstat -p :::reap

 Thanks! That's much better.

(I should get better with kstat.)

	- cks