[OmniOS-discuss] Clues for tracking down why kernel memory isn't being released?

[Chris Siebenmann]

 We have one ZFS-based NFS fileserver that persistently runs at a very
high level of non-ARC kernel memory usage that never seems to shrink.
On a 128 GB machine, mdb's ::memstat reports 95% memory usage by just
'Kernel' while the ZFS ARC is only at about 21 GB (as reported by
'kstat -m') although c_max should allow it to grow much bigger.

 According to ::kmastat, a *huge* amount of this memory appears to be
vanishing into allocated but not used kmem_alloc_131072 slab buffers:

> ::kmastat
cache                            buf       buf       buf memory      alloc alloc
name                            size    in use     total in use    succeed  fail
------------------------------ ----- --------- --------- ------ ---------- -----
[...]
kmem_alloc_131072               128K         6    613033  74.8G  196862991     0

The next closest slab memory user appears to have much more active buffers
in use:

kmem_va_4096                      4K   6644009  10834464  41.3G   85384529     0

 I assume that under normal circumstances, at least some of the memory
claimed by these free buffers would be released back to the overall
kernel memory pool. Perhaps I'm incorrect here and the kmem_alloc_*
slabs hold on to basically all of the memory they ever grab?

 If I'm correct, does anyone know of a reason why the kmem_alloc_131072
memory here is not being released back to the system? Is there any way
to push OmniOS so that it is released back, eg by allocating and using
a lot of memory at the user level?

 Thanks in advance.

	- cks