[OmniOS-discuss] ARC, L2ARC and metadata questions

[Henrik Johansson]

> On 26 Oct 2014, at 22:50, Richard Elling <richard.elling at richardelling.com> wrote:
> 
> 
> On Oct 26, 2014, at 3:03 AM, Henrik Johansson <henrikj at henkis.net> wrote:
> 
>> Hi,
>> 
>> I have a random read-only workload against 50M which are just above 400KB each. I have built the dataset on a set of mirrored(10 mirrors)  devices for best performance but are looking into using a larger ARC or L2ARC to improve performance. I do realise that the dataset is way to large to cache, but caching metadata improves performance and latency for reading.
> 
> [I presume you mean quantity 50M of files, each just above 400KB in size]

Correct.

> 
>> 
>> In a test on a subset and accesses 1M files with primarycache set to metadata for the dataset, it did improve performance but used 12-18GB of RAM in the ARC (will run it again when possible forgot the exact number). Anyway, it was far to much RAM to be realistic for us to put in each node for metadata caching.
> 
> You'll need to look at arc_meta_used, not the (ARC) size. For convenience, the high water
> mark of arc_meta_used is stored in arc_meta_max. For example, on a smallish VM I have:
> 	# kstat -p ::arcstats:arc_meta_*
> 	zfs:0:arcstats:arc_meta_limit	1202388480
> 	zfs:0:arcstats:arc_meta_max	48319112
> 	zfs:0:arcstats:arc_meta_used	48273456
> 
> NB, for OmniOS/illumos, arc_meta_limit is not a hard limit -- it is a decision point above which
> the eviction algorithm changes priority.

I have tuned the arc_meta_limit, the problem is that the amount of ARC needed is very large.

> 
>> 
>> I have three questions I need help with:
>> 
>> 1. How do I calculate required ARC metadata size for a dataset, x bytes per block and file size / block size to get the number of blocks?
> 
> It is much easier to measure.
> 
>> 
>> 2. If I where to use a L2ARC for metadata  I need file size / block size * no files * 200 bytes of RAM ( 400 / 128 * 50M * 200 = ~ 37GB? ) to address the larger L2ARC?
> 
> This is a good starting point. Recall that this is a cache, so it depends on your working set size.
> Again, this is easier to measure the size of the headers used. The "200 bytes" can vary based on
> OS release and whether you are using a debug kernel (non-debug kernels are smaller)
> 	kstat -p ::arcstats:l2_hdr_size

I did a test with 10% of the files, it only used 1GB of the ARC for the L2ARC headers. I had not used the L2ARC with compression before, it was a pleasant surprise that we got 70x for the metadata so we will not be needing a large SSD device, too bad persistent L2ARC have not been integrated, and maybe an option to compress the ARC. I think we will solve our problem with an metadata L2ARC but it will require that we read all files once after each reboot before (~24h warmup).

kstat -n arcstats| egrep "asize|l2_size|hdr_size"
	hdr_size                        98454272
	l2_asize                        3066635264
	l2_hdr_size                     1006407376
	l2_size                         82864884736

Regards
Henrik