[OmniOS-discuss] L2ARC actual size and zpool iostat -v output discrepancy?

[Saso Kiselkov]

Okay, I think I have a convincing analysis for this:

TL;DR Sadly, the zpool vdev space stats are less than useless for L2ARC
here and they're confusing you.

L2ARC devices are rotors, so they start writing at low offsets and
progress higher until they wrap around. What your vdev space stats
(zpool iostat -v) show is really the difference between the maximum and
the minimum buffer offsets. It doesn't actually say how much of the
space in between is occupied by usable buffers.  l2asize, however, does.

The reason why I asked for the l2ad_first measurement is because I
wanted to understand why your space usage was showing ~59GB of free
space on each device - it's because your L2ARC devices have not yet even
completed a single run through the rotor. If it had completed the first,
it would have indicated a serious flaw in the wrap-around logic. So this
is currently the situation:

                             write hand--------,
 L2ARC device                                   \
+------------------------------------------------V---------------------+
|  |lowest|  ...... buffers & gaps ..... |highest|  (nothing here yet) |
+----------------------------------------------------------------------+

   |                "allocated"                  |      "free"        |
   |<----------------- 313G -------------------->|<------ 59G ------->|

The gaps in between the "lowest" and "highest" L2ARC buffers get created
when the buffers previously written there are evicted from L2ARC for
whatever reason (e.g. the filesystem holding them is destroyed, or they
are written to in ARC, invalidating the contents cached in L2ARC).
Unfortunately, the "allocated" vdev space stat is not altered when this
happens, so your L2ARC could really just hold two buffers and still
appear as full in the vdev stats.

The reason why l2asize is different is because it *does* take eviction
into account. So as the vdev space allocated metric was growing due to
the write hand moving to the right, the actual amount of data stored in
the L2ARC didn't grow nearly as quickly since buffers that had
previously been written had to be evicted.

As for why the (uncompressed) l2size is roughly equal to vdev space
allocated, I'd say it's due to a rounding error in the reporting tools
and some luck. The numbers don't actually even add up to the same result:

l2size: 1374605708288 bytes / 2^30 = 1280 GB
vdev space: 313 + 314 + 314 + 313 = 1254 GB

As for general ZFS design, I think we should either fix the vdev space
stats on L2ARC devices so that they precisely correspond to l2asize, or
get rid of them altogether. Right now, the discrepancy there just
confuses people.

Hope this helps.

Cheers,
-- 
Saso