[OmniOS-discuss] kvm io 10 times slower after r151010 -> r151012 upgrade

[Tobias Oetiker]

Yesterday Dan McDonald wrote:

> >
> > I have not tested disk performance explicitly, but even booting a
> > windows host took ages ... so I suspect whatever is causing this
> > influences all kvm guest IO.
>
> What's really REALLY weird about this is that we did not alter
> anything about how we built KVM between these releases.
>
> Tell me, can you run "lockstat sleep <time-of-test>" in the
> global zone while you run your KVM tests?  They will produce a
> lot of output, but they may be very informative about what's
> going on.
>

Unfortunately I don't have a spare 'big' iron box to play with, but
we should be able to do some downtime tonight to run that lockstat
sleep experiment and also to do some simple disk io test (with dd).

> Also, I'd be curious if you might (BEs and rpool space being
> available) upgrade a BE to bloody and repeat your tests?

> We don't have the facilities to stress out VMs like this, which
> is why we didn't notice this before 012 went out the door.
> Clearly something's messing up KVM performance (you're not the
> first to report this, but you seem to have a decent environment
> for comparisons).  Before the next stable (and incidentally
> long-term-support as well) release, I hope to have these problems
> cleared up.  One thing that should happen soon is that Joyent is
> upstreaming the VND changes into illumos-gate, which will allow
> us to be fully caught up to their illumos-kvm-cmd source, which
> we've frozen at revision 1c6181be55d1cadc4426069960688307a6083131
> since r151010.

I know that there was no kvm change ... so this must be some side
effect of another modificiation ...

What seems odd, is that only 2 (or 3) few people reported this
problem on the list.  After all, it's not something that was
difficult to notice.  After the upgrade the kvm guests really are
almost un-usable for interactive work involing network or disk IO,
especially when compared to before.

This leads me to suspect, that either only very few people are
using omnios as a kvm server OR it is also a hardware dependent
problem.

I was also wondering if we should try to boot the current smaros on
the box just to see what it does to kvm perf. But as I said, it is
a production machine, so it is all a bit tricky.

cheers
tobi

-- 
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
www.oetiker.ch tobi at oetiker.ch +41 62 775 9902