Migrating an OmniOS virtual machine from KVM to bhyve

This is a walk-through of how I moved a development OmniOS bloody virtual machine from KVM to bhyve under an OmniOS r151028 global zone. In the end, this process turned out to be much easier than I feared; the main thing was making sure that the root pool was mountable on the new system.

Before proceeding, snapshot the disk zvols in case anything goes wrong.

Set the root disk serial number

Switching an illumos guest like OmniOS from KVM to bhyve will result in the rpool disk gaining a new controller number and a new serial number. This will make the root pool unimportable and the new VM will just panic. The first step therefore is to manually set the serial number to the same as that which bhyve will create.

The KVM machine was managed via kvmadm and so a number of kvmadm commands appear below.

bhyve constructs disk serial numbers based on their underlying physical path. Specifically BHYVE- followed by the first 12 characters from the MD5 checksum of that path, separated into groups of 4 and separated by hyphens:

gz$ kvm list bloody | grep hdd
            "disk_path" : "/dev/zvol/rdsk/data/omnios/hdd-bloody",
gz$ echo '/dev/zvol/rdsk/data/omnios/hdd-bloody\c' | digest -a md5 | cut -c1-12 | tr '[a-z]' '[A-Z]' | sed 's/..../-&/g'
-B066-A256-5589

So in this case, the default disk serial number under bhyve will be

BHYVE-B066-A256-5589

Let’s update the KVM configuration with this new serial number:

gz$ pfexec kvmadm edit bloody
...
gz$ kvmadm list bloody
...
      "disk" : [
         {
            "cache" : "none",
            "model" : "virtio",
            "boot" : "true",
            "disk_path" : "/dev/zvol/rdsk/data/omnios/hdd-bloody",
            "disk_size" : "300G",
            "index" : "0",
            "serial" : "BHYVE-B066-A256-5589"
         },
...

and then restart the KVM machine to check that the new serial number has been applied:

gz$ pfexec kvmadm restart bloody
...
bloody$ iostat -En
c2t0d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: Virtio Product: Block Device Revision: 0000 Serial No: BHYVE-B066-A256-5589
Size: 322.12GB <322122547200 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0

Configure the OS for serial console:

The first line configures loader to use the serial console only and the remaining commands change the default OS console to ttya.

bloody# echo -h > /boot/config
bloody# cat << EOM > /boot/conf.d/serial
console="ttya,text"
os_console="ttya"
ttya-mode="115200,8,n,1,-"
EOM

Check disk ashift

bhyve virtual disks default to an 8K sector size, so check the ashift of the existing root pool so that the disk can be configured for a lower sector size if necessary.

bloody$ zdb | grep ashift
            ashift: 9

In my case, the original KVM disk was using 512B sectors so I configured the bhyve disks with the same as you can see in the configuration below.

Configure the bhyve zone

Based on this KVM configuration:

gz$ kvm list bloody
{
   "bloody" : {
      "disk" : [
         {
            "boot" : "true",
            "index" : "0",
            "cache" : "none",
            "disk_size" : "300G",
            "disk_path" : "/dev/zvol/rdsk/data/omnios/hdd-bloody",
            "serial" : "BHYVE-B066-A256-5589",
            "model" : "virtio"
         },
         {
            "cache" : "none",
            "index" : "1",
            "serial" : "HONALULU",
            "disk_size" : "8G",
            "disk_path" : "/dev/zvol/rdsk/data/omnios/hdd-bloody2",
            "model" : "virtio"
         }
      ],
      "vcpus" : "16",
      "nic" : [
         {
            "over" : "switch10",
            "index" : "0",
            "nic_name" : "omniosr0",
            "model" : "virtio"
         }
      ],
      "ram" : "8192",
      "serial" : [
         {
            "serial_name" : "console",
            "index" : "0"
         }
      ],
      "time_base" : "utc",
      "hpet" : "false",
      "vnc" : "socket"
   }
}

I ended up with this bhyve zone config. Note that I elected to stay with legacy boot, although OmniOS can actually EFI boot with a serial console with a little more work:

gz# zonecfg -z bloody info
zonename: bloody
zonepath: /data/zone/bloody
brand: bhyve
autoboot: false
bootargs:
pool:
limitpriv:
scheduling-class:
ip-type: exclusive
hostid:
fs-allowed:
net:
        address not specified
        allowed-address not specified
        defrouter not specified
        global-nic not specified
        mac-addr not specified
        physical: omniosr0
        vlan-id not specified
device:
        match: /dev/zvol/rdsk/data/omnios/hdd-bloody*
attr:
        name: ram
        type: string
        value: 8G
attr:
        name: vcpus
        type: string
        value: sockets=2,cores=4,threads=2
attr:
        name: bootdisk
        type: string
        value: data/omnios/hdd-bloody,sectorsize=512/512
attr:
        name: disk
        type: string
        value: data/omnios/hdd-bloody2,serial=HONALULU,sectorsize=512/512
attr:
        name: bootrom
        type: string
        value: BHYVE_CSM

Boot it up

Now to boot up the new bhyve VM. I connected to the zone console in one session and then started the zone in another.

gz# zlogin -C bloody
[Connected to zone 'bloody' console]

[NOTICE: Zone booting up]
/boot/config: -h

Consoles: serial port a
... elided ...

    ____   __  __  _   _  _
   / __ \ |  \/  || \ | || |                          .:   ..
  | |  | ||      ||  \| || |                         .o,  .o.
  | |__| || |\/| || , `_||_|  ____                   :d.  ld.   .'
   \____/ |_|  |_||_|\/ __ \ / ___|                  cd;  cd;   l.
                     | |  | ||(__                    .dd'  ld,  cc
         community   | |__| | ___)|                   .;oc. 'l:. c;
              edition \____/ |____/                      .''...;;.,c.
                                                                'c:.'
 +=============Welcome to OmniOS===========+       .c.
 |                                         |      .ddd::loo;.    ;:.
 |  1. Boot Multi User [Enter]             |      ;ddddddl.      .;;:.
 |  2. Boot [S]ingle User                  |      :dddddl           lddo,
 |  3. [Esc]ape to loader prompt           |      .ddddd;             .;do
 |  4. Reboot                              |       .odddo.              .o
 |                                         |         ;dddo,              .
 |  Options:                               |           ,lddo,.;'
 |  5. Configure Boot [O]ptions...         |             .;oddd:.
 |  6. Select Boot [E]nvironment...        |              ...;oddl,
 |                                         |               ,l,.'cdddc.
 |                                         |              .cdddc..:dddc.
 |                                         |                 .;odc..,oddl.
 +=========================================+                  .odddl'.;dddc.
                                                              :ddddddl..cddo'

Loading unix...
Loading /platform/i86pc/amd64/boot_archive...
Loading /platform/i86pc/amd64/boot_archive.hash...
Booting...

SunOS Release 5.11 Version omnios-master-c6cc2b6b38 64-bit
Copyright (c) 1983, 2010, Oracle and/or its affiliates. All rights reserved.
Copyright (c) 2017-2018 OmniOS Community Edition (OmniOSce) Association.
NOTICE: hma_vmx_init: CPU does not support VMX
vioif0: DL_ATTACH_REQ failed: DL_SYSERR (errno 22)
vioif0: DL_BIND_REQ failed: DL_OUTSTATE
vioif0: DL_PHYS_ADDR_REQ failed: DL_OUTSTATE
vioif0: DL_UNBIND_REQ failed: DL_OUTSTATE
Hostname: bloody

bloody console login:

So far, so good!

bloody console login: root
Password:
Last login: Sat Aug 25 15:30:56 2018 on console
Oct  5 12:35:34 bloody login: [ID 644210 auth.notice] ROOT LOGIN /dev/console
OmniOS 5.11     omnios-master-c6cc2b6b38       October 2018
bloody#
bloody# zpool status
  pool: rpool
 state: ONLINE
  scan: scrub repaired 0 in 1h39m with 0 errors on Sat May 12 15:28:36 2018
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          c2t0d0    ONLINE       0     0     0

errors: No known data errors

bloody# cpuid | grep -i hyper
  RAZ (hypervisor)
Maximum hypervisor CPUID leaf: 0x40000000
Hypervisor vendor string: 'bhyve bhyve '
BHYVE hypervisor detected

bloody# dladm show-link
LINK        CLASS     MTU    STATE    BRIDGE     OVER
vioif1      phys      1500   up       --         --

To move the network interface back to vioif0, I did the following: