[OmniOS-discuss] Lights-out management on Dell servers with BMC

[Andy]

Hi,

We're in the process of migrating to OmniOS on Dell hardware from a stack
of aging SPARC servers running Solaris 10 and I thought others may be
interested in our experiences in adapting it for a lights-out environment.
The process is fairly straightforward but has taken a while to work the
kinks out from.

The servers will be doing email processing in the cloud (anti-virus,
anti-spam, policy enforcement etc.) so a stripped down operating system
with just a few additional packages fits the bill. We wanted something
based on IllumOS, Solaris-like, actively maintained and easy to install
so, after evaluating a few different options, OmniOS was the clear winner.
A big thank you to the team for making it available!

We're using Dell R210 II servers with low-power (45W) CPUs for this
application - we can stack them up in the data centres within our current
power budget. The existing Sun servers take around 0.9A (@240V) and the
Dells take < 0.2A and we're looking to replace each Sun with two Dells. To
keep costs down, we're just using the built-in BMC on the servers rather
than a DRAC card. In fact, the DRAC card offers very little over the BMC,
often just adding an unecessary layer between the user and the BMC. We
want to be able to perform complete remote rebuilds using the BMC if
necessary (these things will never have a keyboard or monitor plugged into
them).

The first problem we encountered is that the Broadcom driver (bnx) in
OmniOS causes problems with the BMC. The BMC shares the first network
interface on the server and as soon as the interface is plumbed from the
OS the BMC no longer responds. This is discussed in more detail at
https://www.illumos.org/issues/1552 and hopefully a newer version of the
bnx driver will be incorporated upstream in IllumOS or OmniOS at some
point.

To fix this, we patched the kayak miniroot image and dropped in the latest
version of the driver from the Broadcom site
(http://www.broadcom.com/support/license.php?file=NXII/solaris-7.4.1.zip)

	# cd /tftpboot/kayak
	# gzip -d miniroot
	# mount -o nologging `lofiadm -a /tftpboot/kayak/miniroot` /mnt
	... replace /mnt/kernel/drv/amd64/bnx with new version ...
	# umount /mnt
	# lofiadm -d /tftpboot/kayak/miniroot
	# gzip miniroot

Next, we customised the kayak menu.lst file to allow for installation over
serial console. The BMC console redirection uses COM2/ttyb at 115200 baud.
I stuck -mverbose in there just to get a bit more detail during the boot.

default=1
timeout=10
min_mem64 1024
title OmniOS Install
        kernel$ /boot/platform/i86pc/kernel/$ISADIR/unix -B install_media=http:///kayak/r151004.zfs.bz2,install_config=http:///kayak
        module$ /kayak/miniroot.gz
title OmniOS Install (serial)
        kernel$ /boot/platform/i86pc/kernel/$ISADIR/unix -mverbose -B install_media=http:///kayak/r151004.zfs.bz2,install_config=http:///kayak,console=ttyb,ttyb-mode="115200,8,n,1,-"
        module$ /kayak/miniroot.gz


and finally, the kayak client installation file was modified to fix up a
few things on the resulting image. At the moment, there aren't a lot of
hooks in the kayak install process so this uses a hack to catch the call
to 'bootadm' made by the installer and take the opportunity to update the
GRUB file.


BuildRpool c1t0d0
SetHostname xomni
SetTimezone UTC
UseDNS x.x.x.x citrus-it.net
Postboot '/sbin/ipadm create-if bnx0'
Postboot '/sbin/ipadm create-addr -T dhcp bnx0/v4'
NO_REBOOT=1
log "Updating BNX driver..."
/bin/cp /kernel/drv/amd64/bnx $ALTROOT/kernel/drv/amd64/bnx
log "Increasing console baud rate..."
sed -i -e '/^console:/s/9600/115200/g' $ALTROOT/etc/ttydefs
log "Hook for serial console..."
bootadm()
{
        /sbin/bootadm "$@"

        log "Updating GRUB menu for serial console."
        sed -i -e '
                /^splashimage/s/^/#/
                /^foreground/s/^/#/
                /^background/s/^/#/
                /^kernel/s/BOOTFS/&,console=ttyb,ttyb-mode="115200,8,n,1,-"/
        ' /rpool/boot/grub/menu.lst
}


and that's it! Kayak is beautifully simple and and easy to read so the
(currently) sparse documentation wasn't a problem.

root at xomni:~# modinfo | grep bnx
205 fffffffff7b93000  636c8 189   1  bnx (Broadcom NXII GbE 7.4.1)

Next step is to customise the ZFS image that's installed to include a few
additional packages, our own IPS repository and a few other
customisations. Looks straightforward enough.

Thanks again to the team and I hope this is useful to someone.

Andy

-- 
Citrus IT Limited | +44 (0)870 199 8000 | enquiries at citrus-it.co.uk
Rock House Farm | Green Moor | Wortley | Sheffield | S35 7DQ
Registered in England and Wales | Company number 4899123