[OmniOS-discuss] [smartos-discuss] Supermicro BIOS updates

[Keith Wesolowski]

On Mon, Jul 08, 2013 at 07:21:59PM -0700, Paul B. Henson wrote:

> I usually update firmware/BIOS on a fairly regular basis on my servers, 
> but supermicro has a fairly scary warning on their download page:
> 
> ------
> WARNING!
> Please do not download / upgrade the BIOS/Firmware UNLESS your system 
> has a BIOS/firmware-related issue. Flashing the wrong BIOS/firmware can 
> cause irreparable damage to the system.
> 
> In no event shall Supermicro be liable for direct, indirect, special, 
> incidental, or consequential damages arising from a BIOS/firmware update.
> ------
> 
> Where it seems they pretty much do not want you to ever update unless 
> you know of a specific issue the update will solve. Of course, they also 
> don't post changelogs with their bios updates, so it's kind of hard to 
> know 8-/. I can't remember the last time I had a box die due to a 

The solution here is to press -- hard -- on your sales rep to provide
those changelogs.  We push on them, and if more people are pushing they
will have more incentive to clean this up.  This is fundamentally a
risk-management exercise; SMCI is not at all wrong to recommend against
"random walk" changes to working systems.  In general, my approach is to
invest effort up front in qualifying systems with specific hardware,
firmware, and software and then never change them unless a problem
arises.  This is borne of a very, very, very long and painful career
full of firmware bugs and subtle changes that "seemed to work fine on
the developer's Windows 98 desktop!" but in fact break other software or
simply don't work properly at all.  You are certainly free to choose a
different approach, but in order to do so you need the right
information.  You'll need this anyway, because if and when you do hit a
problem, you'll have to decide how to upgrade, what to upgrade to, and
which existing systems (if any) you want to take a painful outage to
upgrade.  You'll also need to re-qualify from scratch with whatever you
decide to upgrade to, which means you need a test plan.  Knowing what
has changed and why is crucial to all of this.  So push.  Hard.

> corrupted bios update (most of the ones I've worked with won't even let 
> you try to flash the wrong firmware), I was wondering if that's a 
> problem with supermicro boards to the point where they actively 
> discourage updates?

No.  They know better than you just how shoddy their firmware is,
though.  It's an industry-wide problem; engineering practices in
firmware development are horrific, code quality is abysmal, and the
mind-boggling secrecy of all the participants from Intel to the BIOS
duopolists to the OEMs precludes collaboration, self-reliance, and
effective end-user testing.  Everyone knows this and it is not specific
to SMCI.  Basically all firmware on all devices is like this; the BIOS
is middle of the pack all things considered (if you ever want to shatter
some illusions, grab SMCI's "GPL firmware bundle" and take a look at the
bits of source for the BMC that are in it).  With all that in mind, if
you find something that works for your application, you stick with it
and by God don't even think about changing anything without a damn good
reason.  It's often a challenge to get your vendors and staff to
appreciate the importance of this and get consistently correct systems
into production; the last thing you need or want is to layer deliberate
random change on top of it.

> Their technical support sent me a changelog for the motherboard I'm 
> working with upon request (seems like it would be a timesaver for them 
> to just post it in the first place), and I think I do want to go ahead 

Yep.  And what is available is often maddeningly short on details,
assuming you can even read it.  It helps if you understand Mandarin, to
overlay that language's grammar onto the English words in the documents.
But still, push hard.  Make them clean this up -- better documents,
posted automatically.  They'll do it if they have the right incentive.

> and update the bios. I haven't had to boot DOS for a bios update in a 
> *long* time, my workstations for years have supported just sticking in a 
> USB flash drive with the image on it and updating from the bios itself, 
> and the servers I've been using supported bios updates via the IPMI web 
> interface or CLI. What's the preferred way to generate a bootable DOS 
> image with the bios update utility and image on it nowadays? I was 

You might consider having a look at https://github.com/joyent/sdcboot.
It's not complete (unfortunately there are a few other bits that are
SDC-only and thus not available) but you can easily add stuff for your
board.

> thinking of just downloading a freeDOS floppy image, loopback mounting 
> it to copy on the additional files, and then booting it via the IPMI 
> remote media option. Is there an easier way?

That's basically what our build system does, except for the last part.
The BMC remote media (it's not really part of IPMI) is horrible and
usually doesn't work; it also requires a lot of manual intervention to
use.  We just put this, along with tools for Joyent boards and other
devices, directly into the USB key and boot it that way via a separate
GRUB option.  You could also, if you were feeling clever, set things up
to PXE boot this.