IPMI

Each server has a little ARM machine glued on the side with a dedicated ethernet port. Using standard protocols, as well as manufacturer extensions, we can do all sorts of useful things to the server remotely. Each machine's IPMI card has an IP address that resolves from host "rcXXipmi". Some of these include:

There are two main utilities you'll want to use:

ipmitool

ipmitool is linux app that speaks the ipmi protocol to local and remote servers. Here are some example commands to get you started (read the extensive man page for more info):

NB: Our SuperMicro machines appear to log SMART failures as OEM #0xff, e.g. ipmitool outputs something like:

   1 | 08/13/2011 | 07:08:13 | OEM #0xff | OEM Specific | Asserted

ipmitool will ask for a password for user ADMIN in all cases. You can avoid this by putting it in a 0600 file and using the -f flag, or passing it on the command line (-P; not super-secure, but our environment is pretty trusted), or putting it in your environment (see the man page).

ipmitw

<a name="ipmitw"></a>

ipmitool is a bit of a pain to use (lots of arguments, can't specify everything in environment variables or ~/.ipmitool config files, etc, only operates on one machine at a time). That's why I threw together ipmitw (ipmi tool wrap) to simplify things. You can download it here.

All it does is run ipmitool, providing the annoying user/password/protocol args automatically, and executing it once for each host listed. Hosts can be listed in ranges, or comma separated (one contiguous string – no spaces). For example:

ipmitw rc01-rc80 power reset                       (reset power on all 80 machines)
ipmitw 1-80 power reset                            (same as above, but without rc prefix)
ipmitw rc05,rc10,rc15-20,rc57,82 chassis status    (can list hosts, as well as ranges)

 

IPMI View

SuperMicro distributes a multi-platform Java GUI app called "IPMI View" that incorporates the above functions, and more. It's a bit slow and not amenable to scripting, but it does support the KVM console, which can be very useful. Note that your machine will have to be on the internal cluster VLAN to access the ipmi controllers.

IPMI View can be obtained at

ftp://ftp.supermicro.com/utility/IPMIView/

What's the ipmi password?

Ask Steve Rumble. Or, ask John (it's in ~ouster/ipmiPassword).