IPMI

Each server has a little ARM machine glued on the side with a dedicated ethernet port. Using standard protocols, as well as manufacturer extensions, we can do all sorts of useful things to the server remotely. Each machine's IPMI card has an IP address that resolves from host "rcXXipmi". Some of these include:

• Powering on/off, resetting, warm reboots, software shutdowns, etc.
• Accessing a serial-over-lan version of the console (can use this to configure BIOS parameters, as well as get a Linux console)
• Setting PXE boot for the next boot cycle (allows easy network re-installs)
• Listing sensor status (temperatures, voltages, etc)
• Listing error log (ECC errors, even SMART disk errors)
• Getting a remote KVM console (keyboard, video, mouse)

There are two main utilities you'll want to use:

ipmitool

ipmitool is linux app that speaks the ipmi protocol to local and remote servers. Here are some example commands to get you started (read the extensive man page for more info):

• Get a serial-over-lan console on rcXX: ipmitool -I lanplus -H rcXXipmi -U ADMIN -a sol activate
• Get the power status: ipmitool -I lanplus -H rcXXipmi -U ADMIN chassis status
• Reboot a machine: ipmitool -I lanplus -H rcXXipmi -U ADMIN power reset
• Force PXE boot on the next boot only: ipmitool -I lanplus -H rcXXipmi -U ADMIN chassis booted pxe
• Reboot the IPMI card: ipmitool -I lanplus -H rcXXipmi -U ADMIN mc reset cold
• Get sensor output: ipmitool -I lanplus -H rcXXipmi -U ADMIN sdr list
• Get the error log: ipmitool -I lanplus -H rcXXipmi -U ADMIN sel elist

NB: Our SuperMicro machines appear to log SMART failures as OEM #0xff, e.g. ipmitool outputs something like:

   1 | 08/13/2011 | 07:08:13 | OEM #0xff | OEM Specific | Asserted

ipmitool will ask for a password for user ADMIN in all cases. You can avoid this by putting it in a 0600 file and using the -f flag, or passing it on the command line (-P; not super-secure, but our environment is pretty trusted), or putting it in your environment (see the man page).

IPMI View

SuperMicro distributes a multi-platform Java GUI app called "IPMI View" that incorporates the above functions, and more. It's a bit slow and not amenable to scripting, but it does support the KVM console, which can be very useful. Note that your machine will have to be on the internal cluster VLAN to access the ipmi controllers.

IPMI View can be obtained at ftp://ftp.*supermicro*.com/utility/*IPMIView*/

What's the ipmi password?

Ask Steve.