Tonight, we suffered a double power outage in one of our racks taking
out both circuits.  Doug is in the data center now looking at the
circuits and what is plugged in - there is still something shoring out
a circuit bringing that circuit down repeatedly.  He is moving cabling
around so that we keep our power redundancy (though that didn't help
in a double failure) while he researches which power supply, in which
server, is causing this to happen.

This affected our NFS storage for our internal servers (the NetApp
cluster itself), the VMware boxes we host our customers' virtual
machines on, master and slave authentication systems.

This was also the rack with the storage switch for the NFS storage for
all of our VMware environment connections.

Most physical hardware is back online.  I am bringing up internal
servers right now then I'll work on bringing up customer virtual
machines.

This will take me another 90-120 minutes to bring everything back
online, check them, resolve any types of issues that happened.

...

Was going to send this as is but as things are getting done and I was
delayed in sending this, I figure I can provide the update right here,
right now.

Power short in power strip connection in the conduit box for one of
the circuits that was affected, this eventually caused an overload
though why this overload would bring down the other circuit as well is
not really known.  We don't run our circuits at max capacity, that
isn't good for redundancy at all.

So issue is now resolved and I am working hard getting all servers
back online while checking for errors and failures along the way.

Support can be reached Monday thru Friday from 8:00am until 8:00pm via
phone at 612-337-6340, or via email at [log in to unmask]

-- 
Mike Horwath      ipHouse - Welcome home!       [log in to unmask]
        The universe is an island, surrounded by whatever it is
        that surrounds universes. - Berkeley Fortune