On 6/19/2015, at approximately 7:59AM during a routine power generator
load test, while doing the switchover onto generator power, something
happened to glitch power throughout the data center. When the power
techs put the power load back onto grid power, it glitched again.
We have had this sort of maintenance done before, so we're not sure what
happened with the power glitches yet, or why we weren't notified of the
scheduling of this test. It shouldn't glitch doing these sort of power
transfers.
Besides colocation customers having issues with some of their machines,
(the scope of which is being looked at now)
it looks like the biggest problems for ipHouse services were the glitches
rebooted one of the DSL routers, and one of the dialup modem banks.
(even though both are dual powered off separate PDUs).
This would have affected about half the DSL customers, kicking them offline.
The other two things noticible to customers would have been a loss of connectivity
of colo customers in Cage#14, and the loss of one of our feeds into the 511 building.
The alert problems mostly were absorbed internally due to redundancies, and the
problems mitigated without services being affected.
If you still have any problems or questions please let us know at
[log in to unmask], or call us up at 612-337-6340.
Thank you.
--
Doug McIntyre <[log in to unmask]>
-- ipHouse/Goldengate/Bitstream/ProNS --
Network Engineer/Provisioning/Jack of all Trades
