Around 2am MDT this morning IX-Denver lost power to both of its power feeds at the same time at the 910 15th st site.  This outage, therefore took down both our switching and server infrastructure for a brief period of time.   It is unknown at this time how both A and B feeds were down at the same time, we are working with the site to get a root cause.  We will update when we have an answer.

The outage persisted for clients of our route server A until early this morning.  The reason this VM did not recover automatically was eventually found to be due to an old setting going all the way back to when the VM for route server A was installed.  Our hypervisor had a configuration to permit the VM to be reachable via VNC, and the subnet this command was referencing no longer existed on this hypervisor after some hypervisor networking architectural changes earlier this year. So the VM wouldn't boot and also did not output a whole lot of information about what 'device'  failure was preventing the start of the VM.  This was a miss on our part and we apologize, we haven't had to boot this VM in well over a year.  This is should be a one off issue since not only is this configuration removed now, but we also have been using a different hypervisor schema for all new VMs for some time, and this should not be possible again.

The more lengthy route server A outage does highlight a need for at best more bi-lateral peering between our members, and 2nd best, more members connected to all three route servers. We have not done a great job of socializing the new redundant route servers, but we have been soaking them for stability for the last couple of months and it has gone well.  Route servers B and C were also both online immediately after boot as expected.  So I encourage all to look into bringing up sessions to the other route servers, there is information on our technical page ( http://ix-denver.org/?q=technical ) about the new features we support on B and C, if you have the minimum requirement of an IRRDB as-set then there is already a session built and ready to go for your network.

Please feel free to contact me if you have any questions.

Thank you,

-Mark