On Monday, Dec 2nd, 2013 starting around 7:30AM, ipHouse starting
having a complete failure of our main user authentication database cluster.
All DB nodes went offline at once. We started looking into this
immediately, and engineers are working on what is failing on the DB cluster.
This was resolved at 9:00AM, the engineers working on this were able
to bring on a stable pair of nodes in the cluster and allow us to have
full DB service again. It is still unknown why all nodes failed all
at once, or why they could not sync/come up during the initial
investigation of the failure. This DB cluster is built to be highly
redundant & resiliant, and should not have a major failure as such we
experienced this morning.
This DB failure would have deferred incoming email (but no email would
have been lost), while also prevented logins for dialup/DSL services,
logins for inbound POP3 & IMAP & outbound SMTP services, and access to
email through the two webmail portals.
All services are online and tested as of 9:00AM, and fully available.
We will continue to investigate into the failure to find a root cause
and strengthen our services against an interruption like this from
re-occuring. We will continue to monitor our services to ensure they
are online and available.
If you have any problems or questions please let us know at
[log in to unmask], or call us up at 612-337-6340.
--
Doug McIntyre <[log in to unmask]>
-- ipHouse/Goldengate/Bitstream/ProNS --
Network Engineer/Provisioning/Jack of all Trades
