Services affected (warnings or errors):

   Three unrelated events created a perfect storm of critical pages.

   5:18am first issue (load balancer).  First pages start streaming in
   soon after.

   Websites, ipMom, webmail behind our load balancer on our cluster
   were offline (single issue dealing with load balancers).

   Email flow via systems behind our load balancer (double issue
   dealing with load balancers and MySQL Cluster service

   Email flow via systems not load balanced (MySQL Cluster

   A web server had kernel paniced (single issue related to itself
   only and would not have affected web services).

   MySQL Cluster shutdown affecting most ipHouse internal operations
   dealing with email, ipMom, etc (single issue dealing with NFS
   fileserver, unknown issue).

   6:25am all issues have been remediated (load blancer, kernel
   paniced webserver, MySQL Cluster) and all services are online.

Reason for service degradation:

   This is the difficult area to describe because multiple things
   happened concurrently in the timeframe above.

   At first I really thought our monitoring systems (both) had started
   flaking out as they weren't about the same kinds of things.
   Usually during a failure moment and pages start to come in, there
   is a pattern and it doesn't take a leap to see where the potential
   issue may lie.

   Started receiving pages and got up to look into things and found
   that the active load balancer wasn't able to reach the nodes behind
   it so the virtual servers (in load balancer speak) were considered
   failed and taken offline.  I initiated a manual failover from
   active to standby and cut over to the other load balancer and
   recovery occurred.  This is really getting on my nerves and my open
   ticket from a month ago is still open with F5.  This mornings
   problem does *not* match the previous issue(s) though.  This is all
   new symptoms and a different problem (I think).

   The 'then active' load balancer (prior to cutover) could not reach
   the systems on the internal network.  When I did the manual cutover
   making the other load balancer active .. everything recovered and
   it could reach the internal network again.

   One webserver had kernel paniced and was crashed.  HuH?  This is a
   completely random occurence and doesn't seem to be a product of any
   other issue this morning except to create confusion and delay.
   Very weird considering I haven't really seen many UNIX kernel
   panics on a virtualized platform.  Cosmic rays?

   Single (NFS) filesystem burp (drive mapping out a bad block, 3
   notifications in a single second) for our internal virtualization
   cluster caused the MySQL Cluster to shut down (cleanly).  This is
   the weirdest of the issues as there are 6 nodes in 2 groups stored
   on 3 different NFS servers, specifically separated so that a single
   event should not cause a problem.  Guess that design failed or
   MySQL Cluster isn't nearly as resilient as the documentation (or my
   months of testing) suggests.  Thankfully the cluster operated as it
   was supposed to in terms of clean shutdown but it should not have
   shutdown at all.  I have lots of logs to look at to see if I can
   find a root cause for the initiated shutdown.

   The MySQL Cluster has been configured to support multiple data node
   failures without interrupting operations.  As I mentioned, there
   are 6 nodes split into 2 3-node groups.  Each 'pair' of nodes has
   their virtual disks stored on a different NFS server.  This spreads
   the I/O load as well as creating redundancy over and above the
   cluster configuration.  There should not be an interruption as long
   as 1 pair of nodes is online.

I'll need to do a blog post today to describe the network layout with
a graphic to give a picture to words.  You can read it by pointing
your browser to and I welcome your

Support can be reached Monday thru Friday from 8:00am until 6:00pm via
phone at 612-337-6340, or via email at [log in to unmask]

Mike Horwath      ipHouse - Welcome home!       [log in to unmask]
        The universe is an island, surrounded by whatever it is
        that surrounds universes. - Berkeley Fortune