Services affected (warnings or errors):

   Warnings: Email delivery over SMTP
   Warnings: Outbound email from our customers to smtp.iphouse.com
   Warnings: Dialup and DSL authentication *may* have been affected
   Errors: Reading email via pop.iphouse.com and imap.iphouse.com
   Errors: Webmail services, reading and sending of email

No email was lost and no other services were affected.

Reason for outage:

   One of our MySQL database slaves 'ran out of memory' and had to be
   restarted.

   The machine itself did not run out of memory but MySQL did somehow
   though I think in reality that the issue is actually a bug
   triggered and not an actual issue of memory.

Extra information:

Our authentication database, today, is handled by 1 master server and
4 slave servers.  A single slave server going offline shouldn't cause
too many problems except this one was accepting connections and
refusing the queries with errors.  This caused our mail systems to
bark at us and toss warnings to connecting clients trying to send mail
(warnings) or read email (random password failures).

I am bringing this server back online and double checking table
consistency right now and do not expect another service issue to
occur.

What is being done about this going forward?

I have been working on a new authentication database network (still
based on MySQL) and I'm in the process of testing the implementation.
This new MySQL system *may* be built on MySQL Cluster offering higher
redundancy and resilency to failures.  Since I am doing testing I
can't comment much more on this right now.

Support can be reached Monday thru Friday from 8:00am until 6:00pm via
phone at 612-337-6340, or via email at [log in to unmask]

-- 
Mike Horwath      ipHouse - Welcome home!       [log in to unmask]
        The universe is an island, surrounded by whatever it is
        that surrounds universes. - Berkeley Fortune