Services affected (warnings or errors):
Warnings: Email delivery over SMTP
Warnings: Outbound email from our customers to smtp.iphouse.com
Warnings: Dialup and DSL authentication *may* have been affected
Errors: Reading email via pop.iphouse.com and imap.iphouse.com
Errors: Webmail services, reading and sending of email
No email was lost and no other services were affected.
Reason for outage:
One of our MySQL database slaves 'ran out of memory' and had to be
restarted.
The machine itself did not run out of memory but MySQL did somehow
though I think in reality that the issue is actually a bug
triggered and not an actual issue of memory.
Extra information:
Our authentication database, today, is handled by 1 master server and
4 slave servers. A single slave server going offline shouldn't cause
too many problems except this one was accepting connections and
refusing the queries with errors. This caused our mail systems to
bark at us and toss warnings to connecting clients trying to send mail
(warnings) or read email (random password failures).
I am bringing this server back online and double checking table
consistency right now and do not expect another service issue to
occur.
What is being done about this going forward?
I have been working on a new authentication database network (still
based on MySQL) and I'm in the process of testing the implementation.
This new MySQL system *may* be built on MySQL Cluster offering higher
redundancy and resilency to failures. Since I am doing testing I
can't comment much more on this right now.
Support can be reached Monday thru Friday from 8:00am until 6:00pm via
phone at 612-337-6340, or via email at [log in to unmask]
--
Mike Horwath ipHouse - Welcome home! [log in to unmask]
The universe is an island, surrounded by whatever it is
that surrounds universes. - Berkeley Fortune
