On Wednesday, Jun 12th, 2013 from 12:24PM until 13:15PM, the servers
handling POP and IMAP mail retrieval were swamped out and refusing any
more connections. This would have prevented email users from getting
their email stored on our servers.
Incoming email was unaffected, and would have been delivered normally
into email boxes, it was only the retrieval of email that was affected.
No other services were affected for this period.
We traced the problem back to the load balancer itself that is
supposed to protect against outages. It's monitor probes suddenly
started checking thousands of times more often than what it is
configured to do to ensure our servers were up, displacing all user
connections and just blasting the servers with monitor probes.
We failed over the misbehaving load balancer to the standby and
everything restored as the monitor probe storm stopped. We are
rebooting and doing a full health check on the misbehaving load
balancer to clear out whatever made it go nuts for a while.
If you have any problems or questions please let us know at
[log in to unmask], or call us up at 612-337-6340.
--
Doug McIntyre <[log in to unmask]>
-- ipHouse/Goldengate/Bitstream/ProNS --
Network Engineer/Provisioning/Jack of all Trades
