One of our mail servers was accepting emails, but not delivering them
to the destination mailboxes.
To be clear - no email was lost, only delayed.
This issue began sometime yesterday though it is difficult to say
exactly when, but what I have been able to gather from logs, the issue
started in the mid-morning timeframe (~9am-10am).
The problem occurred because the anti-virus system stopped allowing
connections from the local mail subsystems.
Our mail systems is a 3 stage system:
mail from the Internet is checked and accepted, basic
anti-spam occurs
mail from first stage is checked for spam and viruses, the
anti-spam checks are more intensive (this is the system that
does quarantining of messages) and is the first scan for virus
and phishing messages
mail from second stage is checked and accepted, and delivered,
with another round of basic anti-spam, and a final scan to
check for virus and phishing traffic
It was in this third stage where the virus scanner stopped accepting
messages.
The way the anti-virus system works is that it only works on the local
system, it does not accept any connections from outside of the server
itself. Our monitoring systems can see that the mail system was
working and accepting messages, but it can not tell if the messages
are being delivered.
Right now the system is working hard at delivering the many thousands
of messages that are queued up (current count shows it just under
30,000 messages).
I expect all email to be delivered from this server over the next
30-40 minutes.
Support can be reached Monday thru Friday from 8:00am until 8:00pm,
Saturdays from 11:00am until 4:00pm via phone at 612-337-6340, or via
email at [log in to unmask]
--
Mike Horwath [log in to unmask]
ipHouse - Welcome home!
