On 6 April, 2017, starting around 5:50PM CDT, and ending around 6:20PM
CDT we started having failures on *all* our resolving DNS servers all
at once. These are spread out with different networking and different
server versions throughout our network.

The main problems seen were

*) access clients would not be able to lookup any new DNS names while
connecting to various sites.

*) Our main internal systems couldn't find resources required for
normal operation. eg. you might have seen a can't find database
server message while going to webmail. 

*) Some management panels would not be able to contact the
proper hosts they manage in order to operate them.

*) Some managed machines may have not been able to contact the
resources they need to for normal operation. 

While the set of DNS resolving servers are diverse (ie. different
software versions, OS versions, networking, etc), they do all share a
common config base that stretches very far back, handling different
things that have cropped up over the years.

My initial guess about problems is that some of those workarounds
suddenly found something external to us that conflicted. Or there is a
new externally triggered bug pushed out and being exploited that
affects the software in a new unknown way.

I'm going to be reviewing the base config (knowing many of those
workarounds have now been addressed by the software vendors) and I
will be reworking the config. These config tweaks will be invisible to
any clients, and will be well tested before fully rolled out to all
the DNS resolving servers.

We'll be monitoring everything as we normally do. If you have any
questions please let us know at [log in to unmask]

Thank you. 
-- 
Doug McIntyre                            <[log in to unmask]>
          -- ipHouse/Goldengate/Bitstream/ProNS -- 
       Network Engineer/Provisioning/Jack of all Trades