On 1 July 2017, starting at 12:00PM CDT, we started getting alerts of
reachability into our network from our remote probes.
It was determined that one of our upstream backbones, HE.net was having
routing issues, not passing traffic on the backbone, even though our
sessions were still up to them and everything here was behaving normally.
We shut down our feed to HE.net around 12:18PM CDT so everything now
is running cleanly on our other primary feed of Level3 and opened
a ticket with HE.net.
During the interim interval, we were seeing failed connections to
remote sites, especially if they were HE.net connected (they are a
major global provider, so there are many places that are), and failed
inbound connections to systems on our network. After the BGP feed was
shut down, routing converged and things got much better a few minutes
later for everything to settle.
Once HE.net replies and we are satisfied that their issues are over,
we will be turning them back on, probably later in the evening tonight
so we can have full redundancy again.
We'll be monitoring everything as we normally do. If you have any
questions please let us know at [log in to unmask]
Thank you.
--
Doug McIntyre <[log in to unmask]>
-- ipHouse/Goldengate/Bitstream/ProNS --
Network Engineer/Provisioning/Jack of all Trades
