On 16 July, 2015 starting around 03:40 AM CDT, it looks like routing
across CenturyLink's backbone stopping being functional.
We were still talking to CenturyLink's router just fine, they were
feeding us all routes across BGP, so to all equipment, we should
be able to utilize our CenturyLink bandwidth.
But they apparently are having some extended outage, as I was not
able to route past Chicago trying to get to us, nor route out
beyond our upstream router from inside.
As of 4:40AM CDT, we've turned down our BGP session to CenturyLink as
a result, until they can get their act together. It looks like even
the www.qwest.net website is off the Internet right now. We are
running totally on our other primary bandwidth provider, HE.net now.
Their failure should have brought down routes to us, but because they
were still feeding us routing data as if everything was good, our routers
believed them, and kept trying to push traffic over them until we
were able to get in and physically turn down their connection to us.
All traffic routing problems should be restored after their uplink
was disabled, all our tests/alerts are clearing up as I type this.
We'll monitor their sites and routing and wait until their NOC gives
the all clear before we turn them up again.
If you still have any problems or questions please let us know at
[log in to unmask], or call us up at 612-337-6340.
Thank you.
--
Doug McIntyre <[log in to unmask]>
-- ipHouse/Goldengate/Bitstream/ProNS --
Network Engineer/Provisioning/Jack of all Trades
