Print

Print


I saw it, and it appears related to:
 

 
GENERAL INFORMATION
Case Number: TTN-0003669752
Customer Ticket Number: 
Customer Circuit Id: 
Case Subject: Minnesota VoIP / ETYX/176681//ZYO / Packet loss
Case Impact: Impaired
Case Status: Service Restored
Case Sub Status: Monitor
Initial Down Time: Sun Nov 03 20:41:12 GMT 2019
Restore Date Time: Wed Nov 06 15:11:24 GMT 2019

 
Thu Nov 07 13:35:09 GMT 2019


The replacement has been completed please notify Zayo if you have further issues



 
Wed Nov 06 18:05:19 GMT 2019


Current update: Correction. A Zayo technician has been dispatched to site to replace a failed card. ETA is 12:01 AM CDT on 11/07/2019 We will provide updates as they become available.



 
Wed Nov 06 18:05:08 GMT 2019


Current update: Correction. A Zayo technician has been dispatched to site to replace a failed card. ETA is 12:01 AM CDT on 11/07/2019 We will provide updates as they become available.



 
Wed Nov 06 17:05:06 GMT 2019


Current update: A Zayo technician has been dispatched to site to replace a failed card. ETA is 12:01 PM CDT. We will provide updates as they become available.



 
Mon Nov 04 20:00:14 GMT 2019


Here's how the ports in question trace out after our dispatch today: NP_BLLPMN08YB1 1-3-4 = NNP-LS0000129 (MICE) = RR 001.100.05 Juniper 4500 MICE XE 38 NP_BLLPMN08YB1 1-3-6 = NNP-LB0000170 (MPLS) = RR 001.100.05 Juniper 4500 MSP1 XE 37 NP_BLLPMN08YB1 1-3-7 = NNP-LS0000186 (Nebraskalink) = RR 001.100.05 Brocade Netiron Port 2/4 NP_BLLPMN08YB1 1-3-8 = NNP-LB0000231 (MPLS) = RR 001.100.05 Juniper 4500 MSP1 XE 38 Per Ciena ( C10237538 ), since there is no common shelf/equipment, we are going to move forward with scheduling an MR in order to swap out the CYAN card and the optic to have absolute certainty that its not the optical gear. I'll be setting this up for us and letting everyone know when the day/time is for MR. Thanks for your patience throughout this process. Eric P. Reed - Zayo ONCC



 
Mon Nov 04 19:36:04 GMT 2019


Hello, Our transport team is dispatching back out to further isolate the device causing the backbone flapping. -Zach



 
Mon Nov 04 15:40:37 GMT 2019


CURRENT STATUS: I'm picking this tkt up for review and will get back to you once I have an opportunity to work with our TAC support and get some clear direction. Eric P. Reed - Zayo ONCC.



 
Mon Nov 04 13:10:07 GMT 2019


Dear Customer, We are working with the equipment vendor and transport team to resolve the issue More updates to follow as they become available



 
Mon Nov 04 12:00:19 GMT 2019


Dear Customer, We continue to escalate with the equipment vendor to get this issue resolved. More updates to follow as they become available.



 
Mon Nov 04 10:05:25 GMT 2019


Dear Customer, We continue to work with the equipment vendor. More updates to follow as they become available



 
Mon Nov 04 08:55:06 GMT 2019


Dear Customer, The transport team is currently working with the vendor. More updates to follow as they become available.



 
Mon Nov 04 05:15:39 GMT 2019


Dear Customer, We have engaged our transport team to check that portion of the circuit. More updates to follow as they become available.



 
Mon Nov 04 03:20:08 GMT 2019


Dear Customer, We are in the process of dispatching to the POP site to trouble. More updates to follow as they become available. If you are need of immediate assistance please contact the NOC at 1.866.236.2824 Opt 1, 3



 
Mon Nov 04 00:44:46 GMT 2019


Hello Jerey, We have identified a backbone flap which might be impacting your circuit. We are currently investigating the issue. Regards, Akramul H



 
 

 
Zayo US:1.866.236.2824 Zayo Canada:1.888.404.9296 Cloud:+855.528.4797 zColo:+877.926.5687 
Website: https://www.zayo.com 
 
 
 
 
From: MICE Discuss [mailto:[log in to unmask]] On Behalf Of Frank Bulk
Sent: Thursday, November 07, 2019 7:29 AM
To: [log in to unmask]
Subject: Re: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone?
 
I saw some disruption to MICE and all of my private peering sessions between 2 and 3 am this (11/7/2019) morning – anyone else see that?
 
Frank
 
From: MICE Discuss <[log in to unmask]> On Behalf Of Jeremy Lumby
Sent: Wednesday, November 6, 2019 12:33 PM
To: [log in to unmask]
Subject: Re: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone?
 
I just got an update stating that the last update was incorrect, and the card will be replaced tonight at midnight.
 
From: Jeremy Lumby [mailto:[log in to unmask]] 
Sent: Wednesday, November 06, 2019 11:08 AM
To: MICE Discuss
Subject: RE: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone?
 
Even though I never got the notice of when Zayo was going to replace the card, they did replace it around the same time of last night’s damaged fiber replacement.  I just now got notice that it completed, and they closed the ticket.  I will watch for more loss tonight.
 
From: Jeremy Lumby [mailto:[log in to unmask]] 
Sent: Wednesday, November 06, 2019 8:29 AM
To: MICE Discuss
Subject: RE: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone?
 
Unfortunately the maintenance was unrelated, it was to replaced a damaged 288 count fiber.
 
From: MICE Discuss [mailto:[log in to unmask]] On Behalf Of James Urwiller
Sent: Wednesday, November 06, 2019 8:23 AM
To: [log in to unmask]
Subject: Re: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone?
 
Apparently they did the maintenance last night?? I was not notified, nor have I checked to see if its any better yet.  But I did lose that link last night for about an hour from midnight to 1 AM CST. 
 
 
James Urwiller
Network Architect
Stealth Broadband
[log in to unmask]

 
From: MICE Discuss <[log in to unmask]> On Behalf Of Jeremy Lumby
Sent: Tuesday, November 5, 2019 09:30
To: [log in to unmask]
Subject: Re: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone?
 
I got the same update. I did see loss last night at Peak utilization, however it was nowhere near as bad as the previous night. I am also waiting on notification for them to swap the cards in the cyan gear
 
 
 
Jeremy Lumby
Minnesota VoIP
9217 17th Ave S
Suite 216
Bloomington MN 55425
Main 612-355-7740
Direct 612-392-6814
EFax 952-873-7425
[log in to unmask]
 
 
-------- Original message --------
From: James Urwiller <[log in to unmask]> 
Date: 11/5/19 9:26 AM (GMT-06:00) 
To: [log in to unmask] 
Subject: Re: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone? 
 



From : James Urwiller [[log in to unmask]]
To : [log in to unmask] [[log in to unmask]]
Date : Tuesday, November 5 2019 09:26:57
Last ticket I received they traced out some other customers path (not mine) and told me there was no common path, but they wanted to swap a cyan transport card anyway, and would schedule the maintenance “soon” 
However I think its better-ish now?? 
James Urwiller
Network Architect
Stealth Broadband
[log in to unmask]

From: MICE Discuss <[log in to unmask]> On Behalf Of Justin Huffman
Sent: Tuesday, November 5, 2019 09:17
To: [log in to unmask]
Subject: Re: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone?
We also opened up a case with Zayo however they put a loopback on the circuit and did not see any errors apparently. Are you all still seeing loss? If so, what did Zayo come up with?
Thanks in advance,
Justin
From: MICE Discuss <[log in to unmask]> On Behalf Of Jeremy Lumby
Sent: Monday, November 4, 2019 11:01 AM
To: [log in to unmask]
Subject: Re: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone?
WARNING!! This message originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email.

The last few updates I had were that they dispatched a tech to Belle Plaine, then they moved the ticket back to the backbone group who engaged the hardware vendor. I still do not have an ETR, however yesterday I did see a little bit of loss at 8AM, however it got much worse as the traffic to MICE increased in the evening. Currently I am seeing loss, however it is much less than Sunday evening.
Jeremy Lumby
Minnesota VoIP
9217 17th Ave S #216
Bloomington, MN 55425
M: 612-355-7740
D: 612-392-6814
F: 952-873-7425
[log in to unmask]
From: MICE Discuss [mailto:[log in to unmask]] On Behalf Of James Urwiller
Sent: Monday, November 04, 2019 10:43 AM
To: [log in to unmask]
Subject: Re: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone?
Only updates I have on my ticket is that a tech is investigating.. it has certainly been sporadic for us. 
James Urwiller
Network Architect
Stealth Broadband
[log in to unmask]

From: MICE Discuss <[log in to unmask]> On Behalf Of Justin Huffman
Sent: Monday, November 4, 2019 10:30
To: [log in to unmask]
Subject: Re: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone?
We have had a few customers complain about packet loss to google.com today as well and was able to see packet loss from one of our routers up to 511 building. However, I don’t see packet loss anymore, but maybe it is really intermittent. Did you hear any update from Zayo?
Justin
Long Lines Broadband
From: MICE Discuss <[log in to unmask]> On Behalf Of Jeremy Lumby
Sent: Monday, November 4, 2019 8:36 AM
To: [log in to unmask]
Subject: Re: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone?
WARNING!! This message originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email.

I am seeing it on the Zayo path between Belle Plaine, and Minneapolis. I have a ticket open since yesterday afternoon, and they appear to be struggling to nail it down.
Jeremy Lumby
Minnesota VoIP
9217 17th Ave S #216
Bloomington, MN 55425
M: 612-355-7740
D: 612-392-6814
F: 952-873-7425
[log in to unmask]
From: MICE Discuss [mailto:[log in to unmask]] On Behalf Of James Urwiller
Sent: Monday, November 04, 2019 8:33 AM
To: [log in to unmask]
Subject: [MICE-DISCUSS] random packet loss since yesterday afternoon anyone?
Greetings all,
Since yesterday afternoon I have been experiencing random packet loss to things that traceroute to the MICE exchange. It does not appear to be related to any particular participants path out from the exchange. Is anyone else seeing this? 
James Urwiller
Network Architect
Stealth Broadband
[log in to unmask]




To unsubscribe from the MICE-DISCUSS list, click the following link:
http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 



To unsubscribe from the MICE-DISCUSS list, click the following link:
http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 



To unsubscribe from the MICE-DISCUSS list, click the following link:
http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 



To unsubscribe from the MICE-DISCUSS list, click the following link:
http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 



To unsubscribe from the MICE-DISCUSS list, click the following link:
http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 



To unsubscribe from the MICE-DISCUSS list, click the following link:
http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 
 



To unsubscribe from the MICE-DISCUSS list, click the following link:
http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 
 



To unsubscribe from the MICE-DISCUSS list, click the following link:
http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 
 



To unsubscribe from the MICE-DISCUSS list, click the following link:
http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1 
 



To unsubscribe from the MICE-DISCUSS list, click the following link:
http://lists.iphouse.net/cgi-bin/wa?SUBED1=MICE-DISCUSS&A=1