Jonathan,
Very excellent points. Here are some reply thoughts to consider.
1. the member has their own private, long distance, layer 2 link to an IX. The member themself knows it's a long distance but most of the other members of the IX probably will not have any idea, unless they happen to infer it from latency to your router
or you otherwise disclose it.
2. if the IX does a long distance layer 2 link to an official/remote switch, it would be a lot more clear to people at a glance that members on such a far away remote switch are indeed far away
As suggested in an earlier post of this thread, BGP communities could be added on the route server sessions of members marking the member's routes as originating via switch X. Then you as a fellow member could choose to adjust your BGP knobs based on that
data, maybe you want to not take their routes or lower your local pref for them specifically and even choose to AS prepend your routes to them or not advertise to them at all. With members individually connecting a long distance, only the members themselves
are respectively aware of the fact that they are far away. For those not using route servers, they can still easily gauge the member distances if they are clearly documented as connected to switches classed as far away switches and therefore can make their
own routing decisions based on that knowledge, which may include not peering with distant members at all at said IX.
If geographic distance is a concern for connected/connecting members, maybe the conversation should be had on whether remote switches or individual members can be connecting from such a distance greater than XYZ distance or perhaps XYZ avg latency, as
you mentioned. But now, what if I setup a router in a nearby geographic location with low latency but only have a single router and a single connection back to my main established network that is far away? I as the member look to be close to the fabric and
indeed am close to the fabric, yet most of my eyeballs and/or servers are still far away, in this hypothetical situation.
Though I do think, as you imply, there is a point of diminishing returns by adding more and more connectivity at increasing distances, not just in target market / brand but also in the very real operational issues that can arise from two networks taking
a highly inefficient path via an IX because it is shorter AS hops than usual transit links but is much longer geographically. I have seen this happen first hand.
A bit late to the discussion, but here are my thoughts:
On Thu., 2022-03-24 3:58 p.m., Richard Laager wrote:
> At Wiktel, I peer with MN VoIP's far away extensions in Minneapolis.
> For example, I peer at SeattleIX (SIX) in Minneapolis. This has caused
> me some issues. For example, latency-sensitive gaming traffic was
> tromboning
> Wiktel-Minneapolis-Seattle-Chicago-Seattle-Minneapolis-Wiktel rather
> than Wiktel-Chicago-Wiktel.
This is the biggest issue I am concerned about. Layer-2 can't tell us
how far away the peer is. BGP doesn't have knobs for varying latency
across peers on the same Layer-2. The SIX example of really-remote
peers, with resultant latency higher than Transit, is a performance
degradation for SIX and their members, and reduces the quality of SIX,
I'd argue.
I would suggest that MICE have a limit, say 5 ms RTT, on latency to
remote switches. Then we can say that MICE is in a geographic region,
not potentially a pan-North-America layer-2.
Part of MICE's identity should be its location, and the resultant
low-latency across MICE itself.
If MICE is in Kansas City, and KCIX is in Minneapolis, what's the
difference between them? Why do they both exist if they serve the same
datacentres in the same way: a Layer-2 ethernet fabric?
I become confused at what problem is being solved when IXes start
overlapping and becoming non-geographic, rather than limited to a metro
area, or certain radius around a hub.
I think there's a difference between an individual member (me) arriving
over a long layer-2 link (from Winnipeg) to the IX, and the IX doing
long layer-2 links themselves. The difference is that I have knowledge
of and control over my long-haul links, but if the IX does it,
unexpectedly long-haul paths happen without my decision to use them.
Finally, I'm a big fan of what MICE has accomplished so far, I hope this
message doesn't read as being overly critical.
Cheers,
Jonathan
AS18451 - LES.NET
Jonathan Stewart
Network Engineer
LES.NET - AS18451
Desk: 1-204-666-6191
Mobile: 1-204-990-2120
130 Portage Avenue E
Winnipeg, MB R3C 0A1
CANADA
