On 3/17/22 21:09, Richard Laager wrote:
> I will provide a more detailed explanation of my mistakes later.
The existing servers are _only_ route servers. The new servers will be
virtualization hosts running multiple VMs. The route servers will be
running on there, but so will other things: IXP Manager, something to
answer ARP for the blackholing we want to setup, Cacti will be brought
in house, etc.
Therefore, the host has bridging configured.
We are tentatively planning to setup a quarantine VLAN as some other
exchanges have done. This would also have quarantine route servers on
it, which would behave normally except they would not advertise any
routes out. This quarantine VLAN obviously needs to exist on the
exchange switches because participant ports would be put into it.
Because there will be multiple VLANs, the plan was to use VLAN tagging
between the Arista and the new physical servers.
The MICE VLAN is setup as VLAN 1. Before you say it... I don't think
anyone likes that. Jeremy and I are of the opinion that this should
change at some point in the future, but it doesn't seem worth doing
until we are taking down the fabric anyway (e.g. a reboot of the Arista).
In light of all that, our plan was (and in hindsight, I believe this to
be mistake #1) to configure the main MICE VLAN as untagged, with the
idea of adding the quarantine VLAN as tagged in the future.
Cameron and I felt it was important to confirm the network was working
to the MICE fabric before leaving Minneapolis, since we both live so far
away. In hindsight, this was a good idea; had I made the same
configuration mistake while working remote, it would have been slower to
fix (at a minimum).
The new systems are running Ubuntu 22.04 LTS which is increasingly
frozen by the day [1] and will have a final release on April 21. So by
the time we get everything configured, it will likely be released. This
avoids us installing 20.04 LTS now and then wanting an upgrade immediately.
Ubuntu (whether 22.04 LTS, or even 20.04 LTS) uses netplan to configure
network interfaces. netplan uses an exclusively [2] declarative
configuration model (unlike ifupdown, which is mostly declarative but
some things can only be done with imperative commands). I'm actually a
fan of netplan; it has made network configuration quite a bit nicer than
ifupdown for me.
I was not sure how to configure an untagged VLAN in netplan. I reviewed
the documentation and was still unsure. I saw that the vlan "id"
parameter was documented [3] as accepting a value of 0-4094. I figured
I'd try 0 to see if that meant untagged.
That did not work. I did a packet capture and found that 0 meant an
explicit tag of 0. Having no more ideas, I gave up and removed the VLAN
configuration, going back to just a straight interface. I figured I'd
look at it more later.
One of the big advantages of netplan is that it is (supposed to be)
idempotent. That is, you configure it the way you want, run `netplan
apply` and it makes it so. You don't (usually) need to explicitly manage
the transition from the old state to the new state.
Unfortunately, that is not true for bridging and/or VLANs, at least in
some cases. In hindsight, for bridging, that largely makes sense. After
all, you wouldn't want a `netplan apply` to remove the VMs host-side
interfaces from the bridge group(s).
But for VLANs, it seems like this is probably just something nobody
implemented, as opposed to being desirable in theory. This was mistake
#2 and the immediate cause of the loop. I ended up with a bridge
interface that consisted of the physical interface (facing the MICE
switch) and a VLAN interface (with an ID of 0) on that same physical
interface. So as traffic came in, it was bridged back out the same
physical interface, this time with an explicit VLAN 0 tag.
Based on the fact that the explicit 0 tag didn't pass traffic normally
in the first place, I don't think I was necessarily looping traffic at
layer 3. But broadcast traffic would have been looped back to the
switch, with the source MAC staying the same. This would obviously mess
up the switch's MAC table. I believe that was the immediate cause of the
issue.
I noticed that things were not behaving quite right (some packet loss).
I started a packet capture. When that was very quickly 5 GB rather than
some tiny amount, I knew I had made a serious error that likely created
a loop. I reflexively confirmed this with a "brctl show bridge", issued
a reboot (because why not), and immediate jumped up and unplugged the
cables from the back of the servers.
When the systems came back up, I confirmed the network configuration was
now correct and reconnected the cables.
Because the server was expected to have multiple VMs on it, we could not
use `port-security maximum 1`. As a result, port-security was not
configured. Jeremy and I never had an explicit conversation about this,
which was certainly another mistake. Had port-security been enabled,
this loo would have been arrested much faster.
Additionally, while I mentioned the upcoming work at the UG meeting, not
announcing it on MICE-DISCUSS was a mistake. I know we talked about this
before. I just forgot to do it.
On 3/17/22 21:53, Richard Laager wrote:
> Out of an abundance of caution, he is shutting down the ports facing the
> new servers, cutting them (the thing we changed today) off completely.
The ports have been re-enabled.
The ports are now configured with the MICE VLAN _tagged_, which plays
well with Linux & netplan.
The ports have `port-security maximum 6` configured. This gives us
enough headroom to support two route servers (in the event of a hardware
failure where both have to run on the same physical box temporarily),
two quarantine route servers (same note), the ARP responder, and the
host's MAC address (if it shows up somehow).
[1] https://discourse.ubuntu.com/t/jammy-jellyfish-release-schedule/23906
[2] Backends, like networkd, can and do implement imperative hooks. But
netplan itself is only declarative. And Ubuntu is adding, in multiple
cases at my direct suggestion, additional declarative parameters to
eliminate the need for hook scripts in many scenarios.
[3] https://netplan.io/reference/#properties-for-device-type-vlans%3A
--
Richard
|
