I was on the road this afternoon and unable to safely contribute or comment, but it looks like everyone came together and correctly diagnosed and resolved the issues.  I thought I'd follow up to a few questions/comments to make sure all is clear.

> clear ethernet-switching table

From the logs, comments, and graphs I'm sure we ran into that software bug again, regardless of any original leaf-node issues.  The switch gets into a state where MAC addresses are not all correctly added to the L2 forwarding database so some destinations are treated as broadcasts.  After issuing the command above the switch recovers from its funk and the table populates normally.  Once our maintenance is reinstated we'll try to carve out some time to open a ticket, and upgrade software if needed.

Thanks to our friends at Wistrom Telephone for stepping up to the plate once again on that.  Speaking of which, given how important the exchange seems to be to members I assume everyone will be eager to offer both time and financial support as needs arise in the future.  There's nothing magic happening here, just hard work and capital... same as any endeavor.

> It doesn't look like Anthony applied the controls onto
> the leaf switches, like Mankato Networks', that is
> currently not applied.

Also correct, our L2 security plans centered around member facing ports - the definition of which may shift over time, along with configuration.

Note that we do have spanning tree running between the exchange's three switches, and it appears to have worked properly to change the state of the Mankato switch port, but switching-table bug reared its ugly
head at the same time.

> Only suggestion would be to make the storm control limits
> consistent (i.e.: Juniper == 10% of 1g port right now,
> Cisco = 20%)

That's certainly something that can be revisited, the current values were somewhat arbitrary - our goal was to start off liberal to avoid any chance of tripping up normal traffic.

As a side note, we've had several times where the L2 security configs have shutdown individual member ports because of ingress BPDUs or MAC address limits so I'm sure they're working correctly.  The storm
control is a little harder to judge, logs show it kicking in during this afternoon's incident but the underlying bug made it ineffective.

Thanks everyone,
anthony