Print

Print


Well defined and thought out response
On Jan 30, 2015 8:55 PM, "Anthony Anderberg" <
[log in to unmask]> wrote:

> I was on the road this afternoon and unable to safely contribute or
> comment, but it looks like everyone came together and correctly diagnosed
> and resolved the issues.  I thought I'd follow up to a few
> questions/comments to make sure all is clear.
>
> > clear ethernet-switching table
>
> From the logs, comments, and graphs I'm sure we ran into that software bug
> again, regardless of any original leaf-node issues.  The switch gets into a
> state where MAC addresses are not all correctly added to the L2 forwarding
> database so some destinations are treated as broadcasts.  After issuing the
> command above the switch recovers from its funk and the table populates
> normally.  Once our maintenance is reinstated we'll try to carve out some
> time to open a ticket, and upgrade software if needed.
>
> Thanks to our friends at Wistrom Telephone for stepping up to the plate
> once again on that.  Speaking of which, given how important the exchange
> seems to be to members I assume everyone will be eager to offer both time
> and financial support as needs arise in the future.  There's nothing magic
> happening here, just hard work and capital... same as any endeavor.
>
> > It doesn't look like Anthony applied the controls onto
> > the leaf switches, like Mankato Networks', that is
> > currently not applied.
>
> Also correct, our L2 security plans centered around member facing ports -
> the definition of which may shift over time, along with configuration.
>
> Note that we do have spanning tree running between the exchange's three
> switches, and it appears to have worked properly to change the state of the
> Mankato switch port, but switching-table bug reared its ugly
> head at the same time.
>
> > Only suggestion would be to make the storm control limits
> > consistent (i.e.: Juniper == 10% of 1g port right now,
> > Cisco = 20%)
>
> That's certainly something that can be revisited, the current values were
> somewhat arbitrary - our goal was to start off liberal to avoid any chance
> of tripping up normal traffic.
>
> As a side note, we've had several times where the L2 security configs have
> shutdown individual member ports because of ingress BPDUs or MAC address
> limits so I'm sure they're working correctly.  The storm
> control is a little harder to judge, logs show it kicking in during this
> afternoon's incident but the underlying bug made it ineffective.
>
> Thanks everyone,
> anthony
>