Topology changes in a spanning-tree domain are inevitable. But what actually happens, and what impact does it have on the environment? We will find it out in this article!
Topology
We will examine today’s scenarios on the 3-tier topology.
We’ve got one core, two aggregations, and four access switches. All of them are Cisco IOS devices. There are also four host devices, two connected to the ACC-1, one to ACC-2, and one to the ACC-4 switch. All of them are in the same subnet and VLAN.
The switches are interconnected with trunk links. All of them are configured with MST with two instances, VLAN 100 is part of instance number 1. Interfaces facing hosts 2,3 and 4 are configured as access ports with VLAN 100. The exception is Host-1, its link (G0/1 on ACC-1) is configured as a trunk with allowed VLAN 100. Its configuration is shown below.
interface GigabitEthernet0/1
description to_Host-1
switchport trunk allowed vlan 100
switchport trunk encapsulation dot1q
switchport mode trunk
negotiation auto
endIn such a topology, the Host-1 can be a server or another switch, but it doesn’t matter much. The following scenarios are based on simulating the link flap between Host-1 and ACC-1. We will observe the impact of TCN BPDU on the environment.
Scenario 1
In the first scenario, we will observe the behavior of the AGG-1 switch after receiving the TC BPDU.
First of all, let’s make sure that the MAC address of each host is present on the switch. I did it by issuing a ping to not assigned IP address from the same subnet (ex. 10.0.1.100).
Before sending a ICMP echo request message, a host needs a destination MAC address. In IPv4 a way to obain it is to generate ARP request, which is a broadcast frame. Since broadcast frames are populated to all devices in a broadcast domain, we have a MAC address entry of issuing host on each switch.
The MAC address table is complete. We have four hosts and four corresponding entries.
Before doing anything to our network, let’s enable some spanning-tree debugs on the ACC-1 switch. They will give us more information about what’s going on in our environment.
ACC-1#debug spanning-tree mstp state
MSTP port STATE changes debugging is on
ACC-1#debug spanning-tree events
Spanning Tree event debugging is onNow we’re ready to begin our tests. Let’s simulate a Host-1 link flap by shutting/no shutting it on the ACC-1 switch.
ACC-1(config)#int g0/1
ACC-1(config-if)#shutdown
ACC-1(config-if)#no shutdownNow let’s take a look at the logs from the ACC-1 switch.
*Apr 21 10:32:11.230: MST[1]: Gi0/1 state change forwarding -> disabled
*Apr 21 10:32:11.230: MST[0]: Gi0/1 state change forwarding -> disabled
*Apr 21 10:32:13.203: %LINK-5-CHANGED: Interface GigabitEthernet0/1, changed state to administratively down
*Apr 21 10:32:15.504: %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to up
*Apr 21 10:32:16.655: MST[0]: Gi0/1 state change disabled -> blocking
*Apr 21 10:32:16.656: MST[1]: Gi0/1 state change disabled -> blocking
*Apr 21 10:32:31.656: MST[0]: Gi0/1 state change blocking -> learning
*Apr 21 10:32:31.657: MST[1]: Gi0/1 state change blocking -> learning
*Apr 21 10:32:46.657: MST[0]: Gi0/1 state change learning -> forwarding
*Apr 21 10:32:46.657: MST[1]: Gi0/1 state change learning -> forwarding
*Apr 21 10:32:46.661: STP[0]: Generating TC trap for port GigabitEthernet0/1
*Apr 21 10:32:46.661: STP[1]: Generating TC trap for port GigabitEthernet0/1Let’s break it up into smaller chunks.
*Apr 21 10:32:11.230: MST[1]: Gi0/1 state change forwarding -> disabled
*Apr 21 10:32:11.230: MST[0]: Gi0/1 state change forwarding -> disabled
*Apr 21 10:32:13.203: %LINK-5-CHANGED: Interface GigabitEthernet0/1, changed state to administratively down
*Apr 21 10:32:15.504: %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to upHere we can see, that before the interface was admin down, it changed the STP state from forwarding to disabled. Then the interface went down, and two seconds later up.
*Apr 21 10:32:16.655: MST[0]: Gi0/1 state change disabled -> blocking
*Apr 21 10:32:16.656: MST[1]: Gi0/1 state change disabled -> blocking
*Apr 21 10:32:31.656: MST[0]: Gi0/1 state change blocking -> learning
*Apr 21 10:32:31.657: MST[1]: Gi0/1 state change blocking -> learning
*Apr 21 10:32:46.657: MST[0]: Gi0/1 state change learning -> forwarding
*Apr 21 10:32:46.657: MST[1]: Gi0/1 state change learning -> forwardingShortly after, the interface started its cycle through the states of the MST. It went from blocking to learning, and finally to the forwarding state.
*Apr 21 10:32:46.661: STP[0]: Generating TC trap for port GigabitEthernet0/1
*Apr 21 10:32:46.661: STP[1]: Generating TC trap for port GigabitEthernet0/1Here’s the most interesting thing. After entering the forwarding state, the MST generated TC BPDU for port G0/1. Recalling topology, it’s port facing Host-1, so link that flapped recently.
Before jumping to the AGG-1 console, let’s check the Wireshark. I’ve gathered packets between ACC-1 and AGG-1. Here are some of them.
Besides the cyclical MST packets(ex. first two), there are three packets with TC set. Let’s take a closer look at them.
Besides the standard MST information, the Topology Change (TC) flag is set.
Let’s jump to the AGG-1 switch and see, what’s the impact of TC BPDU on the MAC address table there.
After the link-flap occurred, two entries are gone from the VLAN 100 MAC address table. If we take a look at the topology, the remaining entries are from the interface whom which the TC BPDU came from. All entries from other interfaces disappeared. Down below there is our topology with grey area covering part of the network with disappeared MAC address entries.
Scenario 2
Now let’s examine how the ACC-1 switch behaves after the link flap. Again, we have to populate the MAC address table with all the entries.
Now we can begin with the test.
ACC-1(config)#int g0/1
ACC-1(config-if)#shutdown
ACC-1(config-if)#no shutdownAfter our action, there are logs generated from the debug trace, but they’re pretty much the same, so I’ll skip that.
Let’s now examine the MAC address table of the ACC-1 switch.
It’s empty, all entries disappeared.
The situation is similar to the previous example. The entries from all interfaces besides G0/1 are erased due to the topology change. So why the entry from the G0/1 is gone?
It’s because the switch always clears all entries from the interface that went down.
Scenario 3
We can suppress the generation of TC BPDU by configuring a portfast edge on a particular interface. Let’s try that!
As always, we need a complete MAC address table.
Now we can configure the portfast on the G0/1 interface.
ACC-1(config-if)#spanning-tree portfast edge trunk
%Warning: portfast should only be enabled on ports connected to a single
host. Connecting hubs, concentrators, switches, bridges, etc... to this
interface when portfast is enabled, can cause temporary bridging loops.
Use with CAUTION
We got a warning, and it’s a normal situation. Often, trunk links are configured between switches. In that case, we should configure spanning-tree on both devices. But in our situation, we have a server connected to G0/1, and we need a trunk connection. The server won’t participate in our spanning-tree topology, so it’s a good practice to configure a particular switch interface as an edge.
Now let’s flap a link!
ACC-1(config)#int g0/1
ACC-1(config-if)#shutdown
ACC-1(config-if)#no shutdownSince we still have enabled debugs, let’s check the debug trace.
*Apr 21 10:55:21.131: MST[1]: Gi0/1 state change forwarding -> disabled
*Apr 21 10:55:21.132: MST[0]: Gi0/1 state change forwarding -> disabled
*Apr 21 10:55:25.358: MST[0]: Gi0/1 state change disabled -> blocking
*Apr 21 10:55:25.359: MST[0]: Gi0/1 state change blocking -> forwarding
*Apr 21 10:55:25.359: MST[1]: Gi0/1 state change disabled -> blocking
*Apr 21 10:55:25.359: MST[1]: Gi0/1 state change blocking -> forwardingThe MST behavior changed a bit. First of all, we jumped from blocking straight to the forwarding state. In the previous scenarios, between those two, we had also a learning state, which is not present here. The second change is lack of TC log messages. We can assume, that TC BPDU wasn’t sent. Let’s check the MAC address table of VLAN 100.
It’s almost complete. One entry disappeared, but remember the previous scenario? Entries from flapped interfaces are always cleared!
Scenario 4
For the last scenario, let’s take a look at the interface G0/1 of the ACC-2 switch. There is Host-3 connected to it. It’s connected as access with VLAN 100. Here’s the configuration.
interface GigabitEthernet0/1
description to_Host-3
switchport access vlan 100
switchport mode access
negotiation auto
endLet’s now simulate the link flap here. We will see the environment behavior when a normal host flaps.
ACC-2(config)#int g0/1
ACC-2(config-if)#shutdown
ACC-2(config-if)#no shutdownI’ve enabled debug also on the ACC-2 switch, let’s take a look at the debug trace.
*Apr 21 11:05:54.548: MST[1]: Gi0/1 state change forwarding -> disabled
*Apr 21 11:05:54.548: MST[0]: Gi0/1 state change forwarding -> disabled
*Apr 21 11:05:56.523: %LINK-5-CHANGED: Interface GigabitEthernet0/1, changed state to administratively down
*Apr 21 11:05:57.523: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to down
*Apr 21 11:05:58.281: MST[0]: Gi0/1 state change disabled -> blocking
*Apr 21 11:05:58.282: MST[1]: Gi0/1 state change disabled -> blocking
*Apr 21 11:06:00.133: %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to up
*Apr 21 11:06:01.133: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to up
*Apr 21 11:06:13.282: MST[0]: Gi0/1 state change blocking -> learning
*Apr 21 11:06:13.282: MST[1]: Gi0/1 state change blocking -> learning
*Apr 21 11:06:28.282: MST[0]: Gi0/1 state change learning -> forwarding
*Apr 21 11:06:28.282: MST[1]: Gi0/1 state change learning -> forwarding
*Apr 21 11:06:28.284: STP[0]: Generating TC trap for port GigabitEthernet0/1
*Apr 21 11:06:28.284: STP[1]: Generating TC trap for port GigabitEthernet0/1The situation is probably clear for you (I hope so!). Besides MST state change, the TC was also generated. I’ve highlighted logs stating this.
You can suppress the generation of the TC BPDU by configuring portfast, but this time, since it’s an access port, the command would be different than before.
ACC-2(config-if)#spanning-tree portfast edgeTCN impact on the network
You might ask “Why do we even care for the generation of TC during link flaps? It’s a normal STP mechanism“. I agree, but it can also impact your network in a significant way.
Imagine, that you have a large network, that consists of dozens of L2 switches and hundreds of host devices connected to them. In such a case, a single link flap will erase a part/whole(depends on configuration and particular device) MAC address table on each switch. Without entries in the table, switches treat unicast traffic as unknown unicast (since they don’t know where the destination is) and are flooding frames through the entire broadcast domain. And this can make a mess in our environment. Let’s assume, that you’ve 1 Gb/s links to host devices, and just a couple of them are downloading something from the internet with the full link capacity. When the topology change occurs, those packets would be flooded through the entire broadcast domain, causing link congestion till the switches fill their MAC address tables. It won’t be long, but a lot of devices could experience packet loss during this time. The bigger broadcast domains, and more devices you have, the more impact it will have on the network.
Summary
I hope that this article somehow drew your attention to the potential impact that TC BPDU has on the network. Keep in mind, that the presented scenarios only covered one VLAN. The behavior may vary depending on the configuration of MST instances and VLANs on particular links.