Going Deep With L2 Protocol Tunnels
- By Joe Astorino
- December 9, 2011
- 16 Comments
Introduction
Today we are going to talk about the elusive l2protocol-tunnel feature set of Cisco Catalyst switches. This feature is often confusing for people that are just learning it for a number of reasons.
First, people tend to confuse it with the L2 tunneling protocol (L2TP) and they are two totally different and independent things. L2TP is a “VPN” type technology that is primarily used to tunnel PPP traffic from one point to another over a layer 3 network. L2 protocol tunneling on the other hand, is a way for Cisco switches to tunnel specific layer 2 control protocols across a switched network — Namely these protocols are CDP, STP, VTP, PaGP and UDLD (As well as LLDP and LACP in more recent code!)
Secondly, people tend to confuse this technology with another popular tunneling technology known as Q-Q tunneling (Q in Q). Oftentimes we want to use l2protocol-tunneling in conjunction with Q-Q tunneling, but they are two different things that can be used independently of one another or together with each other. We will dig into Q-Q in another upcoming blog. Understanding l2protocol-tunneling by itself will help you when you combine it with Q-Q later on down the road.
Lastly, I see people confused about this topic a lot because well…you are masking things and making things magically appear different than what they physically are. That tends to confuse. Throw in the fact that the description of how it actually works from Cisco is fairly cryptic and you have a recipe for all sorts of fun!
Today, we will examine HOW the technology works and look at three very specific examples. We will start extremely basic, then move into an intermediate example and finally, a very complex example to demonstrate the technology. These labs are done on Cisco 3560 and 3550 switches.
How It Works
The purpose of this technology is so that two devices that happen to be physically separated by one or more devices in between can see each other as if they were directly connected from the perspective of these specific tunneled protocols. The goal is to make the L2 devices in the middle totally transparent from the perspective of the control plane traffic being tunneled. Keep in mind, we are not really talking about user data traffic here. User traffic will not be “tunneled”, only the control traffic for CDP, STP, VTP, PaGP and UDLD depending on configuration.
So, how does it work? It is actually pretty remarkably simple. You configure an ingress port with the l2protocol-tunnel command set and tell that port what control protocols you wish to tunnel from the device connected to that port. You configure the same thing on the other side where the traffic exits to the remote device you are tunneling to.
What happens is this — When the ingress interface configured for l2protocol-tunneling receives a frame that it identifies as being one of the protocols it needs to tunnel, it does some magic. Before we get to that magic, how does the switch receiving the frame know the traffic is one of the magical tunneled protocols? L2 multicast. Protocols like CDP, STP, etc are multicast at L2 to a Cisco proprietary MAC address that the switch knows about.
OK, so we receive a frame destined to one of these MAC addresses, say CDP. Now what? If the ingress port is configured for l2protocol-tunneling it actually rewrites the destination MAC address in hardware to another special proprietary Cisco MAC address reserved for tunneling these protocols. That MAC address is 01-00-0c-cd-cd-d0. At this point, the frame is flooded out every port in the native vlan of the ingress interface. Keep in mind, you can not configure this feature on a trunk port (or at least it won’t work if you do), so the native vlan will always be whatever access vlan the ingress interface happens to be in. When you configure l2protocol-tunnel on the interface, it basically tells the switch to do two things: encapsulation and decapsulation of the tunneled traffic. The encapsulation piece is what we just talked about — When the ingress interface receives a frame of a protocol that needs tunneled it “encapsulates” it by changing the destination MAC. The decapsulation piece tells the interface to “join” that special L2 multicast group and “listen” for messages destined to that MAC address. When the frame exits a switch on another interface configured for l2protocol-tunneling the egress switch “decapsulates” the frame by changing the destination MAC address back to what it was originally. Magic.
That all sounds pretty sweet, but if you are like me you need some hard concrete examples to drive this home. Let’s do it!
Basic L2protocol-Tunneling
Let’s start with something very simple. Imagine you have a single switch with two routers connected to it like this:
On any normal day firing off show cdp neighbor on R1 will show it is connected to Cat1 Fa0/1 and firing off the same command on R2 will show that Fa0/0 is connected to Cat1 Fa0/2, right? For the sake of our own sanity, let’s verify this before we screw with things.
R1#sh cdp neigh | b Device Device ID Local Intrfce Holdtme Capability Platform Port ID Cat1 Fas 0/0 142 S I WS-C3560- Fas 0/1
R2#sh cdp neigh | b Device Device ID Local Intrfce Holdtme Capability Platform Port ID Cat1 Fas 0/0 121 S I WS-C3560- Fas 0/2
Nice and simple. Now, let’s configure Cat1 so that R1 thinks it is connected directly to R2. R1 and R2 have no concept that this is happening — They simply send out their CDP frames blissfully unaware of what happens to them. Cat1 will then tunnel the CDP traffic between R1 and R2.
interface FastEthernet0/1 switchport mode access l2protocol-tunnel cdp no cdp enable ! interface FastEthernet0/2 switchport mode access l2protocol-tunnel cdp no cdp enable
Why the no cdp enable command though? We are disabling CDP on the two specific ports going to the routers. Why? When those switch ports receive a CDP frame from the routers we do not want the switch to process them. Instead, we want the switch to tunnel those frames using l2protocol-tunneling. Secondly, we don’t want the switch sending CDP frames to the routers either. Otherwise, R1 would see CDP information for R2 and Cat1 which would be even more confusing : ) Let’s see how we did
Cat1#show l2protocol-tunnel
COS for Encapsulated Packets: 5
Drop Threshold for Encapsulated Packets: 0
Port Protocol Shutdown Drop Encaps Decaps Drop
Threshold Threshold Counter Counter Counter
------------------- ----------- --------- --------- --------- --------- ---------
Fa0/1 cdp ---- ---- 4 4 0
--- ---- ---- ---- ---- ----
--- ---- ---- ---- ---- ----
--- ---- ---- ---- ---- ----
--- ---- ---- ---- ---- ----
--- ---- ---- ---- ---- ----
--- ---- ---- ---- ---- ----
Fa0/2 cdp ---- ---- 4 4 0
--- ---- ---- ---- ---- ----
--- ---- ---- ---- ---- ----
--- ---- ---- ---- ---- ----
--- ---- ---- ---- ---- ----
--- ---- ---- ---- ---- ----
--- ---- ---- ---- ---- ----
Things are looking good here. We have both encapsulated and decapsulated frames for CDP on both interfaces. It is important that we see both. If you only see either encapsulated or decapsulated changes are you forgot something along the way. Did it work?
R1#show cdp neigh | b Device Device ID Local Intrfce Holdtme Capability Platform Port ID R2 Fas 0/0 130 R S I 2811 Fas 0/0
R2#show cdp neigh | b Device Device ID Local Intrfce Holdtme Capability Platform Port ID R1 Fas 0/0 146 R S I 1841 Fas 0/0
That my friends is a beautiful thing! Let’s do the play by play commentary here. R1 sends a CDP frame. Cat1 Fa0/1 receives it, but it has CDP disabled so does not process the frame. Cat1 fa0/1 does however have the command l2protocol-tunnel cdp configured so it goes “Oh, I just received a CDP frame and I need to tunnel it. Let me change the destination MAC address to 01-00-0c-cd-cd-d0 and flood it out every port in the native VLAN.” And so it does, flooding the frame out every port in VLAN 1 in this case. At the same time, Cat1 also has l2protocol-tunnel cdp configured on Fa0/2 so it goes “Fa0/2 is listening for this frame, I need to decapsulate things and send the data there.” And so, Cat1 changes the destination MAC address back to the MAC for CDP and sends it out Fa0/2 to R2. The same process happens the other way around.
Intermediate L2protocol-Tunneling
Let’s make things a little more interesting…check this topology out. This time we have moved R2 over to Cat2 and then connected Cat1 and Cat2 together. I have removed the l2protocol-tunnel configuration on Cat1 Fa0/2 and shut down fa0/0 on R2.
We still have our l2protocol-tunnel configuration on Cat1 Fa0/1 so that does not need to change. What else do we need to add? Well, we simply put the same thing on the other side on Cat2 Fa0/2 to make this work.
interface FastEthernet0/2 switchport mode access l2protocol-tunnel cdp no cdp enable
Just to make sure everything is cleaned out I also ran clear cdp table on both R1 and R2…You also want to take special note to make your edge ports connected to the end devices access mode ports. If you fail to do this, the switch will take the commands, they just may not do anything : ) Let’s see how we did..
R1#show cdp neigh | b Device Device ID Local Intrfce Holdtme Capability Platform Port ID R2 Fas 0/0 157 R S I 2811 Fas 0/1
R2#sh cdp neigh | b Device Device ID Local Intrfce Holdtme Capability Platform Port ID R1 Fas 0/1 131 R S I 1841 Fas 0/0
Sweet! Everything looks good. R1 sends the CDP frame, and Cat1 Fa0/1 receives it. Cat1 changes the destination MAC to the l2protocol-tunnel MAC I have been talking about and floods it out every port in VLAN 1, including the cross switch link to Cat2 on Fa0/24. Cat2 receives the frame and floods it out every port in VLAN1, including Fa0/2 connected to R2 Fa0/1. When the frame exits the Fa0/2 interface, Cat2 rewrites the destination MAC address back to the standard CDP MAC address. Let’s move on to the more insane stuff.
*NOTE* I found that for this feature to work consistently when tunneling through more than one switch, the interswitch link needs to be configured as a trunk. Even if the trunk is not really doing anything, as is the case here (one VLAN) it seems to trigger something in the code that makes l2protocol-tunnel encapsulation/decapsulation work properly
Advanced L2protocol-Tunneling
So far we have seen how we can tunnel the CDP protocol between two devices connected to the same switch, and between two devices connected to two different switches. Now it is time to go all out. Take in this diagram
What we have here between Cat1 and Cat4 is a two port etherchannel that we will setup to run PaGP. Cat1 has a third link, Fa0/24 which goes to Cat2, which is then connected to Cat3 and finally to Cat4. What we will do is this: We will tunnel CDP, STP from Cat1 Fa0/24 through Cat2 and Cat3 and down to Cat4 Fa0/24. Basically, from the perspective of CDP and STP on Cat1 and Cat4 they are directly connected over Fa0/24 even though that control plane traffic is being tunneled through Cat2 and Cat3. Finally, we will join Fa0/24 to the existing etherchannel, making it a 3 channel etherchannel. Logically, the etherchannel is directly connected but of course physically only 2 out of the 3 links are physically point to point with the third being tunneled around the topology.
The basic connectivity and port-channel has already been configured. We will verify a few things
Cat1#show spanning-tree | i VLAN|is the root
VLAN0001
This bridge is the root
Cat1#sh etherchan sum | b Group
Group Port-channel Protocol Ports
------+-------------+-----------+-----------------------------------------------
14 Po14(SU) PAgP Fa0/19(P) Fa0/20(P)
Cat1#sh cdp neigh | b Device
Device ID Local Intrfce Holdtme Capability Platform Port ID
Cat4 Fas 0/19 165 S I WS-C3550- Fas 0/19
Cat4 Fas 0/20 165 S I WS-C3550- Fas 0/20
Cat2 Fas 0/24 178 S I WS-C3550- Fas 0/24
Cat2#sh cdp neigh | b Device Device ID Local Intrfce Holdtme Capability Platform Port ID Cat3 Fas 0/19 144 S I WS-C3550- Fas 0/19 Cat1 Fas 0/24 146 S I WS-C3560- Fas 0/24
Cat3#sh cdp neigh | b Device Device ID Local Intrfce Holdtme Capability Platform Port ID Cat4 Fas 0/24 179 S I WS-C3550- Fas 0/24 Cat2 Fas 0/19 145 S I WS-C3550- Fas 0/19
Cat4#sh etherchan sum | b group Number of channel-groups in use: 1 Number of aggregators: 1 Group Port-channel Protocol Ports ------+-------------+-----------+----------------------------------------------- 14 Po14(SU) PAgP Fa0/19(P) Fa0/20(P) Cat4#sh cdp neigh | b Device Device ID Local Intrfce Holdtme Capability Platform Port ID Cat3 Fas 0/24 134 S I WS-C3550- Fas 0/24 Cat1 Fas 0/20 145 S I WS-C3560- Fas 0/20 Cat1 Fas 0/19 145 S I WS-C3560- Fas 0/19
Looks good. Now for the fun stuff. Before we mess with PaGP, lets just get the basic CDP and STP tunneling working. We need to configure this on our ingress/egress points of the tunnel. That would be on Cat2 and Cat3 Fa0/24. Note, that again the “SP” switches Cat2 and Cat3 are linked together with a dot1q trunk to make this feature work consistently between more than one switch, even though we are only playing with VLAN 1 everywhere.
Cat2#conf t Enter configuration commands, one per line. End with CNTL/Z. Cat2(config)#int fa0/24 Cat2(config-if)#switchport mode access Cat2(config-if)#no cdp enable Cat2(config-if)#l2protocol-tunnel cdp Cat2(config-if)#l2protocol-tunnel stp
Cat3#conf t Enter configuration commands, one per line. End with CNTL/Z. Cat3(config)#int fa0/24 Cat3(config-if)#switch mode access Cat3(config-if)#no cdp enable Cat3(config-if)#l2protocol-tunnel cdp Cat3(config-if)#l2protocol-tunnel stp
We’ll go clear the CDP tables on Cat1 and Cat4 and give the l2protocol-tunnel some time to bake…after doing that and a few minutes lets check our l2protocol-tunnel status
Cat2#show l2protocol-tunnel
COS for Encapsulated Packets: 5
Drop Threshold for Encapsulated Packets: 0
Port Protocol Shutdown Drop Encapsulation Decapsulation Drop
Threshold Threshold Counter Counter Counter
---------- -------- --------- --------- ------------- ------------- -------------
Fa0/24 cdp ---- ---- 21 21 0
stp ---- ---- 641 0 0
Cat3#show l2protocol-tunnel
COS for Encapsulated Packets: 5
Drop Threshold for Encapsulated Packets: 0
Port Protocol Shutdown Drop Encapsulation Decapsulation Drop
Threshold Threshold Counter Counter Counter
---------- -------- --------- --------- ------------- ------------- -------------
Fa0/24 cdp ---- ---- 22 23 0
stp ---- ---- 0 662 0
Cool. Is it working from an endpoint perspective? Let’s look at CDP on Cat1
Cat1#sh cdp neigh | b Device Device ID Local Intrfce Holdtme Capability Platform Port ID Cat4 Fas 0/24 152 S I WS-C3550- Fas 0/24 Cat4 Fas 0/20 120 S I WS-C3550- Fas 0/20 Cat4 Fas 0/19 120 S I WS-C3550- Fas 0/19
Beautiful…we have our two links in the port channel as well as Fa0/24 showing directly connected! On Cat3 we should see the same thing.
Cat4#sh cdp neigh | b Device Device ID Local Intrfce Holdtme Capability Platform Port ID Cat1 Fas 0/24 157 S I WS-C3560- Fas 0/24 Cat1 Fas 0/20 145 S I WS-C3560- Fas 0/20 Cat1 Fas 0/19 145 S I WS-C3560- Fas 0/19
What about spanning-tree? Cat1 is the root bridge here. When you think about things as they are now from a spanning-tree perspective we now have 2 logical links between Cat1 and Cat4 as far as STP is concerned. We have Po14 (seen as one logical link by STP) and we have Fa0/24. The port channel on Cat4 should be the root port and forwarding. If our STP is truly being tunneled, Fa0/24 on cat4 should be blocking because it is neither a root or designated port. Let’s see…
Cat4#show span | b Interface Interface Role Sts Cost Prio.Nbr Type ------------------- ---- --- --------- -------- -------------------------------- Fa0/24 Altn BLK 19 128.24 P2p Po14 Root FWD 12 128.65 P2p
Awesome. From Cat1/Cat4’s perspectives, they are directly connected on Fa0/24. Now let’s add to the insanity, and throw Fa0/24 into the port channel.
Cat2#conf t Enter configuration commands, one per line. End with CNTL/Z. Cat2(config)#int fa0/24 Cat2(config-if)#l2protocol-tunnel point-to-point pagp
Cat3#conf t Enter configuration commands, one per line. End with CNTL/Z. Cat3(config)#int fa0/24 Cat3(config-if)#l2protocol-tunnel point-to-point pagp
Cat1#conf t Enter configuration commands, one per line. End with CNTL/Z. Cat1(config)#int fa0/24 Cat1(config-if)#channel-group 14 mode desirable
Cat4#conf t Enter configuration commands, one per line. End with CNTL/Z. Cat4(config)#int fa0/24 Cat4(config-if)#channel-group 14 mode desirable
Did it work?
Cat2#sh l2protocol-tunnel | b pagp
pagp ---- ---- 18 6 0
--- ---- ---- ---- ---- ----
--- ---- ---- ---- ---- ----
Cat3#show l2protocol-tunnel | b pagp
pagp ---- ---- 7 8 0
Cat1#sh etherchan sum | b Group Group Port-channel Protocol Ports ------+-------------+-----------+----------------------------------------------- 14 Po14(SU) PAgP Fa0/19(P) Fa0/20(P) Fa0/24(P) Cat1#show span | b Interface Interface Role Sts Cost Prio.Nbr Type ------------------- ---- --- --------- -------- -------------------------------- Po14 Desg FWD 9 128.160 P2p
Cat4#sh etherchan sum | b Group Group Port-channel Protocol Ports ------+-------------+-----------+----------------------------------------------- 14 Po14(SU) PAgP Fa0/19(P) Fa0/20(P) Fa0/24(P) Cat4#show span | b Interface Interface Role Sts Cost Prio.Nbr Type ------------------- ---- --- --------- -------- -------------------------------- Po14 Root FWD 9 128.65 P2p
That about does it for l2protocol-tunneling!
Keep in mind that in this case, we are simply tunneling control traffic and using l2protocol-tunneling alone. The only traffic that is actually tunneled is the control traffic for STP, CDP and PaGP. User traffic / normal data is not “tunneled” per say. But you say “Joe, if 1 of the 3 port-channel links is tunneled, how can the data not be? Don’t some of the frames / packets still have to pass through Cat2/Cat3?” Excellent question. The data itself is not “tunneled” even though some of it passes through the SP switches because the destination mac address is only changed for these specific control protocols.
If you use plain l2protocol-tunneling without the use of Q-Q tunneling at the same time, you only have one choice for your tunneling ports — they have to be access ports (the only ports you can configure this on are access and dot1q-tunnel ports). This means that with this technology alone, every customer can only pass data on a single VLAN because the customer device will not be capable of trunking multiple VLANs to the SP edge switch. It also does not do a good job of separating customer and SP traffic very well. If you have a need to pass more than 1 VLAN through the SP, Q-Q tunneling is really the technology you want to look at. You can then run l2protocol-tunneling on top of that to do what we have done here in conjunction with actually passing multiple VLANs and the SP has the benefit of keeping everything separated at the same time
great job :)
Great job Joe! I’ve never used L2 Tunneling mostly because I didn’t know why you would want to. But now that I know what it’s for I may give it a try.
There is a minor typo in one of your outputs that had me confused for a second. Under the Advanced part of the article, right after you configured the tunneling for CDP and STP your show l2protocol-tunnel command output is shifted to the right so that it looks like no traffic was encapuslated, only decapsulated. I couldn’t understand how the switches were decapsulating traffic but nobody was encapuslating it! :P
Hey there Josh! I JUST finished updating the article. There were some major issues in the configuration that needed fixed! If something looked weird to you in the configuration, I recommend you give it another read now. I also fixed that cosmetic issue. Thanks for reading!
Nice post, Joe. One note regarding trunk port configuration: Whether it is “supposed to” or not, the 3750 will allow L2PT to be configured on a trunk port, and respects the VLAN assignments of any tagged control-plane frames that come in. This is the secret sauce to getting a Dynamips + real switch study lab working reliably with correct CDP in both directions when using a break-out switch. The Dynamips server is not processing those control protocol frames, so it encapsulates them with a VLAN header and passes them to the 3750. The 3750 accepts them even though they’re VLAN tagged, and associates them with the correct VLAN.
@Bob
I was just digging into this today, and you are correct. Technically the documentation states what is below (3560 software config guide)…
Notice it says you configure it on access or dot1q tunnel ports and that you can NOT configure it on ports configured for DTP (dynamic), but it never mentions anything about a hard static trunk (switchport mode trunk). Go figure : )
“You can enable Layer 2 protocol tunneling on ports that are configured as access ports or tunnel ports. You cannot enable Layer 2 protocol tunneling on ports configured in either switchport mode dynamic auto (the default mode) or switchport mode dynamic desirable. “
[…] back everybody! Last week we covered a tunneling technology known as l2protocol-tunneling or L2pt here on the blog. If you missed that article, you may want to go through and get a handle on that technology […]
Hi,
This is very interesting.I like this.I will come back and read again and get more information about this.Thanks for this post.
Joe,
I have been to your blog a couple of times and today, I am writing to thank you for an excellent explanation of l2protocol-tunneling. I am studying for my CCIE (R&S) lab exam and was just labbing l2pt today on my lab rack. I actually tested it with the ether-channel over l2pt and thought it’s pretty cool. Thank you for explaining the MAGIC in lil more detail.
You are doing a great job. Keep it up.
@Khan — Thanks man, I appreciate that! I’m always thrilled to know that anything I contribute is helping people out there. I remember when I first was wrapping my head around L2PT (studying for my R&S CCIE as you are), and there were not a lot of great resources to answer some of these questions I had. Glad I could help, and good luck going into your lab exam!
[…] http://astorinonetworks.com/2011/12/09/going-deep-with-l2-protocol-tunnels/ […]
Hi,
This is a very useful post. Thank you for your sharing.
You did a very good job :)
Hi JOe,
very useful article. but if we use EFP at ME3600 with l2protocol tunnel and also a customer VLAN X-qinq with l2protocol tunnel , will the bpdu of customer interfere with ISP?
So the diagram is as below:
Customer–Switch A—Core Switches–ME3600(1) —L3—-ME3600 (2)–SwitchB—Customer
On the port at Switch A facing customer the port is configured for qinq vlan X and also l2protocol tunnel enabled
Switch A port toward Core switches is configured as Trunk allowing the vlan X.
At ME3600 (facing Core switches) is configured service instance with xconnect (IP of ME3600 -2) encapsulation default and also l2protocol tunnel enabled.
Also the same thing at ME3600 at port facing the switch B is configured service instance encapsulation default, xconnect ip of ME36001 and also l2protocol tunnel enabled
So the question is, will the BPDU of Customer network (or VTP) will interfere with BPDU (or VTP of Core switches)
Sorry for my long long explanation.
Regards
[…] http://astorinonetworks.com/2011/12/09/going-deep-with-l2-protocol-tunnels/ […]
Hi Joe.
First of all, thank you for this excellent article.
It’s not clear to me why data traffic is not going through Po14 interface Fa0/24 (l2 tunnel) and just control traffic passes. It’s not possible to form LACP through tunneled QinQ, for examepl? Could you explain to me?
Appreciate.
Fa0/24 at the end of this example is bundled as a part of Po14. Thus, any data traffic that Cat1 hashes out of the fa0/24 link of Po24 is indeed going towards Cat2 and Cat3 and being switched as normal by those switches.
While the data traffic leaving fa0/24 as part of the port channel is going towards cat2 and cat3 I say it is not tunneled “per say” because Cat2 and Cat3 still need to do traditional mac learning for this traffic just like any normal traffic. The control plane traffic like PaGP is truly tunneled because destination MAC addresses for the control plane traffic are being changed. For the data plane, the traffic is just being switched as usual. You could see that either way I suppose.
So yes, some of the traffic between Cat1 and Cat4 is physically going from Cat1 Fa0/24 —> Cat2 –> Cat3 –> Cat4. However, the data plane traffic is not technically being tunneled.
HTH
Thank you for the excellent explanation, it’s clear now.