Setting up Enhanced vPC on Nexus 5600

Sep 10, 2014 · 9 min read · Cisco Cisco Career Certifications Data center Network networking Nexus NX-OS tutorial vPC ·

So I've played around with two Cisco Nexus 5672UP switches which will be in production soon but before that I wanted to see what it took to get enhanced vPC online. Along the way I was into some new territory as I never stood up vPC before, so in this post I have posted some things to keep in mind and running-config examples, all of which I hope is helpful for a reference.

Before we go into the configuration of vPC I wanted to mentioned that the Nexus 5600 series switch is a newer product in the Cisco Data Center portfolio. I would guess that it would eventually replace the 5500 Nexus switch series but as of this writing Cisco has not put any end of life notice on the 5500 hardware. One of the main differences on the 5600 is the built-in layer 3 capabilities. The 5500 required a layer 3 card that used an expansion slot and only the higher-end Nexus products offered layer 3. To find out more information about the 5600 on Cisco's website: Cisco Nexus 5600 Platform Switches Data Sheet

So when I was looking at configuration examples I noticed some confusion between a dual-homed FEX setup and an enhanced vPC setup, so I'll try to clear the air. The "Enhanced" vPC is able to set up an etherchannel link from two different Nexus switches. The host/server that connects to these two different Nexus switches must support Active/Active NICs. This configuration is almost the same if you were setting up a dual-homed FEX but with just Active/Passive NICs setup on the host/server. So here is a question if the configuration is almost the same what needs to be added to support enhanced vPC? The software I am working with is version 7.0(1)N1(1) and we need to enable the features of vpc, fex and lacp.

1feature vpc
2feature fex
3feature lacp

We also need to have some VLANs created on both 5ks. In this example I created 25-35

1vlan 25-35

We need to start a VPC domain, this must be the same on both switches.

SWITCH 1:

1vpc domain 1 peer-keepalive destination 169.254.1.2 source 169.254.1.1

SWITCH 2:

1vpc domain 1 peer-keepalive destination 169.254.1.1 source 169.254.1.2

A note about the peer-keepalive, the 5500/5600 Nexus has a dedicated management NIC card on the front of the unit. (Where the power is) The reason I did this was I did not want to waste a 10Gb connection for just a keep-alive link. (Although it can be done :/ ) In this example I don't have a "dedicated" management network so I decided to use the management NIC of both 5ks. By default you have a another vrf context called management on the 5ks running-config. So on both 5ks I give that vrf context an IP address and directly connected both 5ks together.

SWITCH 1:

1interface mgmt0 vrf member management ip address 169.254.1.1/24

SWITCH 2:

1interface mgmt0 vrf member management ip address 169.254.1.2/24

Another helpful tip that I could not find out was if the management NICs would auto MDI/MDIX. So I just plugged in a straight-though and it works, so you don't need a crossover cable if you don't have one nearby. :)

We now need to create a vPC peer-link between the two Nexus switches. In this example I am using two 10Gb links, which are the same interfaces on both Nexus switches. (Consistency is your friend)

1interface Ethernet1/31
2  channel-group 1 mode active
3 
4interface Ethernet1/32
5  channel-group 1 mode active

Both switches are referring to port-channel one for interfaces 31 and 32. In the port-channel we will trunk the vlans we have created earlier along with setting this port-channel into a vpc peer-link.

1interface port-channel1
2  switchport mode trunk
3  switchport trunk allowed vlan 25-35
4  vpc peer-link

To keep the config short I am only referring to two FEXes in this example. (Remember so far in this post you would do exactly the same configuration if you just wanted to set up dual-homed FEXes) Starting with FEX 101 on SWITCH 1:

 1interface Ethernet1/1
 2  description FEX-101-PORT-1
 3  switchport mode fex-fabric
 4  fex associate 101
 5  channel-group 101
 6 
 7interface Ethernet1/2
 8  description FEX-101-PORT-3
 9  switchport mode fex-fabric
10  fex associate 101
11  channel-group 101
12 
13interface port-channel101
14  switchport mode fex-fabric
15  fex associate 101
16  vpc 101

Continue with FEX 101 on SWITCH 2:

 1interface Ethernet1/1
 2  description FEX-101-PORT-2
 3  switchport mode fex-fabric
 4  fex associate 101
 5  channel-group 101
 6 
 7interface Ethernet1/2
 8  description FEX-101-PORT-4
 9  switchport mode fex-fabric
10  fex associate 101
11  channel-group 101
12 
13interface port-channel101
14  switchport mode fex-fabric
15  fex associate 101
16  vpc 101

Start with FEX 102 on SWITCH 1:

 1interface Ethernet1/3
 2  description FEX-102-PORT-1
 3  switchport mode fex-fabric
 4  fex associate 102
 5  channel-group 102
 6 
 7interface Ethernet1/4
 8  description FEX-102-PORT-3
 9  switchport mode fex-fabric
10  fex associate 102
11  channel-group 102
12 
13interface port-channel102
14  switchport mode fex-fabric
15  fex associate 102
16  vpc 102

Continue with FEX 102 on SWITCH 2:

 1interface Ethernet1/3
 2  description FEX-102-PORT-2
 3  switchport mode fex-fabric
 4  fex associate 102
 5  channel-group 102
 6 
 7interface Ethernet1/4
 8  description FEX-102-PORT-4
 9  switchport mode fex-fabric
10  fex associate 102
11  channel-group 102
12 
13interface port-channel102
14  switchport mode fex-fabric
15  fex associate 102
16  vpc 102

So we have just setup two dual-homed FEXes on the 5672UP switches, here is where we can kick it up a notch to set up enhanced vPC. If I had a host that plugged into FEX 101 and FEX 102 and I wanted to set up etherchannel so that host could send traffic on out both links I would put the following configuration on BOTH switches:

1interface eth101/1/1, eth101/1/2
2channel-group 2 mode active
3interface eth102/1/1, eth102/1/2
4channel-group 2 mode active
5interface po2
6switchport access vlan 25

Here is the result, this is now an enhanced vPC. Some key points of this setup is be consistent with your configuration, it all has to match on both switches otherwise expect problems to arise. Enhanced vPC offers some high level redundancy, by just looking at the picture below you could have multiple failures and servers and applications would still be accessible. This makes this type of setup a common practice if business needs demand it. So I'll stop it there like before I hope this information is helpful as a reference. Feel free to comment below if you have any questions.

Static Comments:

John Naggets - Apr 5, 2015

What about using the QSFP ports instead of the Ethernet1/31 and Ethernet1/32 to connect both switches together using vPC, would that also work?

Ryan - Apr 5, 2015

Hey John, Yes I would think that would work. However I don't know if I would use the QSFP ports for a vPC link. Those type of ports would be better for uplinks. I don't think you would be running that much traffic thorough a vPC link unless something is wrong. From what I looked at it seemed like you would want to balance it out. So for example if I had an uplink on 5k-1 to Core-1, I would also want an uplink on 5k-2 going to Core-1. That way the traffic does not need to go through the vPC unless there is a problem. Hope that helps.

Joel - Dec 2, 2014

Hello, are you sure of your drawing ? have a look at Fig 6-10 on ^this CISCO page : http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/mkt_ops_guides/513_n1_1/n5k_enhanced_vpc.html

Zak Post - Jan 3, 2015

Thanks for this, it was easy to follow up to get a working configuration.

Ryan - Jan 3, 2015

Glad you were able to use it. :)

Nishant - Sep 5, 2015

Thanks, I found your post really helpful. I am in the process of implementing eVPC on 5672 UP. I have few questions with respect to the same, 1.You had mentioned that the host/server that connects to these different Fexes must support Active/Active NICs, does it mean that the host or server should support LACP? What if my host doesn't support LACP? 2.Can I dual home a VMware ESXi (active-active NICs on route based originating virtual port ID load balancing policy) to the fexes? … 3.Can I configure the fex port channel to static ( mode on) instead of active or lacp ? interface eth101/1/1, eth101/1/2 channel-group 2 mode on interface eth102/1/1, eth102/1/2 channel-group 2 mode on interface po2 switchport access vlan 25 Thanks in advance, Nishant

Ryan - Dec 2, 2014

Hey Joel, Thanks for pointing that out I fixed the image and thanks for the reference.

Ryan - Jan 1, 2015

Hey Kerry, Yes you would be able to add 4 ports in a port-channel to the FEXes. You could almost do the same thing with two 5600s without the FEXes if you wanted. Just setup a VPC and port-channel on both 5600s instead of using the FEXes.

Kerry - Jan 1, 2015

Thank you for this write up, great information. In my setup the host has 2 10g NICS for a total of 4 uplinks. Would I be able to follow your example here for PO2 but add in all 4 ports? (Ex 101/1/1-4 and 102/1/1-4

Ryan - Sep 1, 2015

Hey Nishant, You don't have to have Active/Active on the FEXs, you can have a single uplink from a host to a FEX and treat it like a standard port. The example that I showed is something you could do if you had a host that supported LACP. I also believe you can turn off LACP if you needed to and manually put up a port-channel on the FEX. Here is some more info: (Figure 6-5) http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/mkt_ops_guides/513_n1_1/n5k_enhanced_vpc.html

John - Jan 6, 2016

Does this setup really need 8 network connections between the two nexus 5k and the two nexus 2k? Would it also work with only 4 network connections instead? and in that case would it still be called "dual homed"? I am not sure to understand the advantage of 8 network connections.

Ryan - Jan 6, 2016

Hey John, You could use 4 connections instead of 8 you just loose the amount of bandwidth available instead of 40Gbps you would have 20Gbps that is available to the 2ks. It would also still be called a dual homed setup because that 2k switch is connected two different 5k switches. Ryan.

hostingnuggets - Apr 0, 2016

Ryan, I was wondering when connecting your server redundatly on Po2 shouldn't you specify a VPC? For instance you have: interface po2 switchport access vlan 25 but as far as I understand shouldn't it be assigned a VPC, for example: interface po2 switchport access vlan 25 vpc 2 ?

Ryan - Apr 1, 2016

You are correct if you would have connected the server directly to the 5ks you have to specify a VPC or if you had a single FEX connected to the 5k. In this example however I'm using FEXes which already have a VPC configured from both 5ks and I'm using port-channels off of those FEXes this is called "Enhanced vPC". There is a difference between them, when you create an Enhanced vPC you don't specify a VPC on the FEXes because there is already a vPC created upstream. Regular vPC with FEXes: LINK Enhanced vPC with FEXes: LINK Hope that Helps, Ryan