Migrating from fabricpath to EVPN/VxLAN


Do you have a 3 tier, switched, or vendor proprietary data center design?

Does it rely on spanning tree or proprietary solutions to eliminate spanning tree?

Not sure how to migrate to a new architecture without serious downtime?

If you answered yes to any of these questions then this post is for you. We’ll be looking at deploying an EVPN/VxLAN Data Center fabric and migrating from a cisco fabricpath environment to the new design.

Although we will be focusing on a fabricpath migration many, if not all, of the principles apply to migrating a 3 tier architecture.

1. Building the new Data Center Fabric
2. Connecting the current fabricpath and new fabric
3. Migrating switched virtual interfaces
4. Migrating various types of physical devices

Building the new Data Center Fabric

The easiest part of designing and building the new fabric is the physical topology. This should be a symmetric topology to easily take advantage of equal cost multipath and add additional switches with ease. This is also known as a spine/leaf or clos topology. The basic idea is leafs connect to spines and spines connect to super spines. A leaf/spine should not connect to another switch of the same type expect for multichassis lag or virtual port-channel at the access layer if you’re utilizing this.


ISIS as an underlay routing protocol

Next you must decide on routing protocols. We will not examine layer 2 as this will be a completely routed fabric eliminating the need for any STP in your datacenter. Remember if you’re not Facebook, Amazon, Netflix, or Google (FANG) or some other webscaler you probably don’t have FANG problems i.e. there is no need to run a BGP underlay and learn to turn all the associated knobs to make that work; nor to engage in troubleshooting complex problems like path hunting.

For this reason we will look at utilizing Intermediate System to Intermediate System (ISIS) as an underlay with Internal Border Gateway Protocol (iBGP) as an overlay.

We prefer ISIS as an underlay network for data centers because:

  • it is easier to scale than OSPF
  • is extensible from the beginning (Type Length Values for additional capabilities)
  • better stability at scale

The secondary loopback is to enable the advertisement of a virtual IP address for traffic destined to the vPC pair. Single attached or routed links will advertise the physical IP address of the leaf so traffic returns the that specific leaf and not the pair.

iBGP as the overlay

The overlay is pretty straight forward. We will run iBGP with loopback peerings to exchange EVPN routes. EVPN scales significantly better than other VxLAN control plane protocols so we will not explore flood and learn or static assignment.

We will be utilizing vPC on the access layer for the remainder of the post. There are other methods for dual attached devices such as EVPN-multihoming but as this is cisco specific for fabricpath migrations they will not be discussed.

See an example configuration below of how the VIP/PIP mentioned earlier operate

Leaf BGP and NVE

interface nve1
  no shutdown
  host-reachability protocol bgp
  advertise virtual-rmac ## for advertising the VIP 
  source-interface loopback1

router bgp 8675309
  address-family l2vpn evpn
    advertise-pip ## for advertising PIP if single attached
    remote-as 8675309
    update-source loopback0
    address-family l2vpn evpn
      send-community extended

Connecting the current fabricpath DC and new fabric

The first thing to do is decide on the physical point of interconnection. You’ll want to ensure you chose a place you have enough ports to do a dual sided vPC with enough bandwidth to cover lateral traffic between new/old until the migrations are complete.

Next we have to think about the layer 2 protocols in play. Since spanning tree isn’t in play on either side we need to take special consideration to make sure we do not introduce a layer 2 loop.

The EVPN/VxLAN side will not do anything with STP BPDUs but there is a requirement on the fabricpath side that it remains the root bridge. This is due to the entire fabricpath domain looking like one physical bridge. If a port in the fabricpath domain receives a superior BPDU a root-guard of sorts is enacted and the content edge port begins blocking.

Why do we care if STP doesn’t pass over the EVPN fabric? If the fabricpath environment is interconnected at two points then there will be a loop back to the fabricpath domain. This is a situation we want to avoid.

It can be avoided by:

  1. only having one interconnect
  2. manually pruning vlans at the two+ points of interconnect to ensure vlans remain on exactly ONE path

Migrating Switched Virtual Interfaces

Our preferred method of migrating SVIs from the old fabricpath environment to the new fabric is to:

  • build all of the new Distributed Anycast Gateways (DAG) on the new fabric
    • keep them shutdown
  • establish a L3 adjacency via BGP for routing traffic back to exit points until the migrations are complete
  • add the VLANs being migrated to the dual side vPC
  • shutdown the SVIs on the fabricpath side and no shut the DAGs on the new fabric
  • manually clear ARP on any hosts that did not update with the new DAG MAC

Migrating physical devices

Most of the physical devices are “easy” since there is no option but to physically move cables and you know this will result in a slight outage while the new uplinks come online.

However, with HA pairs of devices it is possible to migrate by moving the standby unit, waiting for the HA to reestablish, forcing a failover, move the active unit, and then “fail” back to the primary unit. This will test your HA setup as well as provide a seamless migration.

If you have new compute and storage you can migrate your workloads directly to the new environment and age out the legacy compute/storage.

Finally, ensure there are no more devices in use on your old environment and decommission the devices.

If you have questions or need assistance do not hesitate to reach out to us at ip architechs.

Utilizing BGP Communities for traffic steering – part 1: Firewalls


I typically spend more time in the enterprise data center than most of our team members and this comes with its own unique set of problems. One discussion that seems to never fail to come up is “where do I put the Firewalls (FWs)?”. That is typically followed by I have a disaster recovery or backup site with FWs there as well. This inevitably leads to a state management problem. Let’s look at how we can utilize BGP to address this problem:

  • what is a BGP standard community
  • BGP best path selection process
  • how to utilize them to steer traffic

This is something most service providers deal with on a daily basis but can be new to an enterprise.

BGP Standard communities

A BGP community is a route attribute that, essentially provides extra information for someone to take action or glean information from the route such as where it came from (location, type, organizational role).

By definition, a community is a 32 bit number that can be included with a route and when utilizing the new community format is displayed as (0-65535):(0-65535). It is recommend to utilize the new community format versus the old community format which is just a number. It is typically to utilize $YOURASN:$VALUE such as 65000:1000 which would be a community within ASN 65000 to signify something such as 1000 came from datacenter 1. This insures that you know the community was originated by your organization.

There are some well known communities that have global meaning. BGP communities are also what is called an optional transitive BGP attribute meaning that they can be passed from autonomous system (ASN) to autonomous system. It is typically recommended to strip communities sent from other organizations to prevent interference with your local policy. Telia has a great looking glass utility (https://lg.telia.net)that gives information on the communities they’ve attached to your routes.

BGP path selection

Now that we know what is a community is lets look at the BGP path selection process. It’s not uncommon for vendors to have vendor specific selection criteria with “weight” being at the top. However, typically local preference is the first with highest being the best. Then locally originated, shortest AS path, origin check, and Multi Exit Discriminator (MED). There are more selection criteria than this but we will focus on local preference (LP) since it is at the top of the list.

Modifying local preference will ALWAYS beat one of the other criteria with the exception of a weight value where applicable. For example in a Cisco environment the highest weight will beat the LP, if the weight is equal it will move to LP.

You might have noticed that there is no “community” value in the path selection process. That is because we are going to combine the community value with a policy to take an action on a route. Let’s tie this back to firewalls, security zones, and policy in the next section.

Modifying behavior with communities and LP

There are two ways to extend a security zone through a firewall. You can either route and utilize a Virtual Routing and Forwarding instance (VRF) or switch and put the Switched Virtual Interface (SVI) on the firewall. Switching limits your options as you scale and is not recommended. For this reason we will focus on utilizing VRFs. In this case we have VRF blue with community 65000:1 and VRF Orange with community 65000:2. These are segregated by firewalls and have different security policy.

With two firewalls it is easy to introduce a state problem and have packets rejected because of invalid state. When a packet ingresses one firewall and egresses another this introduces the problem as firewall 2 does not have a session for the flow. “How come we don’t run an HA pair?” is a common question; these might already be an HA pair and you’re preparing for a migration, they’re different vendors, or they’re in different data centers completely.

So now I can tie communities and local preference (LP) together to pin flows to firewalls and manage state. I can raise or lower the LP based on the community to ensure routing always returns to the firewall with the active session state. If the firewalls are in different data centers you can even do this to move flows of the same security level below the firewalls and maintain state across the DCs. Doing so involves advanced network techniques and policy configurations which IParchitechs has successfully implemented in dozen of DCs in spite of vendor attestation that this is not feasible.