WISP Design – Building Highly Available VPLS for Public Subnets

What is VPLS?

Virtual Private LAN Service or VPLS is a Layer 2 overlay or tunnel that allows for the encapsulation of ethernet frames (with or without VLAN tags) over an MPLS network.

https://tools.ietf.org/html/rfc4762

VPLS is often found in Telco networks that rely on PPPoE to create centralized BRAS deployments by bringing all of the end users to a common point via L2.

MikroTik VPLS example (https://wiki.mikrotik.com/wiki/Transparently_Bridge_two_Networks_using_MPLS)

Wlink

Background

The idea for this post came out of a working session (at the bar of course) at WISPAmerica 2018 in Birmingham, Alabama.

There was a discussion about how to create redundancy for VPLS tunnels on multiple routers. I started working on this in EVE-NG as we were talking about it.

The goal is creating highly available endpoints for VPLS when using them to deploy a public subnet that can be delivered to any tower in the WISP. The same idea works for wireline networks as well.

Use Case

As IPv4 becomes harder to get, ISPs like WISPs, without large blocks of public space find it difficult to deploy them in smaller subnets. The idea behind breaking up a /23 or /24 for example, is that every tower has public IP addresses available.

However, the problem with this approach is that some subnets may not be used if there isn’t much demand for a dedicated public IP by customers.

What makes VPLS attractive in this scenario is that the public subnet (a /24 in this example) can be placed at the data center as an intact prefix.

VPLS tunnels then allow for individual IP addresses to exist at any tower in the network which provides flexibility and conserves IPv4 space by not subnetting the block into /29 /28 /27 at the tower level.

Lab Network

VPLS

Deployment

In this lab, the VPLS tunnels terminate in two different data centers as well as at a tower router to create an L2 segment for 203.0.113.0/24. VRRP is then run between the two data center VPLS routers so that the gateway of 203.0.113.1 can failover to the other DC if needed.

Failover

Here is an example of the convergence time when we manually fail R1 and the gateway flips over to R2 in the other DC. The yellow highlight marks the point where R1 has failed and R2 VRRP has become master.

vpls-failover

Configurations

R1-vpls-agg

/interface bridge
add name=Lo0
add name=vpls1-1
/interface vrrp
add interface=vpls1-1 name=vpls1-1-vrrp priority=200
/interface vpls
add disabled=no l2mtu=1500 mac-address=02:2C:0B:61:64:CB name=vpls1 remote-peer=1.1.1.2 vpls-id=1:1
add disabled=no l2mtu=1500 mac-address=02:7C:8C:C9:CE:8E name=vpls2 remote-peer=1.1.1.3 vpls-id=1:1
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/interface bridge port
add bridge=vpls1-1 interface=vpls1
add bridge=vpls1-1 interface=vpls2
/ip address
add address=1.1.1.1 interface=Lo0 network=1.1.1.1
add address=10.1.1.1/24 interface=ether1 network=10.1.1.0
add address=203.0.113.2/24 interface=vpls1-1 network=203.0.113.0
add address=203.0.113.1/24 interface=vpls1-1-vrrp network=203.0.113.0
/ip dhcp-client
add disabled=no interface=ether1
/mpls ldp
set enabled=yes lsr-id=1.1.1.1 transport-address=1.1.1.1
/mpls ldp interface
add interface=ether1
/routing ospf network
add area=backbone network=10.1.1.0/24
add area=backbone network=1.1.1.1/32
/system identity
set name=R1-vpls-agg

R2-vpls-agg

/interface bridge
add name=Lo0
add name=vpls1-1
/interface vrrp
add interface=vpls1-1 name=vpls1-1-vrrp
/interface vpls
add disabled=no l2mtu=1500 mac-address=02:C3:4C:31:FB:C9 name=vpls1 remote-peer=1.1.1.1 vpls-id=1:1
add disabled=no l2mtu=1500 mac-address=02:02:34:C0:A3:3C name=vpls2 remote-peer=1.1.1.3 vpls-id=1:1
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/interface bridge port
add bridge=vpls1-1 interface=vpls1
add bridge=vpls1-1 interface=vpls2
/ip address
add address=10.1.1.2/24 interface=ether1 network=10.1.1.0
add address=1.1.1.2 interface=Lo0 network=1.1.1.2
add address=203.0.113.3/24 interface=vpls1-1 network=203.0.113.0
add address=203.0.113.1/24 interface=vpls1-1-vrrp network=203.0.113.0
/ip dhcp-client
add disabled=no interface=ether1
/mpls ldp
set enabled=yes lsr-id=1.1.1.2 transport-address=1.1.1.2
/mpls ldp interface
add interface=ether1
/routing ospf network
add area=backbone network=10.1.1.0/24
add area=backbone network=1.1.1.2/32
/system identity
set name=R2-vpls-agg

R3-Tower-1

/interface bridge
add name=Lo0
add name=vpls-1-1
/interface vpls
add disabled=no l2mtu=1500 mac-address=02:CB:47:7A:92:0B name=vpls1 remote-peer=1.1.1.1 vpls-id=1:1
add disabled=no l2mtu=1500 mac-address=02:E3:C5:5B:EC:BF name=vpls2 remote-peer=1.1.1.2 vpls-id=1:1
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/interface bridge port
add bridge=vpls-1-1 interface=ether1
add bridge=vpls-1-1 interface=vpls1
add bridge=vpls-1-1 interface=vpls2
/ip address
add address=10.1.1.3/24 interface=ether2 network=10.1.1.0
add address=1.1.1.3 interface=Lo0 network=1.1.1.3
/ip dhcp-client
add disabled=no interface=ether1
/mpls ldp
set enabled=yes lsr-id=1.1.1.3 transport-address=1.1.1.3
/mpls ldp interface
add interface=ether2
/routing ospf network
add area=backbone network=10.1.1.0/24
add area=backbone network=1.1.1.3/32
/system identity
set name=R3-tower-vpls

Put 500,000+ BGP routes in your lab network!!! Download this VM and become your own upstream BGP ISP for testing.

[adrotate banner=”5″]

 

Happy New Year and welcome to the VM you can punish your routers with 🙂

Hello from stubarea51.net and Happy New Year! We are back from the holidays and recharged with lots of new stuff in the world of network engineering. If you ever thought it would be cool to put a full BGP table into a lab router, GNS3 or other virtualized router, you’re not alone.

A while back, I tackled this post and got everything up and running:

http://evilrouters.net/2009/08/21/getting-bgp-routes-into-dynamips-with-video/

First of all, thanks to evilrouters.net for figuring out the hard parts so we could build this into a VM. After basking for a while in the high geek factor of this project, it gave me an idea to build a VM that could be distributed among network engineers and IT professionals. The idea is to easily spin up one or more full BGP tables to test a particular network design or convergence speed, playing with BGP attributes, etc. After a few months of tweaking it and getting the VM ready for distribution, we finally are ready to put it out for everyone to use.

Network Diagram

Here is an overview of the topology we used for testing our full BGP table. This can be done a number of different ways and you can use just about any combination of Hypervisors including VM Ware and VirtualBox which are the two downloads included in this post. In this setup, we are using a MikroTik x86 VM to peer into the Ubuntu VM that has copies of the global table. It established an EBGP peering over 10.254.253.0/24 and takes in a full table.

BGP-VM-Network-Diagram

Getting started 

First you need to download either the VM Ware or VirtualBox OVA files and import them into your hypervisor. The setup and installation of VM Ware ESXi or VirtualBox is beyond the scope of this post, so please google it if you need help.

Downloads

Download .OVA for VM Ware

Download .OVA for VirtualBox

Powering up the VM

Once you have successfully imported the VM, you will get a screen that looks like this:

BGP-VM-Startup-page

Credentials

Here are the credentials which you can change if needed.

username: bgpuser

password: bgpuser

sudo password: bgpuser

Bridging the VM NIC to your lab network

In order to have IP connectivity to another router (physical or virtual) you will need to setup the VM NIC to connect to the network you want to test on. There are a number of different ways to connect VMs into a virtual or physical network.

VM Ware – we connected the VM to the default VM management network (which is a physical server NIC) so it could reach other VMs and physical lab routers

ESXi-vswitch

 

VirtualBox – we bridged the VM to the NIC of the desktop we are running VirtualBox on so it could reach other VMs and physical lab routers

VirtualBox-network-bridged

BGP Feeds that are used in this VM

The BGP feeds that are available come from the RIPE RIS Raw Data page and were archived in January 2016. We included 6 different tables from 4 continents so you can have up to 6 unique BGP tables to use in your lab testing. See the next section for the syntax to use for one of these files.

RIPE RRCFile Name
rrc00.ripe.netISP1-Europe-Amsterdam-Jan-2016
rrc01.ripe.netISP2-Europe-London-Jan-2016
rrc06.ripe.netISP3-Asia-Tokyo-Jan-2016
rrc15.ripe.netISP4-SouthAmerica-SaoPaulo-Jan-2016
rrc11.ripe.netISP5-NorthAmerica-NewYork-Jan-2016
rrc14.ripe.netISP6-NorthAmerica-PaloAlto-Jan-2016

Important Note !!!! – Using this VM does not provide connectivity to the Internet and will likely cause an outage when connected to a production network with live BGP peerings. This VM is intended to simulate an upstream peering for testing and lab development.

Setting up a BGP peering – BGP VM

Once you have IP connectivity and can ping the router you want to peer with, you can set up a peering on the VM. Here is the command syntax – first change to the bgp directory and issue the command below (with edits for your IPs and AS numbers)

bgpuser@Full-BGP-Global-Table-VM:~$ cd bgp
bgpuser@Full-BGP-Global-Table-VM:~/bgp$ sudo ./bgp_simple.pl -myas 65000 -myip 10.254.253.112 -peerip 10.254.253.75 -peeras 65051 -p ISP1-Europe-Amsterdam-Jan-2016

Options for the BGP Peering (using the program bgp_simple ver 0.12)

bgpuser@Full-BGP-Global-Table-VM:~/bgp$ ./bgp_simple.pl

Please provide -myas, -myip, -peerip and -peeras!

bgp_simple.pl: Simple BGP peering and route injection script.
Version v0.12, 22-Jan-2011.

usage:
bgp_simple.pl:
                -myas           ASNUMBER        # (mandatory) our AS number
                -myip           IP address      # (mandatory) our IP address to source the sesion from
                -peerip         IP address      # (mandatory) peer IP address
                -peeras         ASNUMBER        # (mandatory) peer AS number
                [-holdtime]     Seconds         # (optional) BGP hold time duration in seconds (default 60s)
                [-keepalive]    Seconds         # (optional) BGP KeepAlive timer duration in seconds (default 20s)
                [-nolisten]                     # (optional) dont listen at $myip, tcp/179
                [-v]                            # (optional) provide verbose output to STDOUT, use twice to get debugs
                [-p file]                       # (optional) prefixes to advertise (bgpdump formatted)
                [-o file]                       # (optional) write all sent and received UPDATE messages to file
                [-m number]                     # (optional) maximum number of prefixes to advertise
                [-n IP address]                 # (optional) next hop self, overrides original value
                [-l number]                     # (optional) set default value for LOCAL_PREF
                [-dry]                          # (optional) dry run; dont build adjacency, but check prefix file (requires -p)
                [-f KEY=REGEX]                  # (optional) filter on input prefixes (requires -p), repeat for multiple filters
                                                        KEY is one of the following attributes (CaSE insensitive):

                                                        NEIG            originating neighbor
                                                        NLRI            NLRI/prefix(es)
                                                        ASPT            AS_PATH
                                                        ORIG            ORIGIN
                                                        NXHP            NEXT_HOP
                                                        LOCP            LOCAL_PREF
                                                        MED             MULTI_EXIT_DISC
                                                        COMM            COMMUNITY
                                                        ATOM            ATOMIC_AGGREGATE
                                                        AGG             AGGREGATOR

                                                        REGEX is a perl regular expression to be expected in a
                                                        match statement (m/REGEX/)

Without any prefix file to import, only an adjacency is established and the received NLRIs, including their attributes, are logged.

Setting up a BGP peering – Your peering router

We used a MikroTik x86 VM in ESXi for this test, but any brand of virtual or physical router that supports BGP can be used.

[[email protected]] > routing bgp export          
# jan/21/2016 10:34:43 by RouterOS 6.30.1
# software id = KC33-08AQ
#
/routing bgp instance
set default as=65001
/routing bgp peer
add hold-time=30m keepalive-time=4m15s name=BGP-VM remote-address=10.254.253.112 remote-as=65000 ttl=default

Sit back and watch hundreds of thousands of prefixes torture the CPU of your router

MikroTik-VM-500k-routes