VPLS is a pretty common technology in ISPs to either sell layer 2 services or backhaul traffic to a centralized aggregation point to conserve IPv4 space; check out more on that here.
How can I take advantage of segment routing but still deliver the same services? We already looked at how to setup the label switched paths utilizing IP Infusion as a segment routing mapping server in this interop post. Now let’s see how we can deliver a VPLS service over this with mikrotik as a provider edge router.
Delivering a service with a L2VPN
After setting up the IGP and label distribution between the PEs we will start building the L2VPN.
Why a BGP signaled VPLS session instead of LDP signaled VPLS?
In the segment-routing domain there is no LDP running. However, some vendors support static pseudowires or other methods to bring up a targeted LDP session for VPLS. I did some basic testing here couldn’t easily identify the right combination of knobs to make this work. Don’t worry; I’ll come back to it.
BGP signaled VPLS is a standards based technology that both vendors support.
First thing we need to do after having loopback reachability is to build the BGP sessions.
On OcNOS the route-target and route distinguisher is automatically derived from the vpls instance creation. While you explicitly set it on MikroTik.
ipi-1.lab.jan1.us.ipa.net# show mpls vpls TEST
Virtual Private LAN Service Instance: TEST, ID: 10
SIG-Protocol: BGP
Route-Distinguisher :65000:10
Route-Target :65000:10
VE-ID :10
The site-id (Tik) or VE-ID (OcNOS) are the vpls endpoint identifiers which uniquely identify each PE.
The final step before we can start to test is to setup the attachment circuit. On OcNOS we will have to setup a service template to pair with the vpls instance on attachment circuit. Then you can assign the vpls instance and the service template to a switchport. We will match vlan tag 10 on our service template. Anything coming in with dot1q 10 from the CCR2004 will enter the VPLS tunnel.
ipi-1.lab.jan1.us.ipa.net:
service-template TEST
match outer-vlan 10
!
interface xe2
switchport
mpls-vpls TEST service-template TEST
exit-if-vpls
!
On the MikroTik side we need to setup a bridge for the autocreation of the tunnels and assign appropriate interfaces to it. Again, we will match on dot1q 10.
Finally, we can do testing and verification. We want to look for the mesh peer which is the autodiscovered PE also in this vpls instance.
ipi-1.lab.jan1.us.ipa.net#show mpls vpls TEST
Virtual Private LAN Service Instance: TEST, ID: 10
SIG-Protocol: BGP
Route-Distinguisher :65000:10
Route-Target :65000:10
VE-ID :10
Attachment-Circuit :UP
Learning: Enabled
Control-Word: Disabled
Group ID: 0, Configured MTU: 1500
Description: none
service-tpid: dot1.q
Operating mode: Raw
Configured interfaces:
Interface: xe2
Service-template : TEST
Match criteria : 10
Mesh Peers:
100.127.2.0 (Up)
Then we want to make sure we are learning mac-addresses. As you can see the MACs not learned from the attached device are learned from the BGP peer at 100.127.2.0.
Next we can test reachability. I did have a strange issue where if the AC went down on the MikroTik-1 the mesh wouldn’t form on reattachment. This was resolvable with the following command on IPI-1: clear bgp 100.127.2.0 l2vpn vpls
One of the challenges service providers have faced in the last decade is lowering the cost per port or per MB while maintaining the same level of availability and service level.
And then add to that the constant pressure from subscribers to increase capacity and meet the rising demand for realtime content.
This can be an especially daunting task when routers with the feature sets ISPs need cost an absolute fortune – especially as new port speeds are released.
Whitebox, also called disaggregated networking, has started changing the rules of the game. ISPs are working to figure out how to integrate and move to production on disaggregated models to lower the cost of investing in higher speeds and feeds.
Whitebox often faces the perception problem of being more difficult to implement than traditional vendors – which is exactly why I wanted to highlight some of the work we’ve been doing at iparchitechs.com integrating whitebox into production ISP networks using IP Infusion’s OcNOS.
Things are really starting to heat up in the disaggregagted network space after the announcement by Amazon a few days ago that it intends to build and sell whitebox switches.
As I write this, I’m headed to Networking Field Day 18 where IP Infusion will be presenting and I expect whitebox will again be a hot topic.
This will be the second time IPI has presented at Networking Field Day but the first time that I’ve had a chance to see them present firsthand.
It’s especially exciting for me as I work on implementing IPI on a regular basis and integrating OcNOS into client networks.
What is OcNOS?
IP Infusion has been making network operating systems (NOS) for more than 20 years under the banner of its whitelabel NOS – ZebOS.
As disaggregated networking started to become popular, IPI created OcNOS which is an ONIE compatible NOS using elements and experience from 20 years of software development with ZebOS.
There is a great overview of OcNOS from Networking Field Day 15 here:
What does a production OcNOS based MPLS network look like?
Here is an overview of the EVE-NG lab we built based on an actual implementation.
Use case – Building an MPLS core to deliver L2 overlay services
Although certainly not a new use case or implementation, MPLS and VPLS are very expensive to deploy using major vendors and are still a fundamental requirement for most ISPs.
This is where IPI really shines as they have feature sets like MPLS FRR, TE and the newer Segment Routing for OSPF and IS-IS that can be used in a platform that is significantly cheaper than incumbent network vendors.
The cost difference is so large that often ISPs are able to buy switches with a higher overall port speeds than they could from a major vendor. This in turn creates a significant competitive advantage as ISPs can take the same budget (or less) and roll out 100 gig instead of 10 gig – as an example
Unlike enterprise networks, cost is more consistently a significant driver when selecting network equipment for ISPs. This is especially true for startup ISPs that may be limited in the amount of capital that can be spent in a service area to keep ROI numbers relatively sane for investors.
Lab Overview
In the lab (and production) network we have above, OcNOS is deployed as the MPLS core at each data center and MikroTik routers are being used as MPLS PE routers.
VPLS is being run from one DC to the other and delivered via the PE routers to the end hosts.
Because the port density on whitebox switches is so high compared to a traditional aggregation router, we could even use LACP channels if dark fiber was available to increase the transport bandwidth between the DCs without a significant monetary impact on the cost of the deployment.
The type of switches that you’d use in production depend greatly on the speeds and feeds required, but for startup ISPs, we’ve had lots of success with Dell 4048s and Edge-Core 5812.
How hard is it to configure and deploy?
It’s not hard at all!
If you know how to use the up and down arrow keys in the bootloader and TFTP/FTP to load an image onto a piece of network hardware, you’re halfway there!
Here is a screenshot of the GRUB bootloader for an ONIE switch (this is a Dell) where you select which OS to boot the switch into
The configuration is relatively straightforward as well if you’re familiar with industry standard Command Line Interfaces (CLI).
While this lab was configured in a more traditional way using a terminal session to paste commands in, OcNOS can easily be orchestrated and automated using tools like Ansible (also presenting at Networking Field Day 18) or protocols like NETCONF as well as a REST API.
Lab configs
I’ve included the configs from the lab in order to give engineers a better idea of what OcNOS actually looks like for a production deployment.
IPI-MPLS-1
!
!Last configuration change at 12:24:27 EDT Tue Jul 17 2018 by ocnos
!
no service password-encryption
!
hostname IPI-MPLS-1
!
logging monitor 7
!
ip vrf management
!
mpls lsp-model uniform
mpls propagate-ttl
!
ip domain-lookup
spanning-tree mode provider-rstp
data-center-bridging enable
feature telnet
feature ssh
snmp-server enable snmp
snmp-server view all .1 included
ntp enable
username ocnos role network-admin password encrypted $1$HJDzvHS1$.4/PPuAmCUEwEhs
UWeYqo0
!
ip pim register-rp-reachability
!
router ldp
router-id 100.127.0.1
!
interface lo
mtu 65536
ip address 127.0.0.1/8
ip address 100.127.0.1/32 secondary
ipv6 address ::1/128
!
interface eth0
ip address 100.64.0.1/29
label-switching
enable-ldp ipv4
!
interface eth1
ip address 100.64.0.9/29
label-switching
enable-ldp ipv4
!
interface eth2
ip address 100.64.1.1/29
label-switching
enable-ldp ipv4
!
interface eth3
!
interface eth4
!
interface eth5
!
interface eth6
!
interface eth7
!
router ospf 1
ospf router-id 100.127.0.1
network 100.64.0.0/29 area 0.0.0.0
network 100.64.0.8/29 area 0.0.0.0
network 100.64.1.0/29 area 0.0.0.0
network 100.127.0.1/32 area 0.0.0.0
cspf disable-better-protection
!
bgp extended-asn-cap
!
router bgp 8675309
bgp router-id 100.127.0.1
neighbor 100.127.0.3 remote-as 8675309
neighbor 100.127.0.3 update-source lo
neighbor 100.127.2.1 remote-as 8675309
neighbor 100.127.2.1 update-source lo
neighbor 100.127.2.1 route-reflector-client
neighbor 100.127.0.4 remote-as 8675309
neighbor 100.127.0.4 update-source lo
neighbor 100.127.0.4 route-reflector-client
neighbor 100.127.0.2 remote-as 8675309
neighbor 100.127.0.2 update-source lo
neighbor 100.127.0.2 route-reflector-client
neighbor 100.127.1.1 remote-as 8675309
neighbor 100.127.1.1 update-source lo
neighbor 100.127.1.1 route-reflector-client
!
line con 0
login
line vty 0 39
login
!
end
IPI-MPLS-2
!
!Last configuration change at 12:23:31 EDT Tue Jul 17 2018 by ocnos
!
no service password-encryption
!
hostname IPI-MPLS-2
!
logging monitor 7
!
ip vrf management
!
mpls lsp-model uniform
mpls propagate-ttl
!
ip domain-lookup
spanning-tree mode provider-rstp
data-center-bridging enable
feature telnet
feature ssh
snmp-server enable snmp
snmp-server view all .1 included
ntp enable
username ocnos role network-admin password encrypted $1$RWk6XAN.$6H0GXBR9ad8eJE2
7nRUfu1
!
ip pim register-rp-reachability
!
router ldp
router-id 100.127.0.2
!
interface lo
mtu 65536
ip address 127.0.0.1/8
ip address 100.127.0.2/32 secondary
ipv6 address ::1/128
!
interface eth0
ip address 100.64.0.2/29
label-switching
enable-ldp ipv4
!
interface eth1
ip address 100.64.0.17/29
label-switching
enable-ldp ipv4
!
interface eth2
ip address 100.64.1.9/29
label-switching
enable-ldp ipv4
!
interface eth3
!
interface eth4
!
interface eth5
!
interface eth6
!
interface eth7
!
router ospf 1
network 100.64.0.0/29 area 0.0.0.0
network 100.64.0.16/29 area 0.0.0.0
network 100.64.1.8/29 area 0.0.0.0
network 100.127.0.2/32 area 0.0.0.0
cspf disable-better-protection
!
bgp extended-asn-cap
!
router bgp 8675309
bgp router-id 100.127.0.2
neighbor 100.127.0.3 remote-as 8675309
neighbor 100.127.0.3 update-source lo
neighbor 100.127.0.1 remote-as 8675309
neighbor 100.127.0.1 update-source lo
!
line con 0
login
line vty 0 39
login
!
end
IPI-MPLS-3
!
!Last configuration change at 12:25:11 EDT Tue Jul 17 2018 by ocnos
!
no service password-encryption
!
hostname IPI-MPLS-3
!
logging monitor 7
!
ip vrf management
!
mpls lsp-model uniform
mpls propagate-ttl
!
ip domain-lookup
spanning-tree mode provider-rstp
data-center-bridging enable
feature telnet
feature ssh
snmp-server enable snmp
snmp-server view all .1 included
ntp enable
username ocnos role network-admin password encrypted $1$gc9xYbW/$JlCDmgAEzcCmz77
QwmJW/1
!
ip pim register-rp-reachability
!
router ldp
router-id 100.127.0.3
!
interface lo
mtu 65536
ip address 127.0.0.1/8
ip address 100.127.0.3/32 secondary
ipv6 address ::1/128
!
interface eth0
ip address 100.64.0.25/29
label-switching
enable-ldp ipv4
!
interface eth1
ip address 100.64.0.10/29
label-switching
enable-ldp ipv4
!
interface eth2
ip address 100.64.2.1/29
label-switching
enable-ldp ipv4
!
interface eth3
!
interface eth4
!
interface eth5
!
interface eth6
!
interface eth7
!
router ospf 1
ospf router-id 100.127.0.3
network 100.64.0.8/29 area 0.0.0.0
network 100.64.0.24/29 area 0.0.0.0
network 100.64.2.0/29 area 0.0.0.0
network 100.127.0.3/32 area 0.0.0.0
cspf disable-better-protection
!
bgp extended-asn-cap
!
router bgp 8675309
bgp router-id 100.127.0.3
neighbor 100.127.0.1 remote-as 8675309
neighbor 100.127.0.1 update-source lo
neighbor 100.127.2.1 remote-as 8675309
neighbor 100.127.2.1 update-source lo
neighbor 100.127.2.1 route-reflector-client
neighbor 100.127.0.4 remote-as 8675309
neighbor 100.127.0.4 update-source lo
neighbor 100.127.0.4 route-reflector-client
neighbor 100.127.0.2 remote-as 8675309
neighbor 100.127.0.2 update-source lo
neighbor 100.127.0.2 route-reflector-client
neighbor 100.127.1.1 remote-as 8675309
neighbor 100.127.1.1 update-source lo
neighbor 100.127.1.1 route-reflector-client
!
line con 0
login
line vty 0 39
login
!
end
IPI-MPLS-4
!
!Last configuration change at 12:24:49 EDT Tue Jul 17 2018 by ocnos
!
no service password-encryption
!
hostname IPI-MPLS-4
!
logging monitor 7
!
ip vrf management
!
mpls lsp-model uniform
mpls propagate-ttl
!
ip domain-lookup
spanning-tree mode provider-rstp
data-center-bridging enable
feature telnet
feature ssh
snmp-server enable snmp
snmp-server view all .1 included
ntp enable
username ocnos role network-admin password encrypted $1$6OP7UdH/$RaIxCBOGxHIt1Ao
IUyPks/
!
ip pim register-rp-reachability
!
router ldp
router-id 100.127.0.4
!
interface lo
mtu 65536
ip address 127.0.0.1/8
ip address 100.127.0.4/32 secondary
ipv6 address ::1/128
!
interface eth0
ip address 100.64.0.26/29
label-switching
enable-ldp ipv4
!
interface eth1
ip address 100.64.0.18/29
label-switching
enable-ldp ipv4
!
interface eth2
ip address 100.64.2.9/29
label-switching
enable-ldp ipv4
!
interface eth3
!
interface eth4
!
interface eth5
!
interface eth6
!
interface eth7
!
router ospf 1
ospf router-id 100.127.0.4
network 100.64.0.16/29 area 0.0.0.0
network 100.64.0.24/29 area 0.0.0.0
network 100.64.2.8/29 area 0.0.0.0
network 100.127.0.4/32 area 0.0.0.0
cspf disable-better-protection
!
bgp extended-asn-cap
!
router bgp 8675309
bgp router-id 100.127.0.4
neighbor 100.127.0.3 remote-as 8675309
neighbor 100.127.0.3 update-source lo
neighbor 100.127.0.1 remote-as 8675309
neighbor 100.127.0.1 update-source lo
!
line con 0
login
line vty 0 39
login
!
end
