Cisco to MikroTik – MPLS

About the Cisco to MikroTik series

One of the hardest things to do quickly in network engineering, is learn a new syntax for a NOS. Especially if you have a tight deadline and need to stand up equipment you’ve never worked with before. The command structure for RouterOS can be cumbersome if you are used to the Cisco CLI.

If you’ve been in networking for a while, you probably started with learning the Cisco CLI. Therefore, it is helpful to compare the commands if you want to implement a network with a MikroTik and Cisco routers.

This is the third post in a series that creates a Rosetta stone between IOS and RouterOS. We plan to tackle  other command comparisons like VLANs, QoS and basic operations to make it easier for network engineers trained in Cisco IOS to successfully implement Mikrotik / RouterOS devices.

Click here for the first article in this series – “Cisco to MikroTik BGP command translation”
Click here for the second article in this series – “Cisco to MikroTik OSPF command translation”

While many commands have almost the exact same information, others are as close as possible. Since there isn’t always an exact match, sometimes you may have to run two or three commands to get the information needed.

Using  EVE-NG for testing

In the last article, we began using EVE-NG instead of GNS3 to emulate both Cisco IOS and RouterOS so we could compare the different commands and ensure the translation was as close as possible. Don’t get me wrong, I like GNS3, but the web interface of EVE-NG makes it really easy to keep all the horsepower for complex labs at a central location and then VPN in to work on labs as needed.

Network for Basic mpls commands

MPLS-Cisco-to-MikroTik
Cisco commandMikroTik Command
show mpls ldp neighbormpls ldp neighbor print
show mpls interfacesmpls ldp interface print
show mpls forwarding-tablempls forwarding-table print
show mpls bindingmpls remote-bindings print
sh mpls ip binding localmpls local-bindings print
sh mpls label rangempls print
sh mpls ldp parametersmpls ldp print
interface GigabitEthernet0/1
mpls ip
/mpls ldp interface
add interface=ether1
mpls ldp router-id Loopback0/mpls ldp
set enabled=yes lsr-id=10.1.1.3

Examples of the MikroTik RouterOS commands from the table above


[admin@MikroTik] > mpls ldp neighbor print

This command will show LDP neighbors and detail on whether they are Dynamic, Targeted, Operational or using VPLS

mpls-ldp-neighbor-print

[admin@MikroTik] > mpls ldp interface print

This command will list the interfaces that LDP is enabled on

mpls-ldp-interface-print

[admin@MikroTik] > mpls forwarding-table print

Use this command to display the MPLS forwarding table which shows what labels are assigned, the interface used and the next hop.

mpls-forwarding-table-print

[admin@MikroTik] > mpls remote-bindings print

This is a quick way to show remote bindings which displays the labels desired and used by the next hop routers for each prefix.

mpls-remote-bindings-print

[admin@MikroTik] > mpls local-bindings print

This is a quick way to show local bindings which displays the labels desired and used by the local router – in this case R3.

mpls-local-bindings-print

[admin@MikroTik] > mpls print

This is a quick way to show basic mpls settings for RouterOS which includes the label range and whether or not to propagate TTL which affects what a traceroute looks like over an MPLS network.

mpls-print

[admin@MikroTik] > mpls ldp print

This is a quick way to show mpls ldp settings for Router-OS including whether or not LDP is enabled.

mpls-ldp-print

Configurations

R1

Cisco-R1#sh run
Building configuration...

Current configuration : 3062 bytes
!
version 15.5
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname Cisco-R1
!
boot-start-marker
boot-end-marker
!
!
!
no aaa new-model
ethernet lmi ce
!
!
!
mmi polling-interval 60
no mmi auto-configure
no mmi pvc
mmi snmp-timeout 180
!
!
!
!
!
!
!
!
!
!
!
ip cef
no ipv6 cef
!
multilink bundle-name authenticated
!
!
!
!
!
redundancy
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
interface Loopback0
 ip address 10.1.1.1 255.255.255.255
!
interface GigabitEthernet0/0
 ip address 203.0.113.1 255.255.255.248
 duplex auto
 speed auto
 media-type rj45
 mpls ip
!
interface GigabitEthernet0/1
 no ip address
 shutdown
 duplex auto
 speed auto
 media-type rj45
!
interface GigabitEthernet0/2
 no ip address
 shutdown
 duplex auto
 speed auto
 media-type rj45
!
interface GigabitEthernet0/3
 no ip address
 shutdown
 duplex auto
 speed auto
 media-type rj45
!
router ospf 1
 network 10.1.1.1 0.0.0.0 area 0
 network 203.0.113.0 0.0.0.7 area 0
!
ip forward-protocol nd
!
!
no ip http server
no ip http secure-server
!
!
!
mpls ldp router-id Loopback0
!
control-plane
!
banner exec ^C
**************************************************************************
* IOSv is strictly limited to use for evaluation, demonstration and IOS  *
* education. IOSv is provided as-is and is not supported by Cisco's      *
* Technical Advisory Center. Any use or disclosure, in whole or in part, *
* of the IOSv Software or Documentation to any third party for any       *
* purposes is expressly prohibited except as otherwise authorized by     *
* Cisco in writing.                                                      *
**************************************************************************^C
banner incoming ^C
**************************************************************************
* IOSv is strictly limited to use for evaluation, demonstration and IOS  *
* education. IOSv is provided as-is and is not supported by Cisco's      *
* Technical Advisory Center. Any use or disclosure, in whole or in part, *
* of the IOSv Software or Documentation to any third party for any       *
* purposes is expressly prohibited except as otherwise authorized by     *
* Cisco in writing.                                                      *
**************************************************************************^C
banner login ^C
**************************************************************************
* IOSv is strictly limited to use for evaluation, demonstration and IOS  *
* education. IOSv is provided as-is and is not supported by Cisco's      *
* Technical Advisory Center. Any use or disclosure, in whole or in part, *
* of the IOSv Software or Documentation to any third party for any       *
* purposes is expressly prohibited except as otherwise authorized by     *
* Cisco in writing.                                                      *
**************************************************************************^C
!
line con 0
line aux 0
line vty 0 4
 login
 transport input none
!
no scheduler allocate
!
end

R2

Cisco-R2#sh run
Building configuration...

Current configuration : 3122 bytes
!
version 15.5
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname Cisco-R2
!
boot-start-marker
boot-end-marker
!
!
!
no aaa new-model
ethernet lmi ce
!
!
!
mmi polling-interval 60
no mmi auto-configure
no mmi pvc
mmi snmp-timeout 180
!
!
!
!
!
!
!
!
!
!
!
ip cef
no ipv6 cef
!
multilink bundle-name authenticated
!
!
!
!
!
redundancy
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
interface Loopback0
 ip address 10.1.1.2 255.255.255.255
!
interface GigabitEthernet0/0
 ip address 203.0.113.2 255.255.255.248
 duplex auto
 speed auto
 media-type rj45
 mpls ip
!
interface GigabitEthernet0/1
 ip address 203.0.113.9 255.255.255.248
 duplex auto
 speed auto
 media-type rj45
 mpls ip
!
interface GigabitEthernet0/2
 no ip address
 shutdown
 duplex auto
 speed auto
 media-type rj45
!
interface GigabitEthernet0/3
 no ip address
 shutdown
 duplex auto
 speed auto
 media-type rj45
!
router ospf 1
 network 10.1.1.2 0.0.0.0 area 0
 network 203.0.113.0 0.0.0.7 area 0
 network 203.0.113.8 0.0.0.7 area 0
!
ip forward-protocol nd
!
!
no ip http server
no ip http secure-server
!
!
!
mpls ldp router-id Loopback0
!
control-plane
!
banner exec ^C
**************************************************************************
* IOSv is strictly limited to use for evaluation, demonstration and IOS  *
* education. IOSv is provided as-is and is not supported by Cisco's      *
* Technical Advisory Center. Any use or disclosure, in whole or in part, *
* of the IOSv Software or Documentation to any third party for any       *
* purposes is expressly prohibited except as otherwise authorized by     *
* Cisco in writing.                                                      *
**************************************************************************^C
banner incoming ^C
**************************************************************************
* IOSv is strictly limited to use for evaluation, demonstration and IOS  *
* education. IOSv is provided as-is and is not supported by Cisco's      *
* Technical Advisory Center. Any use or disclosure, in whole or in part, *
* of the IOSv Software or Documentation to any third party for any       *
* purposes is expressly prohibited except as otherwise authorized by     *
* Cisco in writing.                                                      *
**************************************************************************^C
banner login ^C
**************************************************************************
* IOSv is strictly limited to use for evaluation, demonstration and IOS  *
* education. IOSv is provided as-is and is not supported by Cisco's      *
* Technical Advisory Center. Any use or disclosure, in whole or in part, *
* of the IOSv Software or Documentation to any third party for any       *
* purposes is expressly prohibited except as otherwise authorized by     *
* Cisco in writing.                                                      *
**************************************************************************^C
!
line con 0
line aux 0
line vty 0 4
 login
 transport input none
!
no scheduler allocate
!
end

R3

[admin@MikroTik-R3] > export
# may/03/2018 16:34:51 by RouterOS 6.38.7
# software id =
#
/interface bridge
add name=Loopback0
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip address
add address=203.0.113.10/29 interface=ether1 network=203.0.113.8
add address=10.1.1.3 interface=Loopback0 network=10.1.1.3
add address=203.0.113.17/29 interface=ether2 network=203.0.113.16
/ip dhcp-client
add disabled=no interface=ether1
/mpls ldp
set enabled=yes lsr-id=10.1.1.3
/mpls ldp interface
add interface=ether1
add interface=ether2
/routing ospf network
add area=backbone network=203.0.113.8/29
add area=backbone network=10.1.1.3/32
add area=backbone network=203.0.113.16/29
/system identity
set name=MikroTik-R3

R4

[admin@MikroTik-R4] > export
# may/03/2018 16:35:28 by RouterOS 6.38.7
# software id =
#
/interface bridge
add name=Loopback0
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip address
add address=203.0.113.18/29 interface=ether1 network=203.0.113.16
add address=10.1.1.4 interface=Loopback0 network=10.1.1.4
/ip dhcp-client
add disabled=no interface=ether1
/mpls ldp
set enabled=yes lsr-id=10.1.1.4
/mpls ldp interface
add interface=ether1
/routing ospf network
add area=backbone network=203.0.113.16/29
add area=backbone network=10.1.1.4/32
/system identity
set name=MikroTik-R4

WISP/FISP Design – Building your future MPLS network with whitebox switching.

 

MPLS-Whitebox-drawings

The role of whitebox in a WISP/FISP MPLS core

Whitebox, if you aren’t familiar with it, is the idea of separating the network operating system and switching hardware into commodity elements that can be purchased separately. There was a good overview on whitebox in this StubArea51.net article a while back if you’re looking for some background.

Lately, in my work for IP ArchiTechs, I’ve had a number of clients interested in deploying IP Infusion with either Dell, Agema or Edge Core switches to build an MPLS core architecture in lieu of an L2 ring deployment via ERPs. Add to that a production deployment of Cumulus Linux and Edge Core that I’ve been working on building out and it’s been a great year for whitebox.

There are a number of articles written that extoll the virtues of whitebox for web scale companies, large service providers and big enterprises. However, not much has been written on how whitebox can help smaller Tier 2 and 3 ISPs – especially Wireless ISPs (WISPs) and Fiber ISPs (FISPs).

And the line between those types of ISPs gets more blurry by the day as WISPs are heavily getting into fiber and FISPs are getting into last mile RF. Some of the most successful ISPs I consult for tend to be a bit of a hybrid between WISP and FISP.

The goal of any ISP stakeholder whether large or small should be getting the lowest cost per port for any network platform (while maintaining the same level of service – or even better) so that service offerings can be improved or expanded without being required to pass the financial burden down to the end subscriber.

Whitebox is well positioned to aid ISPs in attaining that goal.

Whitebox vs. Traditional Vendor

Whitebox is rapidly gaining traction and working towards becoming the new status quo in networking. The days of proprietary hardware as the dominant force are numbered. Correspondingly, the extremely high R&D/manufacturing cost that is passed along to customers also seems to be in jeopardy for mainstream vendors like Cisco and Juniper.

Here are a few of the advantages that whitebox has for Tier2 and 3 ISPs:

  • Cost – it is not uncommon to find 48 ports of 10 gig and 4 ports of 40 gig on a new whitebox switch with licensing for under $10k. Comparable deployments in Cisco, Juniper, Brocade, etc typically exceed that number by a factor of 3 or more.
  • SDN and NFV – Open standards and development are at the heart of the SDN and NFV movement, so it’s no surprise that whitebox vendors are knee deep in SDN and NFV solutions. Because whitebox operating systems are modular, less cluttered and have built in hardware abstraction, SDN and NFV actually become much easier to implement.
  • No graymarket penalty – Because the operating system and hardware are separate, there isn’t an issue with obtaining hardware from the graymarket and then going to get a license with support. While the cost of the hardware brand new is still incredibly affordable, some ISPs leverage the graymarket to expand when faced with limited financial resources.
  • Stability – whitebox operating systems tend to implement open standards protocols and stick to mainstream use cases. The lack of proprietary corner case features allows the development teams for a whitebox NOS to be more thorough about testing for stability, interop and fixing bugs.
  • Focus on software – One of the benefits that comes from separating hardware and software for network equipment is a singular focus on software development instead of having to jump though hoops to support hundreds of platforms that sometimes have a very short product lifecycle. This is probably the single greatest challenge traditional vendors face in producing high quality software.
  • ISSU – Often touted as a competitive advantage by the likes of Cisco and others, In Service Software Upgrade (ISSU ) is now supported by some whitebox NOS vendors.

1466540435IpInfusion interivew questions

IP Infusion

IP Infusion (IPI) first got on my radar about 2 years ago when I was working through a POC for Cumulus Linux and just getting my feet wet in understanding the world of whitebox.  What struck me as unique about them is that IP Infusion has been writing code for protocol stacks  and modular network operating systems (ZebOS) for the last 20 years – essentially making them a seasoned veteran in turning out stable code for a NOS. As the commodity hardware movement started gathering steam, IP Infusion took all of the knowledge and experience from ZebOS and created OcNOS, which is a platform that is compatible with ONIE switches.

Earlier this year, I attended Networking Field Day 14 (NFD14) as a delegate and was pleasantly surprised to learn that IP Infusion presented at Networking Field Day 15 (NFD15) back in April. I highly recommend watching all of the NFD15 videos on IP Infusion, as you’d be hard pressed to find a better technical deep dive on IPI anywhere else. Some of the technical and background content here is taken from the video sessions at NFD15.

Background

  • Has its roots in GNU Zebra routing engine
  • Strong adherence to standards-based protocol implementations
  • Original white label NOS ZebOS has been around for 20+ years and is used by companies like F5, Fortinet and Citrix

Advantages

  • Very service provider focused with advanced feature sets for BGP/MPLS
  • OcNOS benefits from 20 years of white label NOS development and according to IP Infusion’s marketing material is reputed to have “six 9’s” of stability as observed by their larger ISP customers.
  • Perpetual licensing – once the license is purchased, the only recurring cost is the annual maintenance which is a much smaller fee (typically around 15% of the license)
  • Extensive API support – IPI has extensive API support for protocols like BGP to facilitate integration of automation and orchestration.
  • Easier hardware abstractions than proprietary NOS – look for chassis based whitebox and form factors beyond 1U in the future
  • Increased focus on the 1 Gbps switch market with Broadcom’s incredibly feature rich Qumran chipset so that Start-up and very small ISPs can still leverage the benefits of whitebox. Also, larger Tier 2 and 3 ISPs will have a switching solution for edge, aggregation and customer CPE needs.

Integrating OcNOS with MikroTik/Ubiquiti

I’ve specifically listed IP Infusion instead of doing a more in depth comparison of all the various whitebox operating systems, because IP Infusion is really positioned to be the best choice for Tier 2 and 3 ISPs due to the available feature set and modular approach to building protocol support. Going a step further, it’s a natural fit for ISPs that are running MikroTik or Ubiquiti as the OcNOS operating system fills in many of the gaps in protocol support (MPLS TE and FRR especially) that are needed when building an MPLS core for a rapidly expanding ISP.

While I’ve successfully built MPLS into many ISPs with MikroTik and Ubiquiti and continue to do so, there is a scaling limit that most ISPs eventually hit and need to start using ASIC based hardware with the ability to design comprehensive traffic engineering policies.

The good news is that MikroTik and Ubiquiti still have a role to play when building a whitebox core. Both work very well as MPLS PE routers that can be attached to the IP Infusion MPLS core. Last mile services can then be delivered in a very cost effective way leveraging technologies like VPLS or L3VPN.

Other Whitebox NOS offerings

There are a number of other whitebox network operating systems to choose from. Although the focus has been on IPI due to the feature set, Cumulus Linux and Big Switch are both great options  if you have a need to deploy data center services.

Cumulus Linux is also rapidly working on developing and putting MPLS and more advanced routing protocol support into the operating system and it wouldn’t surprise me if they become more of a contender in the ISP arena in the next few years.

This actually touches on one of the other great benefits of whitebox. You can stock a common switch and put the operating system on that best fits the use case.

For example, the Dell S4048-ON switch (48x10gig,4x40gig) can be used for IPI, Cumulus Linux and Big Switch depending on the feature set required.

Some ISPs are getting into or already run cloud and colocation services in their data centers. If a compatible whitebox switch is used then stocking replacement hardware and operational maintenance of the ISP and Data Center portions of the network become far simpler.

Design elements of a WISP/FISP based on a whitebox MPLS core

Here are some examples of the most common elements we are trending towards as we start building WISPs and FISPs on a whitebox foundation coupled with other common low cost vendors like MikroTik and Ubiquiti.

MPLS-Whitebox-core-2

Whitebox MPLS Core

As ISPs grow, the core tends to move from pure routers to Layer 3 switches to be able to better support higher speeds and take advantage of technologies like dark fiber and DWDM/CWDM to increase speeds. Many smaller ISPs are starting to compete using the “Google Fiber” model of selling 1Gbps synchronous to residential customers and need the extra capacity to handle that traffic.

MPLS support on ASICs has traditionally been extremely expensive with costs soaring as the port speeds increase from 1 gig to 10 gig and 40 gig. And yet MPLS is a fundamental requirement for the multi-tenancy needs of an ISP.

Leveraging whitebox hardware allows for MPLS switching in hardware at 10, 40 and 100 gig speeds for a fraction of the cost of vendors like Cisco and Juniper.

This allows ISPs to utilize dark fiber, wave and 10Gig+ layer 2 services in more cost effective way to increase the overall capacity of the core.

MPLS-PE-MikroTIk

MPLS PE for Aggregation

MikroTik and Ubiquiti both have hardware with economical MPLS feature sets that work well as an MPLS PE. Having said that, I give MikroTik the edge here as Ubiquiti has only recently implemented MPLS and is still working on expanding the feature set.

MikroTik in contrast has had MPLS in play for a long time and is a very solid choice when aggregation and PE services are needed. The CCR series in particular has been very popular and stable as a PE router.

Virtual BGP edge

Virtual BGP Edge

MikroTik has made great strides in the high performance virtual market with the introduction of the Cloud Hosted Router (CHR) a little over a year ago.

Due to the current limitation of the MikroTik kernel to only using one processor for BGP, there has been a trend towards using x86 hardware with much higher clock speed per core than the CCR series to handle the requirement of a full BGP table.

The CHR is able to process changes in the BGP table much faster as a result and doesn’t suffer from the slow convergence speeds that can happen on CCRs with a large number of full tables.

Couple that with license costs that max out at $200 USD for unlimited speeds and the CHR becomes incredibly attractive as the choice for an edge BGP router.

NFV-Platform

NFV platform

Network Function Virtualization (NFV) has been getting a lot of press lately as more and more ISPs are turning to hypervisors to spin up resources that would traditionally be handled in purpose built hardware. NFV allows for more generic hardware deployments of hypervisors and switches so that more specific network functions can be handed virtually.

Some examples are:

  • BGP Edge routers (smiliar to the previous BGP CHR use case)
  • BRAS for PPPoE
  • QoE engines
  • EPC for LTE deployments
  • Security devices like IPS/IDS and WAF
  • MPLS PE routers

There are many ways to leverage x86 horsepower to get NFV into a WISP or FISP. One platform in particular that is gaining attention is Baltic Networks’ Vengeance router which runs VMWARE ESXi and can be used in a number of different NFV deployments.

We have been testing a Vengeance router in the StubArea51.net lab for several months and have seen very positive results. We will be doing a more in depth hardware review on that platform as a separate article in the future.

Closing thoughts

Whitebox is poised for rapid growth in the network world, as the climate is finally becoming favorable – even in larger companies – to use commodity hardware and not be entirely dependent on incumbent network vendors. This is already opening up a number of options for economical growth of ISPs in a platform that appears to be surpassing the larger vendors in reliability due to a more concentrated focus on software.

Commodity networking is here to stay and I look forward to the vast array of problems that it can solve as we build out the next generation of WISP and FISP networks.