MikroTik – RouterOSv7 first look – MLAG on CRS 3xx switches

What is MLAG?

Multi-Chassis Link Aggregation Group or MLAG is an idea that’s been around for a while.

It allows for the ability to form LACP channels across multiple physical switches.

Wikipedia shows a few different topology examples here


Vendor implementations are proprietary but the idea of MLAG was first mentioned in 802.1AX-2008 in 2008.

It first started to become popular in data center networking in the late 2000s

What makes the addition of MLAG to MikroTik’s RouterOS feature set notable is that it lowers the barrier to entry for this particular feature.

CRS 3xx switches are very inexpensive (starting at $149 USD) and may very well be the lowest cost MLAG capable hardware available on the market.

Contact Us | IP ArchiTechs | Network Consulting Firm


Introduced in 7.1beta6

MLAG has been asked for by the MikroTik community a number of times and the most active feature request thread started here in 2020:

new feature request MLAG!!! – MikroTik

MikroTik added several version 7 beta releases in 2021 and included MLAG for all CRS 3xx series switches in 7.1beta6 on May 18th, 2021.

Overview of protocol requirements

MLAG is fairly consistent across vendors with the need for a link between physical devices that manages the MLAG groups. In MikroTik, these are called peer ports which facilitate the ICCP.

Here are a few terms for MikroTik MLAG:

ICCP (Inter Chassis Control Protocol). – Responsible for determining active/secondary switches and maintaining and updating the bridge table between physical switches.

Peer port – An interface that will be used as a peer port. Both peer devices use inter-chassis communication over the peer ports to establish MLAG and update the host table. The Peer port should be isolated on a different untagged VLAN using a pvid setting. The Peer port can be configured as a bonding interface.

System-id – The lowest MAC address between both peer bridges will be used as the system-id. This system-id is used for (R)STP bridge identifier.

Active-role – The peer with the lowest bridge MAC address will be acting as a primary device. The primary device is responsible for sending the correct LACP system ID on all MLAG ports.

mlag-id – An integer from 0 to 4294967295, it is used to set the MLAG ID for bonding interfaces. The same MLAG ID should be used on both peer devices to successfully create a single MLAG.

MikroTik’s requirements for ICCP and MLAG are:

  • RouterOS ICCP does not require an IP configuration
  • It should be isolated from the rest of the network using a dedicated untagged VLAN
  • Peer ports can also be configured as LACP bonding interfaces
  • MLAG requires enabled STP or RSTP protocol.


In order to present a single MAC address for the L2 spanning tree topology, ICCP functions on top of the peer ports to manage the MLAG/LACP system-id.

The system-id is used as the MAC address presented to the LACP client for RSTP/MSTP bridge identification.

reference for images and MLAG definitions: Multi-chassis Link Aggregation Group – RouterOS – MikroTik Documentation


Lab Example

In order to test the new MLAG functionality, we decided to setup a lab with CRS326-24S+2Q switches and CCR2004-1G-12S+2XS routers.

Below is the lab physical and logical topology.

Configuring an MLAG Group

Configure Bond and MLAG ID on CSW-01

/interface bonding
add mlag-id=100 mode=802.3ad name=Po1 slaves=sfp-sfpplus1


Configure Bond and MLAG ID on CSW-02

/interface bonding
add mlag-id=100 mode=802.3ad name=Po1 slaves=sfp-sfpplus1


* Apply each configuration step below on both switches to complete mlag setup and mlag-id 100. *


Configure the bridge and enable VLAN filtering. Add MLAG bonded interfaces and peer port to the bridge.

/interface bridge
add name=Bridge-MLAG vlan-filtering=yes
/interface bridge port
add bridge=Bridge-MLAG interface=Po1
add bridge=Bridge-MLAG interface=qsfpplus1-1 pvid=777


Configure a VLAN to be used over the MLAG

/interface bridge vlan
add bridge=Bridge-MLAG tagged=Po1 vlan-ids=3000


Set the peer port

/interface bridge mlag
set bridge=Bridge-MLAG peer-port=qsfpplus1-1


Validate the MLAG group


Show the status of the MLAG group, active and secondary ports and verify the system-id the client LACP receives

######## MLAG Switches - 2 x CRS326 ############

[[email protected]] > interface/bridge/mlag/monitor 
       status: connected
    system-id: 48:8F:5A:3A:44:BA
  active-role: primary

[[email protected]] > interface/bridge/mlag/monitor                        
       status: connected
    system-id: 48:8F:5A:3A:44:BA
  active-role: secondary

######## NON-MLAG LACP Router ############

[[email protected]] > /interface bonding monitor Po1
                    mode: 802.3ad
            active-ports: sfp-sfpplus3,sfp-sfpplus4
          inactive-ports: 
          lacp-system-id: 48:8F:5A:00:4F:80
    lacp-system-priority: 65535
  lacp-partner-system-id: 48:8F:5A:3A:44:BA


Configurations

RTR-01

/interface bridge
add name=Lo0
/interface bonding
add mode=802.3ad name=Po1 slaves=sfp-sfpplus3,sfp-sfpplus4 transmit-hash-policy=layer-2-and-3
/interface vlan
add interface=Po1 name=vlan3000 vlan-id=3000
/routing table
add fib name=""
/ip address
add address=100.126.0.1/29 interface=vlan3000 network=100.126.0.0
add address=100.127.0.1 interface=Lo0 network=100.127.0.1
/ipv6 address
add address=200:100:126::1 interface=vlan3000
add address=200:100:127::1/128 advertise=no interface=Lo0
/system identity
set name=RTR-01

RTR-02

/interface bridge
add name=Lo0
/interface bonding
add mode=802.3ad name=Po1 slaves=sfp-sfpplus3,sfp-sfpplus4 transmit-hash-policy=layer-2-and-3
/interface vlan
add interface=Po1 name=vlan3000 vlan-id=3000
/routing table
add fib name=""
/ip address
add address=100.126.0.2/29 interface=vlan3000 network=100.126.0.0
add address=100.127.0.2 interface=Lo0 network=100.127.0.2
/ipv6 address
add address=200:100:126::2 interface=vlan3000
add address=200:100:127::2/128 advertise=no interface=Lo0
/system identity
set name=RTR-02

CSW-01

/interface bridge
add name=Bridge-MLAG vlan-filtering=yes
/interface bonding
add mlag-id=100 mode=802.3ad name=Po1 slaves=sfp-sfpplus1
add mlag-id=101 mode=802.3ad name=Po2 slaves=sfp-sfpplus2
/interface bridge mlag
set bridge=Bridge-MLAG peer-port=qsfpplus1-1
/interface bridge port
add bridge=Bridge-MLAG interface=Po1
add bridge=Bridge-MLAG interface=qsfpplus1-1 pvid=777
add bridge=Bridge-MLAG interface=Po2
/interface bridge vlan
add bridge=Bridge-MLAG tagged=Po1,Po2 vlan-ids=3000
/system identity
set name=CSW-01

CSW-02

/interface bridge
add name=Bridge-MLAG vlan-filtering=yes
/interface bonding
add mlag-id=100 mode=802.3ad name=Po1 slaves=sfp-sfpplus1
add mlag-id=101 mode=802.3ad name=Po2 slaves=sfp-sfpplus2
/interface bridge mlag
set bridge=Bridge-MLAG peer-port=qsfpplus1-1
/interface bridge port
add bridge=Bridge-MLAG interface=Po1
add bridge=Bridge-MLAG interface=qsfpplus1-1 pvid=777
add bridge=Bridge-MLAG interface=Po2
/interface bridge vlan
add bridge=Bridge-MLAG tagged=Po1,Po2 vlan-ids=3000
/system identity
set name=CSW-02

MikroTik – RouterOSv7 first look – Dynamic routing with IPv6 and OSPFv3/BGP

If you missed it, take a look at MikroTik’s video on RouterOS v7 routing performance and changes.



Overview

One of the long awaited benefits of RouterOS version 7 is a new routing protocol stack that enables new capabilities and fixes limitations in RouterOSv6 caused by the use of a very old Linux kernel.

The new routing stack in v7 has created quite a buzz in the MikroTik community as lab tests have shown that it’s significantly more efficient in processing large numbers of BGP routes.

The ability to use MikroTik’s new generation of CCR routers with ARM64 to quickly process BGP routes is a blog post all to itself and we’ll tackle that in the future – however, the information below provides a quick look into the performance comparison between ROS v6 and v7.

The new routing stack also paves the way to add a number of features that have been needed for a long time like RPKI and large community support.

Using a lab based on EVE-NG, we’ll take a look at configuration changes and iBGP using the IPv6 AFI with OSPFv3 as the IGP for loopback/next hop reachability. Prior to 7.1beta2, this has been nonfunctional for years due to routing recursion limitations.

v7 Routing Protocol Status

For the most up to date information about features and capabilities in v7, MikroTik created a page that tracks feature status across the different beta releases

https://help.mikrotik.com/docs/display/ROS/v7+Routing+Protocol+Status



Lab design

ROS Version: 7.1beta2 (7.1beta3 was released just before I published this – at some point i’ll update with testing on beta3)
Network Modeling: EVE-NG Pro



OSPFv3

One of the biggest changes in OSPF for both version 2 (IPv4) and version 3 (IPv6) is the consolidation of menus into a single location for OSPF configuration.

In ROSv7, all configuration occurs under /routing/ospf/ and instances can be created for v2 or v3.

Change from ROSv6: OSPF Menu options have changed in ROSv7, this is partly due to combining OPSFv2 and OSPFv3 into a single configuration framework.

OSPF command options in ROSv6 for OSPFv2 and OSPFv3


OSPF command options in ROSv7 for both versions of OSPF

Change from ROSv6: There is a new flag in the IPv6 routing table for ECMP and no flag for RIP

When looking at the new output for the routing table, a few things stand out. ECMP has a new flag using the “+” symbol to denote two or more equal paths.

ECMP in IPv6 is a feature limitation that RouterOSv6 had and this will make it easier to deploy IPv6 networks with MikroTik.

RIP or Routing Internet Protocol is missing from the routing flags. It’s unclear at this point whether RIPv2 or RIP-NG will make it into RouterOSv7 since it’s not used very often anymore in prod networks.

Correcting issues with recursive routing in IPv6.

Being able to use recursive routing for advertising loopbacks and using iBGP with IPv6 has been a limitation of ROSv6 for a long time due to the older linux kernel in use.

Now that ROSv7 has added the initial support for OSPF and BGP, we are able to test IPv6 routing recursion.

Here is a test from PE-1 to PE-2 (2001:db8:101::12) using iBGP

It works!


Change from ROSv6: Using filters in OSPF

One of the first major challenges I had to solve when working with ROSv7 was figuring out why every route available became advertised into OSPF.

At first it looked like a bug, but when I dug deeper, I came across this snippet in the new MikroTik help docs

ROSv7 Basic Routing Examples – RouterOS – MikroTik Documentation

As it turns out, the default behavior is to advertise all routes in the absence of an outbound filter.

The next challenge was figuring out the new filtering syntax.

/routing filter rule
add chain=OSPF-permit-only-configured
/routing filter select-rule
add chain=OSPF-permit-only-configured_select do-where=\
OSPF-permit-only-configured

In order to use a rule in ROSv7, the “/routing filter select-rule” command must be used and reference the filter rule or no action will be taken.

In the example above, only interfaces that have been configured for OSPF will be advertised.

OSPF Config

Here is a summary of the OSPF configuration from the PE-1 router

/routing ospf instance
add name=IPv6 out-filter=OSPF-permit-only-configured_select router-id=\
100.127.0.11 version=3
/routing ospf area
add area-id=0.0.0.0 instance=IPv6 name=area-0
/routing filter rule
add chain=OSPF-permit-only-configured
/routing filter select-rule
add chain=OSPF-permit-only-configured_select do-where=\
OSPF-permit-only-configured
/routing ospf interface
add area=area-0 network=ether1 network-type=point-to-point
add area=area-0 network=ether2 network-type=point-to-point
add area=area-0 network=lo-ospf network-type=broadcast




BGP

As with OSPF, BGP saw a change in menu structure.

In ROSv7, BGP configuration has been revamped and is much closer to the style of configuration that Cisco/Juniper use with config elements that can be nested and reused.

Considering all the work that’s being done to improve full table convergence time on ROSv7, this change is a step in the right direction to allow MikroTik to compete with larger network vendors in the area of peering and transit.

Change from ROSv6: BGP Menu options have changed in ROSv7 to accommodate new features like Templates and RPKI

BGP command options in ROSv6


BGP command options in ROSv7



New Feature: BGP Roles

This is a new capability in BGP as of July 2020 and MikroTik was one of the first to have it implemented.

draft-ietf-idr-bgp-open-policy-13 – Route Leak Prevention using Roles in Update and Open messages


The main goal is to classify peerings into different roles that prevent inadvertent route leaks by adopting some basic filtering policies as a component of the role assignment.

Acceptable pairings are:

Here is an example of role types in ROSv7


This is an overview of how the roles deal with route advertisements and filtering.



New Feature: BGP Templates

BGP Templates allow specific settings for a peer connection to be reused in the connection configuration.

This saves quite a bit of time when deploying a large number of iBGP peerings, transit peerings, IX peerings, etc

Options available to set in templates

Here is a BGP template as configured in the lab for this post. The template is referenced by the connection config (aka peer config)

/routing bgp template
add address-families=ipv6 as=8675309 instance=bgp name=ASN-8675309

/routing bgp connection
add local.address=2001:db8:127::11 .role=ibgp-rr-client remote.address=\
2001:db8:127::1 .as=8675309 template=ASN-8675309

New Feature: iBGP ECMP for IPv6

ECMP has been working in ROSv6 for a ling time, but due to kernel limitations, it hasn’t been available in IPv6 due to the problems in routing recursion and making iBGP operational.

Now that routing recursion is fixed for IPv6, ECMP is possible.

ECMP capable IPV6 routes in BGP noted by the new “+” symbol in the routing table for ECMP.

Here is an example of a traceroute to the same prefix that’s using two different paths with ECMP.


BGP Configuration

Here is an overview of the BGP configuration for PE-1

/routing instance
add id=100.127.0.11 name=bgp
/routing bgp template
add address-families=ipv6 as=8675309 instance=bgp name=ASN-8675309
/routing bgp connection
add local.address=2001:db8:127::11 .role=ibgp-rr-client remote.address=\
2001:db8:127::1 .as=8675309 template=ASN-8675309



Lab configurations


All Lab configs for ROSv7 are listed below (tested in 7.1beta2)

PE-1

/interface bridge
add name=lo-bgp
add name=lo-ospf
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip vrf
add list=all name=main
/routing instance
add id=100.127.0.11 name=bgp
/routing bgp template
add address-families=ipv6 as=8675309 instance=bgp name=ASN-8675309
/routing ospf instance
add name=IPv6 out-filter=OSPF-permit-only-configured_select router-id=\
    100.127.0.11 version=3
/routing ospf area
add area-id=0.0.0.0 instance=IPv6 name=area-0
/ip dhcp-client
add disabled=no interface=ether1
/ipv6 address
add address=2001:db8:126:1::2/126 advertise=no interface=ether1
add address=2001:db8:127::11/128 advertise=no interface=lo-ospf
add address=2001:db8:101::11/128 advertise=no interface=lo-bgp
add address=2001:db8:126:3::2/126 advertise=no interface=ether2
add address=2001:db8:a1a::1 interface=ether4
/routing bgp connection
add local.address=2001:db8:127::11 .role=ibgp-rr-client remote.address=\
    2001:db8:127::1 .as=8675309 template=ASN-8675309
/routing filter rule
add chain=OSPF-permit-only-configured
/routing filter select-rule
add chain=OSPF-permit-only-configured_select do-where=\
    OSPF-permit-only-configured
/routing ospf interface
add area=area-0 network=ether1 network-type=point-to-point
add area=area-0 network=ether2 network-type=point-to-point
add area=area-0 network=lo-ospf network-type=broadcast
/system identity
set name=PE-1

CORE-1

/interface bridge
add name=lo-bgp
add name=lo-ospf
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip vrf
add list=all name=main
/routing instance
add id=100.127.0.1 name=bgp
/routing bgp template
add address-families=ipv6 as=8675309 instance=bgp name=ASN-8675309
/routing ospf instance
add name=IPv6 router-id=100.127.0.1 version=3
/routing ospf area
add area-id=0.0.0.0 instance=IPv6 name=area-0
/ip dhcp-client
add disabled=no interface=ether1
/ipv6 address
add address=2001:db8:126:1::1/126 advertise=no interface=ether1
add address=2001:db8:127::1/128 advertise=no interface=lo-ospf
add address=2001:db8:126:2::1/126 advertise=no interface=ether2
add address=2001:db8:126:3::1/126 advertise=no interface=ether3
add address=2001:db8:101::1/128 advertise=no interface=lo-bgp
add address=2001:db8:126:4::1/126 advertise=no interface=ether4
/routing bgp connection
add listen=yes local.address=2001:db8:127::1 .role=ibgp-rr remote.address=\
    2001:db8:127::11 template=ASN-8675309
add listen=yes local.address=2001:db8:127::1 .role=ibgp-rr remote.address=\
    2001:db8:127::12 template=ASN-8675309
/routing filter rule
add chain=OSPFv3-in match-prfx-value=dst<equal>2001:db8:101::/128
add chain=OSPFv3-in match-prfx-value=dst<equal>2001:db8:101::/128
/routing ospf interface
add area=area-0 network=ether1 network-type=point-to-point
add area=area-0 network=ether2 network-type=point-to-point
add area=area-0 network=ether3 network-type=point-to-point
add area=area-0 network=lo-ospf network-type=broadcast
add area=area-0 network=ether4 network-type=point-to-point
/system identity
set name=CORE-1

PE-2

/interface bridge
add name=lo-bgp
add name=lo-ospf
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip vrf
add list=all name=main
/routing instance
add id=100.127.0.12 name=bgp
/routing bgp template
add address-families=ipv6 as=8675309 instance=bgp name=ASN-8675309
/routing ospf instance
add name=IPv6 out-filter=OSPF-permit-only-configured_select router-id=\
    100.127.0.12 version=3
/routing ospf area
add area-id=0.0.0.0 instance=IPv6 name=area-0
/ip dhcp-client
add disabled=no interface=ether1
/ipv6 address
add address=2001:db8:126:2::2/126 advertise=no interface=ether1
add address=2001:db8:127::12/128 advertise=no interface=lo-ospf
add address=2001:db8:101::12/128 advertise=no interface=lo-bgp
add address=2001:db8:a1b::1 interface=ether4
add address=2001:db8:126:4::2/126 advertise=no interface=ether2
/routing bgp connection
add local.address=2001:db8:127::12 .role=ibgp-rr-client remote.address=\
    2001:db8:127::1 .as=8675309 template=ASN-8675309
/routing filter rule
add chain=OSPF-permit-only-configured
/routing filter select-rule
add chain=OSPF-permit-only-configured_select do-where=\
    OSPF-permit-only-configured
/routing ospf interface
add area=area-0 network=ether1 network-type=point-to-point
add area=area-0 network=lo-ospf network-type=broadcast
add area=area-0 network=ether2 network-type=point-to-point
/system identity
set name=PE-2