WISP cheat sheet for MPLS/VPLS MTU

A guide for working with MTU and MPLS in a Wireless ISP

MPLS/VPLS MTU math can be complicated and is always a struggle to unravel.

To make it a little easier and put it into a WISP context, I designed this cheat sheet on 8.5 x 11 (to print for those that actually trust printers) and used common WISP equipment like MikroTik routers, Ubnt and Cambium radios with real world MTU values.

The MTU values are displayed in layers to make it easier to see where each value fits.

PDF is here

A starting point

These values are meant to be a starting point by representing the minimum values required for MPLS/VPLS with a single 802.1q VLAN tag.

In general, after going through hundreds of WISP migrations, I’ve found it to be easier to implement the minimum values required when working on a production WISP to identify the effective lowest MTU in the network.

Once the network equipment has been modified and has been running in a stable way on the minimum values, then higher values can be considered and implemented (now that the effective lowest MTU on the network is documented)

https://iparchitechs.com/contact

MikroTik – RouterOSv7 first look – L3 ASIC performance testing

When MikroTik announced the CRS3xx series switches a few years ago, one of the most exciting aspects of that news release was the prospect of L3 forwarding in hardware on very inexpensive devices.

A quick review of the Marvell Prestera ASIC family showed a number of advanced routing, switching, MPLS and VxLAN capabilites.

Fast forward to 2020, where MikroTik has started to enable some of those features in RouterOS v7 beta.

Now we can finally take some of the CRS3xx switches and test their capabilities with L3 forwarding performance in hardware


CRS 3xx series capabilities overview

Before getting into the testing, it’s probably helpful to review some of the basic specs and capabilities of the CRS3xx switch line.

Here is a chart from MikroTik that outlines ACL rule count, Unicast FDB entries and MTU size.

CRS 3xx model comparison

MIkroTik has been working on the development of the features listed below to offload into hardware.

For the tests in this article, we’ll be using IPv4 Unicast and Inter-VLAN routing.

Supported feature list

Currently, the following switches are supported.

For the testing in this article, we are using the CRS317-1G-16S+

Switches supported by 7.1beta2



Performance testing – overview

The physical setup for testing is fairly straightforward.

  • HP DL380 gen7 with ProxMox PVE 6.2-4
  • 2 x 10G Multimode OM4 fiber jumpers
  • 4 x 10G Multimode SFPs
  • CRS 317-1G-16S+ running RouterOS 7.1beta2

The logical setup is also very straightforward

  • ProxMox/KVM hypervisor using PVE 6.2-4
  • Two RFC1918 subnets and VLANs to test intervlan routing performance
  • Two Ubuntu 20.04 LTS VMs
  • iperf3 on both Ubuntu VMs

MikroTik Configuration

/interface bridge
add mtu=9216 name=bridge1 vlan-filtering=yes
add name=lo0
/interface ethernet
set [ find default-name=sfp-sfpplus15 ] comment="Proxmox - ens2f0" l2mtu=10218 mtu=9216
set [ find default-name=sfp-sfpplus16 ] comment="Proxmox - ens2f1" l2mtu=10218 mtu=9216
/interface vlan
add interface=bridge1 mtu=9216 name=vlan103 vlan-id=103
add interface=bridge1 mtu=9216 name=vlan104 vlan-id=104
/interface ethernet switch
set 0 l3hw=yes
/ip pool
add name=dhcp_pool0 ranges=10.255.34.11-10.255.34.254
add name=dhcp_pool1 ranges=10.255.35.2-10.255.35.254
/ip dhcp-server
add address-pool=dhcp_pool0 disabled=no interface=vlan103 name=dhcp1
add address-pool=dhcp_pool1 disabled=no interface=vlan104 name=dhcp2
/ip vrf
add list=all name=main
/interface bridge port
add bridge=bridge1 interface=sfp-sfpplus15 pvid=103
add bridge=bridge1 interface=sfp-sfpplus16 pvid=104
/interface bridge vlan
add bridge=bridge1 tagged=bridge1 untagged=sfp-sfpplus15 vlan-ids=103
add bridge=bridge1 tagged=bridge1 untagged=sfp-sfpplus16 vlan-ids=104
/ip address
add address=10.255.34.1/24 interface=vlan103 network=10.255.34.0
add address=10.255.35.1/24 interface=vlan104 network=10.255.35.0
/ip dhcp-server network
add address=10.255.34.0/24 dns-server=9.9.9.9 gateway=10.255.34.1
add address=10.255.35.0/24 dns-server=9.9.9.9 gateway=10.255.35.1


Test #1 – iperf3 testing at 96 byte MTU

CPU – 0% to 3%

CRS 317 – interface sfp-sfpplus15

iperf3 results

Test #2 – iperf3 testing at 512 byte MTU

CPU – 0% to 3%

CRS 317 – interface sfp-sfpplus15

iperf3 results

Test #3 – iperf3 testing at 1500 byte MTU

CPU – 0% to 3%

CRS 317 – interface sfp-sfpplus15

iperf3 results

Conclusions

The initial results are very promising. Getting close to 10G sustained L3 throughput using an ASIC on a device that lists for $399 USD is unheard of.

The most noteworthy items for improvement are the number of TCP retransmits in the iperf testing and the speed reduction as the MTU is reduced.

Normally, most ASIC based platforms will push 96 bytes through as fast as 1500 bytes. The retransmits suggest that more work is needed in the way RouterOS interfaces with the switching buffers for L3 HW offload

This is also on a beta version of RouterOS that will still go through many revisions before going into prod so I would expect to see the performance to improve as the code matures.

All things considered though, things are looking great to take the CRS3xx series and be able to deploy them as a true L3 switch in prod sometime in 2021.