A guide for working with MTU and MPLS in a Wireless ISP
MPLS/VPLS MTU math can be complicated and is always a struggle to unravel.
To make it a little easier and put it into a WISP context, I designed this cheat sheet on 8.5 x 11 (to print for those that actually trust printers) and used common WISP equipment like MikroTik routers, Ubnt and Cambium radios with real world MTU values.
The MTU values are displayed in layers to make it easier to see where each value fits.
These values are meant to be a starting point by representing the minimum values required for MPLS/VPLS with a single 802.1q VLAN tag.
In general, after going through hundreds of WISP migrations, I’ve found it to be easier to implement the minimum values required when working on a production WISP to identify the effective lowest MTU in the network.
Once the network equipment has been modified and has been running in a stable way on the minimum values, then higher values can be considered and implemented (now that the effective lowest MTU on the network is documented)
The initial results are very promising. Getting close to 10G sustained L3 throughput using an ASIC on a device that lists for $399 USD is unheard of.
The most noteworthy items for improvement are the number of TCP retransmits in the iperf testing and the speed reduction as the MTU is reduced.
Normally, most ASIC based platforms will push 96 bytes through as fast as 1500 bytes. The retransmits suggest that more work is needed in the way RouterOS interfaces with the switching buffers for L3 HW offload
This is also on a beta version of RouterOS that will still go through many revisions before going into prod so I would expect to see the performance to improve as the code matures.
All things considered though, things are looking great to take the CRS3xx series and be able to deploy them as a true L3 switch in prod sometime in 2021.