MikroTik RouterOS – v7.0.3 stable (chateau) and status of general release

If you don’t already use it, the MIkroTik v7 BETA forum (forum.mikrotik.com) is a fantastic source of information


When will stable be released?

This is the million dollar question. Technically, it already has been for one hardware platform…

!! Spoiler Alert – There is *already* a stable release of ROSv7 – v7.0.3!!

The Chateau 5G router originally shipped with a beta version of ROSv7 but was quietly moved to a stable version that’s developed specifically for that platform.

https://forum.mikrotik.com/viewtopic.php?t=175201#p865329

Because of the way MikroTik’s code repo works, this version can’t easily be added to the main download page and support provides the software:

ROSv7.0.3 Stable Download (!!! Chateau Only – will brick other hardware !!!)

https://box.mikrotik.com/f/7e3cad5779804d0b878d/?dl=1

It’s worth repeating MikroTik’s warning about using this on any platform other than the Chateau

v7 launch date – MikroTikhttps://forum.mikrotik.com/viewtopic.php?f=1&t=175201#p865452




What’s holding up v7 from being released?

If you’ve been around MikroTik for a while, then you know that version 7 has been in the works for a long time to add new functionality and address limitations of the older Linux kernel in ROSv6.

MikroTik recently added a detailed update on where the development roadmap is at and what the challenges are:

https://forum.mikrotik.com/viewtopic.php?t=175201#p865329

What does this mean?

  • Routing filters need to be rewritten to simplify the syntax and operation – there were a lot of complaints with the original syntax.
  • Routing protocols like OSPF and BGP have been unstable in beta1 through beta6 and need some work to stabilize them.

What’s the issue with routing filters?

The original v7 routing filters were very complicated to work with and had esoteric terms for operations like ‘subsumes’ and required ‘rule’ and ‘select-rule’ config to actually reference the filter in the routing process.


Previous filter syntax:

I wrote an article in late 2020 on IPv6 with BGP/OSPF using beta2 and captured a few screenshots that aren’t in the online docs anymore.

More details are in the article here:

MikroTik – RouterOSv7 first look – Dynamic routing with IPv6 and OSPFv3/BGP – StubArea51.net

New syntax?

If the current filter documentation represents the newer style, there are several differences in the format. The more complicated language like ‘subsumes’ is gone and only one filter rule is required to use the filter in the routing process – the ‘select-rule’ syntax is gone.


The new filter syntax appears to have made it onto help.mikrotik.com under /routing/filter but this may change in the coming weeks.

Example:

Similar to the OSPF example above, this is the example listed under the filter section for the new format.

The options below represent the matching order and the possible readable and set parameters.

Accepted Syntax:

if ( [matchers] ) { [actions] } else { [actions] }

[matchers]:
[prop readable] [bool operator] [prop readable]

[actions]:
[action] [prop writeable] [value]



Accepted parameters:

[num prop readable]
dst-len
bgp-path-len
bgp-input-local-as
bgp-input-remote-as
bgp-output-local-as
bgp-output-remote-as
ospf-metric
ospf-tag
rip-metric
rip-tag

[num prop writable]
distance
scope
scope-target
bgp-weigth
bgp-med
bgp-out-med
bgp-local-pref
bgp-igp-metric
bgp-path-peer-prepend
bgp-path-prepend
ospf-ext-metric
ospf-ext-tag
rip-ext-metric 
rip-ext-tag 

[flag prop readable]
active
bgp-attomic-aggregate
bgp-communities-empty
bgp-communities-ext-empty
bgp-communities-large-empty
bgp-network
ospf-dn

[flag prop writable]
ospf-ext-dn
blackhole
use-te-nexthop

[predicate]
bgp-communities|bgp-communities-ext|bgp-communities-large
    equal|any|includes|subset
	{inline set}	
    equal-set|any-set|includes-set|subset-set
	{set name}
    any-regexp|subset-regexp
	{regexp}
comment
    text|find|regexp
	{string}
chain
    {chain name}
vrf
    {vrf}
rtab
    {rtab}
gw-interface
    {interface}
gw-check
    none|arp|icmp|bfd|bfd-mh	
afi
    ipv4|ipv6|l2vpn|l2vpn-cisco|vpnv4|vpnv6
	,...
protocol
    connected|static|bgp|ospf|rip|dhcp|fantasy|modem|vpn
	,...
bpg-origin
    igp|egp|incomplete
	,...
bgp-as-path
    {regexp}
rpki
    valid|invalid|unknown
ospf-type
    intra|inter|ext1|ext2|nssa1|nssa2
ospf-ext-type
    type1|type2
[num prop readable]
    in
	{int..int}|{int-int}
    ==|!=|<=|>=|<|>
	{int}
[prfx prop readable]
    !=|==|in
	{address 46/}
[flag prop readable]


[block]
if ([predicate] &&/|| ...) { [block] } [ else {[block]} ]
accept|reject|return
jump {chain name}
unset
    pref-src|bgp-med|bgp-out-med|bgp-local-pref
bgp-communities|bgp-communities-ext|bgp-communities-large
    append|replace|filter-in|filter-not-in
	{inline community set}
    append-set|replace-set|filter-in-set|filter-not-in-set
	{set name}
    filter-in-regexp|filter-not-in-regexp
	{regexp}
    delete
	wk|other <-- for communities
	    ,...
	rt|soo|other <-- for ext-communities
	    ,...
	all <-- for large-communities
rpki-verify 
    {rpki group name}
comment
    set|append
	{string}
set
    [num prop writable]
	[num prop readable]|[num prop writable]
	+/-	
	    [num prop readable]|[num prop writable]
    gw
	interface
	    {interface}
	{address 46i}
    gw-check
	none|arp|icmp|bfd|bfd-mh	
    pref-src
	{address 46}
    bgp-origin
	igp|egp|incomplete
    ospf-ext-fwd
	{address 46}
    ospf-ext-type
	type1|type2



[num prop readable]
    in
        {int..int}|{int-int}
    ==|!=|<=|>=|<|>
        {int}
        [num prop readable]

What are the problems with routing protocols?

The routing stack has been completely re-written for ROSv7 from what we are told. This takes some time.

When I was last out in Silicon Valley, I met with a company that just emerged from stealth and had designed a new Network OS. They spent 3 years in stealth working on nothing but the OS with no actual product being sold – just coding and development.

So it’s not surprising this process has taken a while.

Stabilizing protocol issues

There are a number of bugs we’ve seen in the early versions of ROSv7 beta for routing protocols that are being worked through. Things like OSPF checksum, interface templates, high cpu when areas are disabled/enabled….etc.

Now that we know routing protocols are a priority and the new filter syntax is taking shape, I would expect to see some improvement across the next 2 to 3 beta releases to get routing protocols stable with simple configs.

L3 Switching

From talking to a lot of people that write code for Network OSes and work on the interaction with the ASIC, this is one of the hardest areas to get right – pushing routes from the RIB down into the HW FIB.

MikroTik hired new developers to meet this challenge:

https://forum.mikrotik.com/viewtopic.php?f=1&t=175369#p859571

MikroTik has updated the L3 HW doc pages to provide a roadmap of features and functionality for the beta series.

It appears that Jumbo MTU is the next major feature to be added for L3 HW in ROSv7.1beta7

https://help.mikrotik.com/docs/display/ROS/L3+Hardware+Offloading

When will ROSv7 move to release candidate and then stable?

When asked about the state of ROSv7, my typical answer has been that we’ll see a stable version in mid 2022 based on the pace of development.

I think that’s still a fair answer based on the pace of development.

It seems like the routing protocols and filters we need a few more beta versions to get working and then they’ll move to release candidates – my estimate is to look for the RCs towards the end of 2021.

Hopefully this has been helpful…i’ll probably write another summary on the state of ROSv7 once more progress has been made.

ISP Design – Building production MPLS networks with IP Infusion’s OcNOS.

Moving away from incumbent network vendors

 

1466540435IpInfusion interivew questions

 

One of the challenges service providers have faced in the last decade is lowering the cost per port or per MB while maintaining the same level of availability and service level.

And then add to that the constant pressure from subscribers to increase capacity and meet the rising demand for realtime content.

This can be an especially daunting task when routers with the feature sets ISPs need cost an absolute fortune – especially as new port speeds are released.
whitebox-switch_500px-wide

Whitebox, also called disaggregated networking, has started changing the rules of the game. ISPs are working to figure out how to integrate and move to production on disaggregated models to lower the cost of investing in higher speeds and feeds.

Whitebox often faces the perception problem of being more difficult to implement than traditional vendors – which is exactly why I wanted to highlight some of the work we’ve been doing at iparchitechs.com integrating whitebox into production ISP networks using IP Infusion’s OcNOS.

Things are really starting to heat up in the disaggregagted network space after the announcement by Amazon a few days ago that it intends to build and sell whitebox switches.

As I write this, I’m headed to Networking Field Day 18 where IP Infusion will be presenting and I expect whitebox will again be a hot topic.

This will be the second time IPI has presented at Networking Field Day but the first time that I’ve had a chance to see them present firsthand.

It’s especially exciting for me as I work on implementing IPI on a regular basis and integrating OcNOS into client networks.

 

What is OcNOS?

ip-ocnos-main-1

IP Infusion has been making network operating systems (NOS) for more than 20 years under the banner of its whitelabel NOS – ZebOS.

As disaggregated networking started to become popular, IPI created OcNOS which is an ONIE compatible NOS using elements and experience from 20 years of software development with ZebOS.

There is a great overview of OcNOS from Networking Field Day 15 here:

 

What does a production OcNOS based MPLS network look like?

Here is an overview of the EVE-NG lab we built based on an actual implementation.

 

IPI-VPLS-2

Use case – Building an MPLS core to deliver L2 overlay services

Although certainly not a new use case or implementation, MPLS and VPLS are very expensive to deploy using major vendors and are still a fundamental requirement for most ISPs.

This is where IPI really shines as they have feature sets like MPLS FRR, TE and the newer Segment Routing for OSPF and IS-IS that can be used in a platform that is significantly cheaper than incumbent network vendors.

The cost difference is so large that often ISPs are able to buy switches with a higher overall port speeds than they could from a major vendor. This in turn creates a significant competitive advantage as ISPs can take the same budget (or less) and roll out 100 gig instead of 10 gig – as an example

Unlike enterprise networks, cost is more consistently a significant driver when selecting network equipment for ISPs. This is especially true for startup ISPs that may be limited in the amount of capital that can be spent in a service area to keep ROI numbers relatively sane for investors.

Lab Overview

In the lab (and production) network we have above, OcNOS is deployed as the MPLS core at each data center and MikroTik routers are being used as MPLS PE routers.

VPLS is being run from one DC to the other and delivered via the PE routers to the end hosts.

Because the port density on whitebox switches is so high compared to a traditional aggregation router, we could even use LACP channels if dark fiber was available to increase the transport bandwidth between the DCs without a significant monetary impact on the cost of the deployment.

The type of switches that you’d use in production depend greatly on the speeds and feeds required, but for startup ISPs, we’ve had lots of success with Dell 4048s and Edge-Core 5812.


How hard is it to configure and deploy?

It’s not hard at all!

If you know how to use the up and down arrow keys in the bootloader and TFTP/FTP to load an image onto a piece of network hardware, you’re halfway there!

Here is a screenshot of the GRUB bootloader for an ONIE switch (this is a Dell) where you select which OS to boot the switch into

ONIE GRUB

The configuration is relatively straightforward as well if you’re familiar with industry standard Command Line Interfaces (CLI).

While this lab was configured in a more traditional way using a terminal session to paste commands in, OcNOS can easily be orchestrated and automated using tools like Ansible (also presenting at Networking Field Day 18) or protocols like NETCONF as well as a REST API.

Lab configs

I’ve included the configs from the lab in order to give engineers a better idea of what OcNOS actually looks like for a production deployment.

IPI-MPLS-1

 

!
!Last configuration change at 12:24:27 EDT Tue Jul 17 2018 by ocnos
!
no service password-encryption
!
hostname IPI-MPLS-1
!
logging monitor 7
!
ip vrf management
!
mpls lsp-model uniform
mpls propagate-ttl
!
ip domain-lookup
spanning-tree mode provider-rstp
data-center-bridging enable
feature telnet
feature ssh
snmp-server enable snmp
snmp-server view all .1 included
ntp enable
username ocnos role network-admin password encrypted $1$HJDzvHS1$.4/PPuAmCUEwEhs
UWeYqo0
!
ip pim register-rp-reachability
!
router ldp
 router-id 100.127.0.1
!
interface lo
 mtu 65536
 ip address 127.0.0.1/8
 ip address 100.127.0.1/32 secondary
 ipv6 address ::1/128
!
interface eth0
 ip address 100.64.0.1/29
 label-switching
 enable-ldp ipv4
!
interface eth1
 ip address 100.64.0.9/29
 label-switching
 enable-ldp ipv4
!
interface eth2
 ip address 100.64.1.1/29
 label-switching
 enable-ldp ipv4
!
interface eth3
!
interface eth4
!
interface eth5
!
interface eth6
!
interface eth7
!
router ospf 1
 ospf router-id 100.127.0.1
 network 100.64.0.0/29 area 0.0.0.0
 network 100.64.0.8/29 area 0.0.0.0
 network 100.64.1.0/29 area 0.0.0.0
 network 100.127.0.1/32 area 0.0.0.0
 cspf disable-better-protection
!
bgp extended-asn-cap
!
router bgp 8675309
 bgp router-id 100.127.0.1
 neighbor 100.127.0.3 remote-as 8675309
 neighbor 100.127.0.3 update-source lo
 neighbor 100.127.2.1 remote-as 8675309
 neighbor 100.127.2.1 update-source lo
 neighbor 100.127.2.1 route-reflector-client
 neighbor 100.127.0.4 remote-as 8675309
 neighbor 100.127.0.4 update-source lo
 neighbor 100.127.0.4 route-reflector-client
 neighbor 100.127.0.2 remote-as 8675309
 neighbor 100.127.0.2 update-source lo
 neighbor 100.127.0.2 route-reflector-client
 neighbor 100.127.1.1 remote-as 8675309
 neighbor 100.127.1.1 update-source lo
 neighbor 100.127.1.1 route-reflector-client
!
line con 0
 login
line vty 0 39
 login
!
end

IPI-MPLS-2

 

!
!Last configuration change at 12:23:31 EDT Tue Jul 17 2018 by ocnos
!
no service password-encryption
!
hostname IPI-MPLS-2
!
logging monitor 7
!
ip vrf management
!
mpls lsp-model uniform
mpls propagate-ttl
!
ip domain-lookup
spanning-tree mode provider-rstp
data-center-bridging enable
feature telnet
feature ssh
snmp-server enable snmp
snmp-server view all .1 included
ntp enable
username ocnos role network-admin password encrypted $1$RWk6XAN.$6H0GXBR9ad8eJE2
7nRUfu1
!
ip pim register-rp-reachability
!
router ldp
 router-id 100.127.0.2
!
interface lo
 mtu 65536
 ip address 127.0.0.1/8
 ip address 100.127.0.2/32 secondary
 ipv6 address ::1/128
!
interface eth0
 ip address 100.64.0.2/29
 label-switching
 enable-ldp ipv4
!
interface eth1
 ip address 100.64.0.17/29
 label-switching
 enable-ldp ipv4
!
interface eth2
 ip address 100.64.1.9/29
 label-switching
 enable-ldp ipv4
!
interface eth3
!
interface eth4
!
interface eth5
!
interface eth6
!
interface eth7
!
router ospf 1
 network 100.64.0.0/29 area 0.0.0.0
 network 100.64.0.16/29 area 0.0.0.0
 network 100.64.1.8/29 area 0.0.0.0
 network 100.127.0.2/32 area 0.0.0.0
 cspf disable-better-protection
!
bgp extended-asn-cap
!
router bgp 8675309
 bgp router-id 100.127.0.2
 neighbor 100.127.0.3 remote-as 8675309
 neighbor 100.127.0.3 update-source lo
 neighbor 100.127.0.1 remote-as 8675309
 neighbor 100.127.0.1 update-source lo
!
line con 0
 login
line vty 0 39
 login
!
end

IPI-MPLS-3

 

!
!Last configuration change at 12:25:11 EDT Tue Jul 17 2018 by ocnos
!
no service password-encryption
!
hostname IPI-MPLS-3
!
logging monitor 7
!
ip vrf management
!
mpls lsp-model uniform
mpls propagate-ttl
!
ip domain-lookup
spanning-tree mode provider-rstp
data-center-bridging enable
feature telnet
feature ssh
snmp-server enable snmp
snmp-server view all .1 included
ntp enable
username ocnos role network-admin password encrypted $1$gc9xYbW/$JlCDmgAEzcCmz77
QwmJW/1
!
ip pim register-rp-reachability
!
router ldp
 router-id 100.127.0.3
!
interface lo
 mtu 65536
 ip address 127.0.0.1/8
 ip address 100.127.0.3/32 secondary
 ipv6 address ::1/128
!
interface eth0
 ip address 100.64.0.25/29
 label-switching
 enable-ldp ipv4
!
interface eth1
 ip address 100.64.0.10/29
 label-switching
 enable-ldp ipv4
!
interface eth2
 ip address 100.64.2.1/29
 label-switching
 enable-ldp ipv4
!
interface eth3
!
interface eth4
!
interface eth5
!
interface eth6
!
interface eth7
!
router ospf 1
 ospf router-id 100.127.0.3
 network 100.64.0.8/29 area 0.0.0.0
 network 100.64.0.24/29 area 0.0.0.0
 network 100.64.2.0/29 area 0.0.0.0
 network 100.127.0.3/32 area 0.0.0.0
 cspf disable-better-protection
!
bgp extended-asn-cap
!
router bgp 8675309
 bgp router-id 100.127.0.3
 neighbor 100.127.0.1 remote-as 8675309
 neighbor 100.127.0.1 update-source lo
 neighbor 100.127.2.1 remote-as 8675309
 neighbor 100.127.2.1 update-source lo
 neighbor 100.127.2.1 route-reflector-client
 neighbor 100.127.0.4 remote-as 8675309
 neighbor 100.127.0.4 update-source lo
 neighbor 100.127.0.4 route-reflector-client
 neighbor 100.127.0.2 remote-as 8675309
 neighbor 100.127.0.2 update-source lo
 neighbor 100.127.0.2 route-reflector-client
 neighbor 100.127.1.1 remote-as 8675309
 neighbor 100.127.1.1 update-source lo
 neighbor 100.127.1.1 route-reflector-client
!
line con 0
 login
line vty 0 39
 login
!
end

IPI-MPLS-4

 

!
!Last configuration change at 12:24:49 EDT Tue Jul 17 2018 by ocnos
!
no service password-encryption
!
hostname IPI-MPLS-4
!
logging monitor 7
!
ip vrf management
!
mpls lsp-model uniform
mpls propagate-ttl
!
ip domain-lookup
spanning-tree mode provider-rstp
data-center-bridging enable
feature telnet
feature ssh
snmp-server enable snmp
snmp-server view all .1 included
ntp enable
username ocnos role network-admin password encrypted $1$6OP7UdH/$RaIxCBOGxHIt1Ao
IUyPks/
!
ip pim register-rp-reachability
!
router ldp
 router-id 100.127.0.4
!
interface lo
 mtu 65536
 ip address 127.0.0.1/8
 ip address 100.127.0.4/32 secondary
 ipv6 address ::1/128
!
interface eth0
 ip address 100.64.0.26/29
 label-switching
 enable-ldp ipv4
!
interface eth1
 ip address 100.64.0.18/29
 label-switching
 enable-ldp ipv4
!
interface eth2
 ip address 100.64.2.9/29
 label-switching
 enable-ldp ipv4
!
interface eth3
!
interface eth4
!
interface eth5
!
interface eth6
!
interface eth7
!
router ospf 1
 ospf router-id 100.127.0.4
 network 100.64.0.16/29 area 0.0.0.0
 network 100.64.0.24/29 area 0.0.0.0
 network 100.64.2.8/29 area 0.0.0.0
 network 100.127.0.4/32 area 0.0.0.0
 cspf disable-better-protection
!
bgp extended-asn-cap
!
router bgp 8675309
 bgp router-id 100.127.0.4
 neighbor 100.127.0.3 remote-as 8675309
 neighbor 100.127.0.3 update-source lo
 neighbor 100.127.0.1 remote-as 8675309
 neighbor 100.127.0.1 update-source lo
!
line con 0
 login
line vty 0 39
 login
!
end

 

MikroTik PE-1

 

# jul/17/2018 17:33:30 by RouterOS 6.38.7
# software id =
#
/interface bridge
add name=Lo0
add name=bridge-vpls-777
/interface vpls
add disabled=no l2mtu=1500 mac-address=02:BF:0A:4A:55:D0 name=vpls777 
    pw-type=tagged-ethernet remote-peer=100.127.2.1 vpls-id=8675309:777
/interface vlan
add interface=vpls777 name=vlan777 vlan-id=777
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/routing bgp instance
set default as=8675309 router-id=100.127.1.1
/routing ospf instance
set [ find default=yes ] router-id=100.127.1.1
/interface bridge port
add bridge=bridge-vpls-777 interface=ether3
add bridge=bridge-vpls-777 interface=vlan777
/ip address
add address=100.64.1.2/29 interface=ether1 network=100.64.1.0
add address=100.127.1.1 interface=Lo0 network=100.127.1.1
add address=100.64.1.10/29 interface=ether2 network=100.64.1.8
/ip dhcp-client
add disabled=no interface=ether4
/mpls ldp
set enabled=yes lsr-id=100.127.1.1 transport-address=100.127.1.1
/mpls ldp interface
add interface=ether1 transport-address=100.127.1.1
add interface=ether2 transport-address=100.127.1.1
/routing bgp peer
add name=IPI-MPLS-1 remote-address=100.127.0.1 remote-as=8675309 
    update-source=Lo0
add name=IPI-MPLS-3 remote-address=100.127.0.3 remote-as=8675309 
    update-source=Lo0
/routing ospf network
add area=backbone network=100.64.1.0/29
add area=backbone network=100.64.1.8/29
add area=backbone network=100.127.1.1/32
/system identity
set name=MIkroTik-PE1
/tool romon
set enabled=yes

 

 MikroTik PE-2

 

# jul/17/2018 17:34:23 by RouterOS 6.38.7
# software id =
#
/interface bridge
add name=Lo0
add name=bridge-vpls-777
/interface vpls
add disabled=no l2mtu=1500 mac-address=02:E2:86:F2:23:21 name=vpls777 pw-type=tagged-ethernet remote-peer=100.127.1.1 vpls-id=8675309:777
/interface vlan
add interface=vpls777 name=vlan777 vlan-id=777
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/routing bgp instance
set default as=8675309 router-id=100.127.2.1
/routing ospf instance
set [ find default=yes ] router-id=100.127.2.1
/interface bridge port
add bridge=bridge-vpls-777 interface=ether3
add bridge=bridge-vpls-777 interface=vlan777
/ip address
add address=100.64.2.2/29 interface=ether1 network=100.64.2.0
add address=100.127.2.1 interface=Lo0 network=100.127.2.1
add address=100.64.2.10/29 interface=ether2 network=100.64.2.8
/ip dhcp-client
add disabled=no interface=ether1
/mpls ldp
set enabled=yes lsr-id=100.127.2.1 transport-address=100.127.2.1
/mpls ldp interface
add interface=ether1 transport-address=100.127.2.1
add interface=ether2 transport-address=100.127.2.1
/routing bgp peer
add name=IPI-MPLS-1 remote-address=100.127.0.1 remote-as=8675309 update-source=Lo0
add name=IPI-MPLS-3 remote-address=100.127.0.3 remote-as=8675309 update-source=Lo0
/routing ospf network
add area=backbone network=100.64.2.0/29
add area=backbone network=100.64.2.8/29
add area=backbone network=100.127.2.1/32
/system identity
set name=MIkroTik-PE2
/tool bandwidth-server
set authenticate=no
/tool romon
set enabled=yes