MikroTik ISP Design: Building an 802.1q trunk between sites using VPLS and S-tag

Use Case

ISPs that use MikroTik are always looking for new ways to deliver services to customers and expand their offerings. Delivering Layer 2 at scale for customers is a design challenge that comes up frequently.

While it’s easy enough to build a VLAN nested inside of another VLAN  (see below), this requires you to build all of the VLANs a customer wants to use into the PE router or handoff switch.

However, if you have a client that needs a layer 2 service delivered to two or more points and wants to be able to treat it just like an 802.1q trunk and add VLANs in an ad-hoc way, then using the S-Tag feature in RouterOS along with VPLS transport is a great option.

What’s the S-tag do???

S-Tag

Clients will often ask me “what’s the S-Tag check box for?”

So a little background on this, there is a protocol for using outer and inner VLAN tags specified in IEEE 802.1ad that uses Service Tag (or S-Tag) to denote the outer VLAN tag used to transport Customer Tags (or C-Tags).

What makes the S-Tag/C-Tag a little bit different is that it actually changes the ethertype of the Frame.

ProtocolEthertype
802.1q (Normal VLAN Tags) 0x8100
802.1ad (S-tag) 0x88a8

Here is an overview of the frame format of each and links to the Metro Ethernet Forum Wiki for more info.

S-Tag

https://wiki.mef.net/display/CESG/S-Tag
S-Tag

C-Tag

https://wiki.mef.net/display/CESG/C-Tag

C-Tag

Lab Scenario

Here is a very common example of a deployment for a Layer 2 service to an end customer that rides on top of the ISP MPLS core.

In this lab we are using Cisco switches trunked to each other using VLAN 101 and 201 over a VPLS pseudowire with an S-Tag of 777.

s-tag lab

After configuring the P routers, PE routers and Cisco switches, let’s take a look at the Cisco switch and see if we can ping the SVI on the other switch on both trunked VLANs.

Here are the subnets used on the customer side:

Switch-1 subnets

Now let’s ping the .2 address for each VLAN on Switch-2

VLAN 101

vlan-101-ping

VLAN 201

vlan-201-ping

Notes on MTU

A note on MTU sizing, in order to hand off a 1500 byte packet with VPLS, you normally need an MPLS and L2MTU of 1530 bytes. In order to pass a second VLAN tag you’ll want to make sure your network equipment can go up to 1534 for Layer 2 and MPLS MTUs to pass 1500 byte packet with S-Tag.

Configs for the lab

In the section below, here are all the configs for this deployment

Cisco Switch-1

version 15.2
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
service compress-config
!
hostname Switch-1
!
boot-start-marker
boot-end-marker
!
!
!
no aaa new-model
!
!
!
!
!
!
!
!
ip cef
no ipv6 cef
!
!
!
spanning-tree mode pvst
spanning-tree extend system-id
!
vlan internal allocation policy ascending
!
!
!
!
!
!
!
!
!
!
!
!
!
!
interface GigabitEthernet0/0
 switchport trunk encapsulation dot1q
 switchport mode trunk
 media-type rj45
 negotiation auto
!
interface GigabitEthernet0/1
 media-type rj45
 negotiation auto
!
interface GigabitEthernet0/2
 media-type rj45
 negotiation auto
!
interface GigabitEthernet0/3
 media-type rj45
 negotiation auto
!
interface GigabitEthernet1/0
 media-type rj45
 negotiation auto
!
interface GigabitEthernet1/1
 media-type rj45
 negotiation auto
!
interface GigabitEthernet1/2
 media-type rj45
 negotiation auto
!
interface GigabitEthernet1/3
 media-type rj45
 negotiation auto
!
interface Vlan101
 description customer-vlan
 ip address 192.168.101.1 255.255.255.0
!
interface Vlan201
 description customer vlan 2
 ip address 192.168.201.1 255.255.255.0
!
ip forward-protocol nd
!
no ip http server
no ip http secure-server
!
!
!
!
!
!
control-plane
!
banner exec ^C
**************************************************************************
* IOSv is strictly limited to use for evaluation, demonstration and IOS  *
* education. IOSv is provided as-is and is not supported by Cisco's      *
* Technical Advisory Center. Any use or disclosure, in whole or in part, *
* of the IOSv Software or Documentation to any third party for any       *
* purposes is expressly prohibited except as otherwise authorized by     *
* Cisco in writing.                                                      *
**************************************************************************^C
banner incoming ^C
**************************************************************************
* IOSv is strictly limited to use for evaluation, demonstration and IOS  *
* education. IOSv is provided as-is and is not supported by Cisco's      *
* Technical Advisory Center. Any use or disclosure, in whole or in part, *
* of the IOSv Software or Documentation to any third party for any       *
* purposes is expressly prohibited except as otherwise authorized by     *
* Cisco in writing.                                                      *
**************************************************************************^C
banner login ^C
**************************************************************************
* IOSv is strictly limited to use for evaluation, demonstration and IOS  *
* education. IOSv is provided as-is and is not supported by Cisco's      *
* Technical Advisory Center. Any use or disclosure, in whole or in part, *
* of the IOSv Software or Documentation to any third party for any       *
* purposes is expressly prohibited except as otherwise authorized by     *
* Cisco in writing.                                                      *
**************************************************************************^C
!
line con 0
line aux 0
line vty 0 4
!
!
end

Cisco Switch-2

version 15.2
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
service compress-config
!
hostname Switch-2
!
boot-start-marker
boot-end-marker
!
!
!
no aaa new-model
!
!
!
!
!
!
!
!
ip cef
no ipv6 cef
!
!
!
spanning-tree mode pvst
spanning-tree extend system-id
!
vlan internal allocation policy ascending
!
!
!
!
!
!
!
!
!
!
!
!
!
!
interface GigabitEthernet0/0
 switchport trunk encapsulation dot1q
 switchport mode trunk
 media-type rj45
 negotiation auto
!
interface GigabitEthernet0/1
 media-type rj45
 negotiation auto
!
interface GigabitEthernet0/2
 media-type rj45
 negotiation auto
!
interface GigabitEthernet0/3
 media-type rj45
 negotiation auto
!
interface GigabitEthernet1/0
 media-type rj45
 negotiation auto
!
interface GigabitEthernet1/1
 media-type rj45
 negotiation auto
!
interface GigabitEthernet1/2
 media-type rj45
 negotiation auto
!
interface GigabitEthernet1/3
 media-type rj45
 negotiation auto
!
interface Vlan101
 description customer-vlan
 ip address 192.168.101.2 255.255.255.0
!
interface Vlan201
 description customer vlan 2
 ip address 192.168.201.2 255.255.255.0
!
ip forward-protocol nd
!
no ip http server
no ip http secure-server
!
!
!
!
!
!
control-plane
!
banner exec ^C
**************************************************************************
* IOSv is strictly limited to use for evaluation, demonstration and IOS  *
* education. IOSv is provided as-is and is not supported by Cisco's      *
* Technical Advisory Center. Any use or disclosure, in whole or in part, *
* of the IOSv Software or Documentation to any third party for any       *
* purposes is expressly prohibited except as otherwise authorized by     *
* Cisco in writing.                                                      *
**************************************************************************^C
banner incoming ^C
**************************************************************************
* IOSv is strictly limited to use for evaluation, demonstration and IOS  *
* education. IOSv is provided as-is and is not supported by Cisco's      *
* Technical Advisory Center. Any use or disclosure, in whole or in part, *
* of the IOSv Software or Documentation to any third party for any       *
* purposes is expressly prohibited except as otherwise authorized by     *
* Cisco in writing.                                                      *
**************************************************************************^C
banner login ^C
**************************************************************************
* IOSv is strictly limited to use for evaluation, demonstration and IOS  *
* education. IOSv is provided as-is and is not supported by Cisco's      *
* Technical Advisory Center. Any use or disclosure, in whole or in part, *
* of the IOSv Software or Documentation to any third party for any       *
* purposes is expressly prohibited except as otherwise authorized by     *
* Cisco in writing.                                                      *
**************************************************************************^C
!
line con 0
line aux 0
line vty 0 4
!
!
end

MikroTik PE-1

/interface bridge
add name=Lo0
add name=vpls-bridge-vlan-777
/interface vpls
add disabled=no l2mtu=1500 mac-address=02:D5:C2:72:3A:1A name=vpls777 
    pw-type=tagged-ethernet remote-peer=100.127.1.2 vpls-id=8675309:777
/ip neighbor discovery
set ether2 discover=no
/interface vlan
add interface=vpls777 name=vlan777 use-service-tag=yes vlan-id=777
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/routing ospf instance
add name=ospf1 router-id=100.127.1.1
/interface bridge port
add bridge=Lo0 interface=ether2
add bridge=Lo0 interface=vlan777
/ip address
add address=100.64.0.1/29 interface=ether1 network=100.64.0.0
add address=100.127.1.1 interface=Lo0 network=100.127.1.1
/ip dhcp-client
add disabled=no interface=ether1
/mpls interface
set [ find default=yes ] mpls-mtu=1534
/mpls ldp
set enabled=yes lsr-id=100.127.1.1 transport-address=100.127.1.1
/mpls ldp interface
add interface=ether1 transport-address=100.127.1.1
/routing ospf network
add area=backbone network=100.64.0.0/29
add area=backbone network=100.127.1.1/32
/system identity
set name=MikroTik-PE-1
/tool romon
set enabled=yes

MikroTik PE-2

/interface bridge
add name=Lo0
add name=vpls-bridge-vlan-777
/interface vpls
add disabled=no l2mtu=1500 mac-address=02:C1:71:EB:0E:E7 name=vpls777 
    pw-type=tagged-ethernet remote-peer=100.127.1.1 vpls-id=8675309:777
/ip neighbor discovery
set ether2 discover=no
/interface vlan
add interface=vpls777 name=vlan777-s-tag use-service-tag=yes vlan-id=777
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/routing ospf instance
add name=ospf1 router-id=100.127.1.2
/interface bridge port
add bridge=Lo0 interface=vlan777-s-tag
add bridge=Lo0 interface=ether2
/ip address
add address=100.64.0.18/29 interface=ether1 network=100.64.0.16
add address=100.127.1.2 interface=Lo0 network=100.127.1.2
/ip dhcp-client
add dhcp-options=hostname,clientid disabled=no interface=ether1
/mpls interface
set [ find default=yes ] mpls-mtu=1534
/mpls ldp
set enabled=yes lsr-id=100.127.1.2 transport-address=100.127.1.2
/mpls ldp interface
add interface=ether1 transport-address=100.127.1.2
/routing ospf network
add area=backbone network=100.64.0.16/29
add area=backbone network=100.127.1.2/32
/system identity
set name=MikroTik-PE-2
/tool romon
set enabled=yes

MikroTik P-CORE-1

/interface bridge
add name=Lo0
/interface vlan
add interface=ether1 name=vlan101 vlan-id=777
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/routing ospf instance
set [ find default=yes ] router-id=100.127.0.1
/ip address
add address=100.64.0.2/29 interface=ether1 network=100.64.0.0
add address=100.127.0.1 interface=Lo0 network=100.127.0.1
add address=100.64.0.9/29 interface=ether2 network=100.64.0.8
/ip dhcp-client
add dhcp-options=hostname,clientid disabled=no interface=ether1
/mpls interface
set [ find default=yes ] mpls-mtu=1534
/mpls ldp
set enabled=yes lsr-id=100.127.0.1 transport-address=100.127.0.1
/mpls ldp interface
add interface=ether1 transport-address=100.127.0.1
add interface=ether2 transport-address=100.127.0.1
/routing ospf network
add area=backbone network=100.64.0.0/29
add area=backbone network=100.127.0.1/32
add area=backbone network=100.64.0.8/29
/system identity
set name=MikroTik-P-Core-1
/tool romon
set enabled=yes

MikroTik P-CORE-2

/interface bridge
add name=Lo0
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/routing ospf instance
set [ find default=yes ] router-id=100.127.0.2
/ip address
add address=100.64.0.10/29 interface=ether1 network=100.64.0.8
add address=100.127.0.2 interface=Lo0 network=100.127.0.2
add address=100.64.0.17/29 interface=ether2 network=100.64.0.16
/ip dhcp-client
add dhcp-options=hostname,clientid disabled=no interface=ether1
/mpls interface
set [ find default=yes ] mpls-mtu=1534
/mpls ldp
set enabled=yes lsr-id=100.127.0.2 transport-address=100.127.0.2
/mpls ldp interface
add interface=ether1 transport-address=100.127.0.2
add interface=ether2 transport-address=100.127.0.2
/routing ospf network
add area=backbone network=100.64.0.8/29
add area=backbone network=100.127.0.2/32
add area=backbone network=100.64.0.16/29
/system identity
set name=MikroTik-P-Core-2
/tool romon

The one thing private equity firms typically overlook when funding startup WISPs…professional network engineers.

22893884 - telecommunications radio tower or mobile phone base station with engineers in concept background

.

Funding a new WISP

There are a number of ways to fund a startup Wireless Internet Service Provider (WISP), but the two we most commonly see are self-funded by individuals/partners or by leveraging private equity (PE) money.

Private equity has become increasingly popular in the last few years if we are to use our consulting clients as a basis for comparison.

It’s not hard to see why, while you can (and many do) start a WISP on a shoestring budget, getting a significant chunk of initial funding to cover the costs of tower construction/leasing, network equipment, sales/marketing, etc is very attractive as it allows a WISP to build a network that might otherwise take several years of organic growth to achieve.

Network Engineering – the missing ingredient

Many startup WISPs are often borne out of necessity – fast, reliable or economical Internet access – one or more of these is missing in the areas we see WISPs develop.

Typically the stakeholders come from a variety of backgrounds some of which are technical and some aren’t – all of them, however, share a vision of building out Internet access and solving problems that are unique to their corner of the world.

Out of that group, probably less than 5% come out of the ranks of professional network engineers.

And this isn’t to say that you need to be a network engineer to start a successful WISP, quite the contrary, the most successful WISPs are formed by people who understand what a well-run business looks like.

But when you’re in the business of building and selling access to a service provider IP network, at some point, you will benefit from having a network engineer as part of your team – whether that happens in the beginning or down the road is the core focus of this article.

38023860 - frustrated technician working in server room of data center

.

Build it and then fix it

Build it and then fix it later is probably the most common approach when it comes to the network design of a new WISP. Many startups (understandably) want to save money anywhere they can –  consultants or tech labor costs can be steep when money is only going out and not coming in.

“Let’s just get it up and running so we can get some revenue and then we’ll fix things”

I’ve probably heard this phrase uttered a few hundred times when working with startup WISPs over the last decade.

This is the main reason bridged networks are so commonly found in startup WISPs. The network engineering is far simpler when everything is a single network and broadcast domain.

And it’s easy to see the allure of this approach – No subnetting, VLANs, routing protocols or advanced protocols like MPLS are required.

However, this is often the beginning of a painful journey for many WISPs that will result in an initial network redesign somewhere between 500 to 1000 subscribers as the broadcast domain will reach a point where performance starts to suffer and the mad dash to fix it begins.

The cost of redesign

Network redesigns are costly in the form of equipment, labor, downtime and lost opportunities. One of the key pieces of advice we give to prospective clients whether we work for them or not is to get someone that is a network professional involved with the network design from day one – whether it’s a consultant or an in house network engineer.

When we come in and perform a network redesign, there will be elements we use common to most ISP networks like:

  • A Subnet/VLAN plan that allows for both growth at a tower and scaling of multiple towers
  • Use of an IPAM/DCIM like Netbox
  • When to use RFC 1918 and 6598 IP space vs. Public IP space
  • IPv4/IPv6 and CGN
  • Capacity planning – what speeds and feeds are needed now and in the future
  • L2/L3 design for Data Centers/PoPs
  • L2/L3 design for towers and aggregation networks
  • Routing architecture design that includes an IGP like OSPF along with BGP and MPLS
  • Planning for Billing, DNS, DHCP and other services.
  • Security and DDoS planning

Typically, the response we get from most clients is something like “wow, I wish I’d known about all of this before I started and just done it on day one – it would have been cheaper”

The cost of going through the redesign exercise can be substantial, even for a very small WISP, it can exceed $10k and for larger WISPs, it can easily be $100k or more.

It might seem like the list above is fairly complex for a very small network with only one tower and one Internet feed, but the reality is that if you build a design that’s ready to scale and template on day one, it will be far easier to grow quickly and not have to forklift those components in at a later date with large out of pocket costs.

71734956_m

.

How does Private Equity fit in?

When getting private equity firms to fund a new WISP, there are a number of budget items that go into the business plan but a plan to hire network engineers or consultants is rarely among them…why?

Many of the stakeholders involved in WISPs are often savvy tech people that are able to learn and digest new things quickly. I believe there is a perception that it’s less costly to start with a simplistic network design using the existing team and learn ‘on the fly’ rather than hire an experienced networker as a consultant or especially as an employee.

Before I get too much hate mail, there are plenty of success stories of WISPs that were built from the ground up that had to learn on the fly and were able to build a great network – so i’m not trying to paint a picture that it can’t ever be done.

The trial and error approach has a better chance of succeeding when you’re self-funded as you can gauge when you need help instead of getting hammered by investors when the network isn’t working well.

Private equity firms, however, are looking for a predictable return on their investment and skipping proper network design and planning with an experienced professional often puts a huge dent in the projected ROI for a number of reasons like:

  • Poor network performance which leads to slow or even negative subscriber growth
  • Higher install and truck roll costs due to the use of non-standardized and validated templates for subscribers
  • Network redesign costs
  • Outages due to lack of HA design
  • Excessive purchase of equipment due to lack of formal capacity planning and understanding of equipment use cases.

And the list keeps going. These are just a few of the real-world examples we have seen that get categorized as “unforeseen costs.”

Experience is the key for investors to controlling costs and building a reliable production network that will produce a consistent ROI.

The key takeaway here for private equity firms and entrepreneurs that are seeking money via PE is to get professional network engineering incorporated into the budget at the beginning to minimize the list of “unforseen costs” in the first 180 to 365 days of network operation.

Whether you hire a consulting firm or a a full or part time network engineer to assist with this step, the key is to get someone who has experience with ISP operations and design – with WISPs especially.

The money saved from using a design validated by a professional will be likely be substantial and rapid growth is far easier when the network is built right from the beginning – both of these put the PE Firm and the WISP Owner/Operators in a great position to succeed and return value to investors in a shorter amount of time.