IP Infusion: EVPN-MPLS first look on GA 6.0

IP Infusion just released OcNOS version 6.0 and the release notes, as well as press release, show a focus on EVPN with an MPLS data plane. Don’t forget EVPN and VxLAN aren’t mutually exclusive, EVPN runs on and was originally designed for a MPLS data plane. I recently discussed this on a podcast EVPN doesn’t need VxLAN if you want to know more on that topic.

Lets take a look at basic EVPN-VPWS and EVPN-VPLS deployment. Since we’re looking at an MPLS data plane we will utilize ISIS-SR for MPLS. We’re utilizing ISIS-SR as it is increasingly replacing LDP and RSVP-TE for label distribution.

IGP and Label Distribution

First let’s look at the IGP setup and label distribution as everything else will be built on top of this.

ipi-1.lab.jan1.us.ipa.net#show run int lo
interface lo
 ip address 127.0.0.1/8
 ip address 100.127.0.1/32 secondary
 ipv6 address ::1/128
 ipv6 address 2001:db8::1/128
 prefix-sid index 101
 ip router isis UNDERLAY
 ipv6 router isis UNDERLAY
!

We have to set an index to create the node-sid for this device. In this case we use 101.

ipi-1.lab.jan1.us.ipa.net#show run segment-routing
segment-routing
 mpls sr-prefer
 global block 16000 23999

Since our segment routing global block starts at 16000 the node-sid becomes 16101 as the index + the start of the SRGB defines the sid. Additionally, we run mpls sr-prefer as this will prefer SR labels over LDP or RSVP-TE labels.

ipi-1.lab.jan1.us.ipa.net#show run isis
router isis UNDERLAY
 is-type level-1-2
 metric-style wide
 mpls traffic-eng router-id 100.127.0.1
 mpls traffic-eng level-1
 mpls traffic-eng level-2
 capability cspf
 dynamic-hostname
 fast-reroute ti-lfa level-1 proto ipv4
 fast-reroute ti-lfa level-2 proto ipv4
 net 49.0015.1001.2700.0001.00
 segment-routing mpls
!

Finally, we have to enable ISIS for segment routing.

ipi-1.lab.jan1.us.ipa.net#show clns neighbors

Total number of L1 adjacencies: 1
Total number of L2 adjacencies: 1
Total number of adjacencies: 2
Tag UNDERLAY:  VRF : default
System Id      Interface   SNPA                State  Holdtime  Type Protocol
ipi-2.lab.jan1.us.ipa.net xe48        3c2c.99c0.00aa      Up     26        L1L2 IS-IS
ipi-1.lab.jan1.us.ipa.net#show mpls ilm-table
Codes: > - installed ILM, * - selected ILM, p - stale ILM
        K - CLI ILM, T - MPLS-TP, s - Stitched ILM
       S - SNMP, L - LDP, R - RSVP, C - CRLDP
       B - BGP , K - CLI , V - LDP_VC, I - IGP_SHORTCUT
       O - OSPF/OSPF6 SR, i - ISIS SR, k - SR CLI
       P - SR Policy, U - unknown

Code    FEC/VRF/L2CKT    ILM-ID      In-Label    Out-Label   In-Intf    Out-Intf/VRF       Nexthop
     LSP-Type
   i>   100.127.0.1/32     4           16101       Nolabel     N/A        N/A              127.0.0.1
     LSP_DEFAULT
   B>   evpn:1             3           17          Nolabel     N/A        N/A              127.0.0.1
     LSP_DEFAULT
   B>   evpn:100           1           16          Nolabel     N/A        N/A              127.0.0.1
     LSP_DEFAULT
   B>   evpn:1             2           640         Nolabel     N/A        N/A              127.0.0.1
     LSP_DEFAULT
   P>   100.127.0.2/32     7           20          3           N/A        xe48             100.126.0.2
     LSP_DEFAULT
   i>   100.126.0.2/32     5           26240       3           N/A        xe48             100.126.0.2
     LSP_DEFAULT
   i>   100.127.0.2/32     6           16102       3           N/A        xe48             100.126.0.2
     LSP_DEFAULT

Now we can see that we have a clns/isis neighbor with ipi-2 as well as learned labels. We can see both device’s node-sids in the label table on ipi-1.

This image has an empty alt attribute; its file name is IPA-Blog-ad-template-network.jpg
iparchitechs.com/contact

BGP EVPN Setup

Next we can build EVPN on top of the underlay to begin delivering services. First we have to build an EVPN BGP session between the two routers.

ipi-1.lab.jan1.us.ipa.net#show run bgp
!
router bgp 65000
 neighbor 100.127.0.2 remote-as 65000
 neighbor 100.127.0.2 update-source lo
 !
 address-family l2vpn evpn
 neighbor 100.127.0.2 activate
 exit-address-family
 !
ipi-1.lab.jan1.us.ipa.net#show bgp l2vpn evpn summary
BGP router identifier 100.127.0.1, local AS number 65000
BGP table version is 32
1 BGP AS-PATH entries
0 BGP community entries

Neighbor                 V   AS   MsgRcv    MsgSen TblVer   InQ   OutQ    Up/Down   State/PfxRcd     AD  MACIP
MCAST    ESI  PREFIX-ROUTE
100.127.0.2              4 65000 22856      22856      32      0      0  6d18h34m               2      1      0
     1      0      0

EVPN-VPWS

Next we can start build services on top. First we’ll build an EVPN-VPWS service.

ipi-1.lab.jan1.us.ipa.net:
!
evpn mpls enable
!
evpn mpls vtep-ip-global 100.127.0.1
!
mac vrf BLUE
 rd 100.127.0.1:1
 route-target both evpn-auto-rt
!
evpn mpls id 100 xconnect target-mpls-id 2
 host-reachability-protocol evpn-bgp BLUE
!
interface xe46.10 switchport
 encapsulation dot1q 10
 access-if-evpn
  map vpn-id 100
!

EVPN MPLS has to be enabled. *IMPORTANT* This requires a reboot. Next the vtep id needs to be set. These are global settings for the environment.

For the creation of the service we’ll start by making a mac vrf to generate the information needed to create a EVPN type 2 route (mac-ip).

Since this is VPWS it is considered a cross connect xconnect and a target is defined. This is the remote PE vpn-id, in this case 2.

Finally it is assigned to a switchport. It has to be a switchport with a type of access-if-evpn. This maps back to the EVPN mac-vrf via the xconnect. Anything arriving on xe46.10 with a dot1q tag of 10 is placed into this tunnel.

ipi-1.lab.jan1.us.ipa.net#show evpn mpls xconnect
EVPN Xconnect Info
========================
AC-AC: Local-Cross-connect
AC-NW: Cross-connect to Network
AC-UP: Access-port is up
AC-DN: Access-port is down
NW-UP: Network is up
NW-DN: Network is down
NW-SET: Network and AC both are up

Local                            Remote       Connection-Details

================================ ============ ==========================================================================
=========
VPN-ID       EVI-Name      MTU   VPN-ID       Source       Destination                   PE-IP           MTU   Type   NW
-Status
================================ ============ ==========================================================================
=========
100          ----          1500  2            xe46.10      --- Single Homed Port ---     100.127.0.2     1500  AC-NW  NW
-SET

Total number of entries are 1
ipi-1.lab.jan1.us.ipa.net#show evpn mpls xconnect tunnel
EVPN-MPLS Network tunnel Entries
Source           Destination      Status        Up/Down       Update        local-evpn-id remote-evpn-id
========================================================================================================
100.127.0.1      100.127.0.2      Installed     01:31:06      01:31:06      100           2

Total number of entries are 1

The tunnels are up, installed, and ready for forwarding. We can see the CE macs as mac-ip routes in evpn.

ipi-1.lab.jan1.us.ipa.net#show bgp l2vpn evpn vrf BLUE
BGP table version is 1, local router ID is 100.127.0.1
Status codes: s suppressed, d damped, h history, a add-path, * valid, > best, i - internal,
              l - labeled, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

[EVPN route type]:[ESI]:[VNID]:[relevent route informantion]
1 - Ethernet Auto-discovery Route
2 - MAC/IP Route
3 - Inclusive Multicast Route
4 - Ethernet Segment Route
5 - Prefix Route

    Network          Next Hop            Metric    LocPrf	Weight     Path  Peer          Encap
* i  [1]:[0]:[2]:[16]  100.127.0.2          0        100       0    i  100.127.0.2     MPLS
*>   [1]:[0]:[100]:[16]
                       100.127.0.1          0        100       32768  i  ----------      MPLS

Total number of prefixes 2

The mac addresses are sent via an EVPN type-2 route between PEs.

ipi-1.lab.jan1.us.ipa.net#show evpn mpls mac-table
========================================================================================================================
=================
                                                     EVPN MPLS MAC Entries
========================================================================================================================
=================
VNID       Interface VlanId    In-VlanId Mac-Addr       VTEP-Ip/ESI                    Type            Status     MAC mo
ve AccessPortDesc
________________________________________________________________________________________________________________________
_________________


Total number of entries are : 0

Since this is VPWS there are no macs learned on the device.

[email protected]# run ping 172.16.0.2
PING 172.16.0.2 (172.16.0.2): 56 data bytes
64 bytes from 172.16.0.2: icmp_seq=0 ttl=64 time=21.531 ms
64 bytes from 172.16.0.2: icmp_seq=1 ttl=64 time=22.124 ms

Success! The CEs can reach each other over the EVPN-VPWS circuit.

EVPN-VPLS

Now we’ll build an EVPN-VPLS service. The BGP setup is the same so we’ll focus solely on the differences. The first one being the vpn-id creation.

mac vrf ORANGE
 rd 100.127.0.1:2
 route-target both evpn-auto-rt
!
evpn mpls id 1
 host-reachability-protocol evpn-bgp ORANGE
!

There is no end point defined as a xconnect. All that is necessary is to bind the mac vrf to the evpn vpn id.

interface xe46.100 switchport
 encapsulation dot1q 100
 access-if-evpn
  map vpn-id 1
!

Again, a switchport defined as an access-if-evpn is necessary. This is then mapped to the vpn-id for the VPLS service. In this case anything coming in with a dot1q tag of 100 will be placed into vpn-id 1.

ipi-1.lab.jan1.us.ipa.net#show evpn mpls mac-table
========================================================================================================================
=================
                                                     EVPN MPLS MAC Entries
========================================================================================================================
=================
VNID       Interface VlanId    In-VlanId Mac-Addr       VTEP-Ip/ESI                    Type            Status     MAC mo
ve AccessPortDesc
________________________________________________________________________________________________________________________
_________________

1          xe46.100  ----      ----      84c1.c132.5031 100.127.0.1                    Dynamic Local   -------    0
   -------
1          ----      ----      ----      84c1.c132.5032 100.127.0.2                    Dynamic Remote  -------    0
   -------

Total number of entries are : 2

Since this is a VPLS service MACs are learned both locally and remotely. The remote MAC is the MAC of the remote CE. This was learned via EVPN and from the VTEP 100.127.0.2.

ipi-1.lab.jan1.us.ipa.net#show bgp l2vpn evpn mac-ip vrf ORANGE
ESI                            Eth-Tag     Mac-Address    IP-Address                              VNID/LABEL     L3VNID
   Nexthop         GW-Type         Encap
0                              1           84c1:c132:5031 --                                      17             0
   100.127.0.1     --              MPLS
0                              1           84c1:c132:5032 --                                      17             0
   100.127.0.2     --              MPLS

The type-2 routes are populated in the BGP table.

[email protected]# run ping 192.168.0.2
PING 192.168.0.2 (192.168.0.2): 56 data bytes
64 bytes from 192.168.0.2: icmp_seq=0 ttl=64 time=21.894 ms
64 bytes from 192.168.0.2: icmp_seq=1 ttl=64 time=22.159 ms

Success! We have reachability across the service.

Conclusion

IP Infusion is continuing to build their evpn/mpls deployment as well as segment routing. It is exciting to see these feature sets continue to mature as traditional LDP/VPLS deployments move to EVPN/MPLS. If you need assistance on the transition from LDP to segment routing or VPLS to EVPN reach out to IP Architechs.

MikroTik – ROSv7 – VPLS over IPv6 MPLS with LDPv6


The great migration

IPv6 adoption has really picked up in the last 12 months and MikroTik RouterOSv7 development is no exception. Dual stack networks are still the most common and easiest to initially deploy for carriers.

However, single stack networks with IPv4 as a service overlay are definitely on the horizon for MikroTik users now that MPLS can operate purely on IPv6.

Single stack networks are easier and cheaper to operate in the long run and are a natural evolution of dual stack networks as we begin to turn IPv4 off for underlay infrastructure.

LDPv6

There are a few different ways to distribute labels in IPv6 MPLS. SR-MPLS (less common and usually with IS-IS) and SRv6 are the other options besides LDPv6.

While I generally am in favor of SR-MPLS/SRv6 long term due to the protocol simplification and traffic management capabilities, having an IPv6 MPLS stack is a great starting point for MikroTik.

LDPv6 is defined by RFC 7552 and is fairly recent as it finalized in 2015. It generally operates in much the same way as LDPv4.


Use Cases

The most common use case among MikroTik users is more efficient subnetting of IPv4 and directly replaces LDPv4 for this task.

This is the use case tested in this lab by putting 203.0.113.0/24 onto a VPLS interface as an overlay transported by LDPv6.

Other use cases include: Private L2 transport, IPv6 overlay (common when used with v4 overlay to the same subscribers) and IPTV transport to simplify multicast requirements.


Lab overview

Because RouterOSv7 is so new and has hardware dependencies, I’ve opted to do a lot of the testing on actual MikroTik equipment instead of just the CHR VM inside of EVE-NG as I would normally do when testing/labbing protocols.

Hardware Offload – Officially, IPv6 HW offload isn’t listed as “done”, but some of the elements are there in the command output and the ‘H’ flag for hardware offload can be seen in the IPv4 and IPv6 routing tables.

Example of IPv4 hw offloaded routes on the CCR2116


Example of IPv6 hw offloaded routes on the CCR2116 – still not officially supported



Lab topology

The physical lab consists of a CCR2116 as an LSR and two CCR2004s acting as LERs. They are part of the larger 100G development lab so the relevant part has been highlighted on the physical drawing.


https://iparchitechs.com/contact


Lab testing

Here is an overview of the config and verification of LDPv6 operation with an IPv4 ping test between the two LER routers.


LER-01 (CCR2004-16G-2S+)

verification:


config:

# may/15/2022 12:22:03 by RouterOS 7.3beta40
# model = CCR2004-16G-2S+
/interface bridge
add name=br-vpls-101-ipv6
add name=lo-ipv6
/interface vpls
add disabled=no mac-address=02:DA:E8:D2:E5:83 name=vpls-101-ipv6 peer=200:127:1::2 pw-type=raw-ethernet vpls-id=101:6
/interface bonding
add mode=802.3ad name=bonding1 slaves=sfp-sfpplus1,sfp-sfpplus2 transmit-hash-policy=layer-3-and-4
/interface vlan
add interface=bonding1 name=vlan3100 vlan-id=3100
/routing ospf instance
add disabled=no name=ospf-instance-v3-ipv6 version=3
/routing ospf area
add disabled=no instance=ospf-instance-v3-ipv6 name=ospf-area-0-ipv6
/interface bridge port
add bridge=br-vpls-101-ipv6 interface=vpls-101-ipv6
/ip address
add address=203.0.113.1/24 interface=br-vpls-101-ipv6 network=203.0.113.0
/ipv6 address
add address=200:126:1::1 advertise=no interface=vlan3100
add address=200:127:1::1/128 advertise=no interface=lo-ipv6
add address=203:0:113::1 advertise=no interface=br-vpls-101-ipv6
/mpls interface
add disabled=no input=yes interface=vlan3100 mpls-mtu=1530
add disabled=no input=yes interface=vpls-101-ipv6
/mpls ldp
add afi=ipv6 disabled=no lsr-id=100.127.1.1 transport-addresses=200:127:1::1 vrf=main
/mpls ldp interface
add accept-dynamic-neighbors=yes afi=ipv6 disabled=no interface=vlan3100 transport-addresses=200:127:1::1
/routing ospf interface-template
add area=ospf-area-0-ipv6 disabled=no interfaces=vlan3100 type=ptp
add area=ospf-area-0-ipv6 disabled=no interfaces=lo-ipv6 passive
/system identity
set name=ccr2004-2splus-01.test.lab.ipa

LER-02 (CCR2004-16G-2S+)

verification:


config:

# may/15/2022 14:15:04 by RouterOS 7.3beta40
# model = CCR2004-16G-2S+
/interface bridge
add name=br-vpls-101
add name=lo-ipv6
/interface vpls
add disabled=no mac-address=02:87:6F:98:EE:32 name=vpls-101-ipv6 peer=200:127:1::1 pw-type=raw-ethernet vpls-id=101:6
/interface bonding
add mode=802.3ad name=bonding1 slaves=sfp-sfpplus1,sfp-sfpplus2 transmit-hash-policy=layer-3-and-4
/interface vlan
add interface=bonding1 name=vlan3200 vlan-id=3200
/routing ospf instance
add disabled=no name=ospf-instance-v3-ipv6 version=3
/routing ospf area
add disabled=no instance=ospf-instance-v3-ipv6 name=ospf-area-0-ipv6
/interface bridge port
add bridge=br-vpls-101 interface=vpls-101-ipv6
/ip address
add address=203.0.113.2/24 interface=br-vpls-101 network=203.0.113.0
add address=100.126.1.9/29 interface=vlan3200
/ipv6 address
add address=200:127:1::2/128 advertise=no interface=lo-ipv6
add address=200:126:2::2 advertise=no interface=vlan3200
add address=203:0:113::2 interface=br-vpls-101
/mpls interface
add disabled=no input=yes interface=vlan3200 mpls-mtu=1530
add disabled=no input=yes interface=vpls-101-ipv6
/mpls ldp
add afi=ipv6 disabled=no lsr-id=100.127.1.2 transport-addresses=200:127:1::2
/mpls ldp interface
add afi=ipv6 disabled=no interface=vlan3200 transport-addresses=200:127:1::2
add afi=ip disabled=yes interface=vlan3200 transport-addresses=100.127.1.2
/routing ospf interface-template
add area=ospf-area-0-ipv6 disabled=no interfaces=lo-ipv6 passive
add area=ospf-area-0-ipv6 disabled=no interfaces=vlan3200 type=ptp
/system identity
set name=ccr2004-2splus-02.test.lab.ipa

LSR-01 (CCR2116-12G-4S+)

verification:

config:

# may/15/2022 11:54:07 by RouterOS 7.3beta40
# model = CCR2116-12G-4S+

/interface bridge
add frame-types=admit-only-vlan-tagged name=bridge-main vlan-filtering=yes
add name=lo-ipv6
/interface ethernet
set [ find default-name=sfp-sfpplus1 ] auto-negotiation=no speed=10Gbps
set [ find default-name=sfp-sfpplus2 ] auto-negotiation=no speed=10Gbps
/interface vlan
add interface=bridge-main name=vlan3100 vlan-id=3100
add interface=bridge-main name=vlan3200 vlan-id=3200
/interface bonding
add mode=802.3ad name=bonding1-40g-crs326-01 slaves=sfp-sfpplus1,sfp-sfpplus2,sfp-sfpplus3,sfp-sfpplus4 transmit-hash-policy=layer-3-and-4
/interface ethernet switch
set 0 l3-hw-offloading=yes
/routing ospf instance
add disabled=no name=ospf-instance-v3-ipv6 version=3
/routing ospf area
add disabled=no instance=ospf-instance-v3-ipv6 name=ospf-area-0-ipv6
/interface bridge port
add bridge=bridge-main interface=bonding1-40g-crs326-01
/interface bridge vlan
add bridge=bridge-main tagged=bonding1-40g-crs326-01,bridge-main vlan-ids=3100
add bridge=bridge-main tagged=bonding1-40g-crs326-01,bridge-main vlan-ids=3200
/ipv6 address
add address=200:126:1::3 advertise=no interface=vlan3100
add address=200:127:1::3/128 advertise=no interface=lo-ipv6
add address=200:126:2::3 advertise=no interface=vlan3200
/mpls interface
add disabled=no input=yes interface=vlan3100 mpls-mtu=1530
add disabled=no input=yes interface=vlan3200 mpls-mtu=1530
/mpls ldp
add afi=ipv6 disabled=no lsr-id=100.127.1.3 transport-addresses=200:127:1::3 vrf=main
/mpls ldp interface
add accept-dynamic-neighbors=yes afi=ipv6 disabled=no interface=vlan3100 transport-addresses=200:127:1::3
add accept-dynamic-neighbors=yes afi=ipv6 disabled=no interface=vlan3200 transport-addresses=200:127:1::3
/routing ospf interface-template
add area=ospf-area-0-ipv6 disabled=no interfaces=vlan3100 type=ptp
add area=ospf-area-0-ipv6 disabled=no interfaces=vlan3200 type=ptp
add area=ospf-area-0-ipv6 disabled=no interfaces=lo-ipv6 passive
/system identity
set name=ccr2116-01.test.lab.ipa.net


“Why use 200::/7 for labbing instead of 2001:db8::/32?”

IPv6 is long overdue for a dedicated GUA labbing space. I’ve been working with Ed Horley (@ehorley) and several others on a dedicated range for labbing that allows for modeling not just within a /32 but beyond a /32

draft-horley-v6ops-lab-02 – Expanding the IPv6 Lab Use Space (ietf.org)