ISP Design – Building production MPLS networks with IP Infusion’s OcNOS.

Moving away from incumbent network vendors

 

1466540435IpInfusion interivew questions

 

One of the challenges service providers have faced in the last decade is lowering the cost per port or per MB while maintaining the same level of availability and service level.

And then add to that the constant pressure from subscribers to increase capacity and meet the rising demand for realtime content.

This can be an especially daunting task when routers with the feature sets ISPs need cost an absolute fortune – especially as new port speeds are released.
whitebox-switch_500px-wide

Whitebox, also called disaggregated networking, has started changing the rules of the game. ISPs are working to figure out how to integrate and move to production on disaggregated models to lower the cost of investing in higher speeds and feeds.

Whitebox often faces the perception problem of being more difficult to implement than traditional vendors – which is exactly why I wanted to highlight some of the work we’ve been doing at iparchitechs.com integrating whitebox into production ISP networks using IP Infusion’s OcNOS.

Things are really starting to heat up in the disaggregagted network space after the announcement by Amazon a few days ago that it intends to build and sell whitebox switches.

As I write this, I’m headed to Networking Field Day 18 where IP Infusion will be presenting and I expect whitebox will again be a hot topic.

This will be the second time IPI has presented at Networking Field Day but the first time that I’ve had a chance to see them present firsthand.

It’s especially exciting for me as I work on implementing IPI on a regular basis and integrating OcNOS into client networks.

 

What is OcNOS?

ip-ocnos-main-1

IP Infusion has been making network operating systems (NOS) for more than 20 years under the banner of its whitelabel NOS – ZebOS.

As disaggregated networking started to become popular, IPI created OcNOS which is an ONIE compatible NOS using elements and experience from 20 years of software development with ZebOS.

There is a great overview of OcNOS from Networking Field Day 15 here:

 

What does a production OcNOS based MPLS network look like?

Here is an overview of the EVE-NG lab we built based on an actual implementation.

 

IPI-VPLS-2

Use case – Building an MPLS core to deliver L2 overlay services

Although certainly not a new use case or implementation, MPLS and VPLS are very expensive to deploy using major vendors and are still a fundamental requirement for most ISPs.

This is where IPI really shines as they have feature sets like MPLS FRR, TE and the newer Segment Routing for OSPF and IS-IS that can be used in a platform that is significantly cheaper than incumbent network vendors.

The cost difference is so large that often ISPs are able to buy switches with a higher overall port speeds than they could from a major vendor. This in turn creates a significant competitive advantage as ISPs can take the same budget (or less) and roll out 100 gig instead of 10 gig – as an example

Unlike enterprise networks, cost is more consistently a significant driver when selecting network equipment for ISPs. This is especially true for startup ISPs that may be limited in the amount of capital that can be spent in a service area to keep ROI numbers relatively sane for investors.

Lab Overview

In the lab (and production) network we have above, OcNOS is deployed as the MPLS core at each data center and MikroTik routers are being used as MPLS PE routers.

VPLS is being run from one DC to the other and delivered via the PE routers to the end hosts.

Because the port density on whitebox switches is so high compared to a traditional aggregation router, we could even use LACP channels if dark fiber was available to increase the transport bandwidth between the DCs without a significant monetary impact on the cost of the deployment.

The type of switches that you’d use in production depend greatly on the speeds and feeds required, but for startup ISPs, we’ve had lots of success with Dell 4048s and Edge-Core 5812.


How hard is it to configure and deploy?

It’s not hard at all!

If you know how to use the up and down arrow keys in the bootloader and TFTP/FTP to load an image onto a piece of network hardware, you’re halfway there!

Here is a screenshot of the GRUB bootloader for an ONIE switch (this is a Dell) where you select which OS to boot the switch into

ONIE GRUB

The configuration is relatively straightforward as well if you’re familiar with industry standard Command Line Interfaces (CLI).

While this lab was configured in a more traditional way using a terminal session to paste commands in, OcNOS can easily be orchestrated and automated using tools like Ansible (also presenting at Networking Field Day 18) or protocols like NETCONF as well as a REST API.

Lab configs

I’ve included the configs from the lab in order to give engineers a better idea of what OcNOS actually looks like for a production deployment.

IPI-MPLS-1

 

!
!Last configuration change at 12:24:27 EDT Tue Jul 17 2018 by ocnos
!
no service password-encryption
!
hostname IPI-MPLS-1
!
logging monitor 7
!
ip vrf management
!
mpls lsp-model uniform
mpls propagate-ttl
!
ip domain-lookup
spanning-tree mode provider-rstp
data-center-bridging enable
feature telnet
feature ssh
snmp-server enable snmp
snmp-server view all .1 included
ntp enable
username ocnos role network-admin password encrypted $1$HJDzvHS1$.4/PPuAmCUEwEhs
UWeYqo0
!
ip pim register-rp-reachability
!
router ldp
 router-id 100.127.0.1
!
interface lo
 mtu 65536
 ip address 127.0.0.1/8
 ip address 100.127.0.1/32 secondary
 ipv6 address ::1/128
!
interface eth0
 ip address 100.64.0.1/29
 label-switching
 enable-ldp ipv4
!
interface eth1
 ip address 100.64.0.9/29
 label-switching
 enable-ldp ipv4
!
interface eth2
 ip address 100.64.1.1/29
 label-switching
 enable-ldp ipv4
!
interface eth3
!
interface eth4
!
interface eth5
!
interface eth6
!
interface eth7
!
router ospf 1
 ospf router-id 100.127.0.1
 network 100.64.0.0/29 area 0.0.0.0
 network 100.64.0.8/29 area 0.0.0.0
 network 100.64.1.0/29 area 0.0.0.0
 network 100.127.0.1/32 area 0.0.0.0
 cspf disable-better-protection
!
bgp extended-asn-cap
!
router bgp 8675309
 bgp router-id 100.127.0.1
 neighbor 100.127.0.3 remote-as 8675309
 neighbor 100.127.0.3 update-source lo
 neighbor 100.127.2.1 remote-as 8675309
 neighbor 100.127.2.1 update-source lo
 neighbor 100.127.2.1 route-reflector-client
 neighbor 100.127.0.4 remote-as 8675309
 neighbor 100.127.0.4 update-source lo
 neighbor 100.127.0.4 route-reflector-client
 neighbor 100.127.0.2 remote-as 8675309
 neighbor 100.127.0.2 update-source lo
 neighbor 100.127.0.2 route-reflector-client
 neighbor 100.127.1.1 remote-as 8675309
 neighbor 100.127.1.1 update-source lo
 neighbor 100.127.1.1 route-reflector-client
!
line con 0
 login
line vty 0 39
 login
!
end

IPI-MPLS-2

 

!
!Last configuration change at 12:23:31 EDT Tue Jul 17 2018 by ocnos
!
no service password-encryption
!
hostname IPI-MPLS-2
!
logging monitor 7
!
ip vrf management
!
mpls lsp-model uniform
mpls propagate-ttl
!
ip domain-lookup
spanning-tree mode provider-rstp
data-center-bridging enable
feature telnet
feature ssh
snmp-server enable snmp
snmp-server view all .1 included
ntp enable
username ocnos role network-admin password encrypted $1$RWk6XAN.$6H0GXBR9ad8eJE2
7nRUfu1
!
ip pim register-rp-reachability
!
router ldp
 router-id 100.127.0.2
!
interface lo
 mtu 65536
 ip address 127.0.0.1/8
 ip address 100.127.0.2/32 secondary
 ipv6 address ::1/128
!
interface eth0
 ip address 100.64.0.2/29
 label-switching
 enable-ldp ipv4
!
interface eth1
 ip address 100.64.0.17/29
 label-switching
 enable-ldp ipv4
!
interface eth2
 ip address 100.64.1.9/29
 label-switching
 enable-ldp ipv4
!
interface eth3
!
interface eth4
!
interface eth5
!
interface eth6
!
interface eth7
!
router ospf 1
 network 100.64.0.0/29 area 0.0.0.0
 network 100.64.0.16/29 area 0.0.0.0
 network 100.64.1.8/29 area 0.0.0.0
 network 100.127.0.2/32 area 0.0.0.0
 cspf disable-better-protection
!
bgp extended-asn-cap
!
router bgp 8675309
 bgp router-id 100.127.0.2
 neighbor 100.127.0.3 remote-as 8675309
 neighbor 100.127.0.3 update-source lo
 neighbor 100.127.0.1 remote-as 8675309
 neighbor 100.127.0.1 update-source lo
!
line con 0
 login
line vty 0 39
 login
!
end

IPI-MPLS-3

 

!
!Last configuration change at 12:25:11 EDT Tue Jul 17 2018 by ocnos
!
no service password-encryption
!
hostname IPI-MPLS-3
!
logging monitor 7
!
ip vrf management
!
mpls lsp-model uniform
mpls propagate-ttl
!
ip domain-lookup
spanning-tree mode provider-rstp
data-center-bridging enable
feature telnet
feature ssh
snmp-server enable snmp
snmp-server view all .1 included
ntp enable
username ocnos role network-admin password encrypted $1$gc9xYbW/$JlCDmgAEzcCmz77
QwmJW/1
!
ip pim register-rp-reachability
!
router ldp
 router-id 100.127.0.3
!
interface lo
 mtu 65536
 ip address 127.0.0.1/8
 ip address 100.127.0.3/32 secondary
 ipv6 address ::1/128
!
interface eth0
 ip address 100.64.0.25/29
 label-switching
 enable-ldp ipv4
!
interface eth1
 ip address 100.64.0.10/29
 label-switching
 enable-ldp ipv4
!
interface eth2
 ip address 100.64.2.1/29
 label-switching
 enable-ldp ipv4
!
interface eth3
!
interface eth4
!
interface eth5
!
interface eth6
!
interface eth7
!
router ospf 1
 ospf router-id 100.127.0.3
 network 100.64.0.8/29 area 0.0.0.0
 network 100.64.0.24/29 area 0.0.0.0
 network 100.64.2.0/29 area 0.0.0.0
 network 100.127.0.3/32 area 0.0.0.0
 cspf disable-better-protection
!
bgp extended-asn-cap
!
router bgp 8675309
 bgp router-id 100.127.0.3
 neighbor 100.127.0.1 remote-as 8675309
 neighbor 100.127.0.1 update-source lo
 neighbor 100.127.2.1 remote-as 8675309
 neighbor 100.127.2.1 update-source lo
 neighbor 100.127.2.1 route-reflector-client
 neighbor 100.127.0.4 remote-as 8675309
 neighbor 100.127.0.4 update-source lo
 neighbor 100.127.0.4 route-reflector-client
 neighbor 100.127.0.2 remote-as 8675309
 neighbor 100.127.0.2 update-source lo
 neighbor 100.127.0.2 route-reflector-client
 neighbor 100.127.1.1 remote-as 8675309
 neighbor 100.127.1.1 update-source lo
 neighbor 100.127.1.1 route-reflector-client
!
line con 0
 login
line vty 0 39
 login
!
end

IPI-MPLS-4

 

!
!Last configuration change at 12:24:49 EDT Tue Jul 17 2018 by ocnos
!
no service password-encryption
!
hostname IPI-MPLS-4
!
logging monitor 7
!
ip vrf management
!
mpls lsp-model uniform
mpls propagate-ttl
!
ip domain-lookup
spanning-tree mode provider-rstp
data-center-bridging enable
feature telnet
feature ssh
snmp-server enable snmp
snmp-server view all .1 included
ntp enable
username ocnos role network-admin password encrypted $1$6OP7UdH/$RaIxCBOGxHIt1Ao
IUyPks/
!
ip pim register-rp-reachability
!
router ldp
 router-id 100.127.0.4
!
interface lo
 mtu 65536
 ip address 127.0.0.1/8
 ip address 100.127.0.4/32 secondary
 ipv6 address ::1/128
!
interface eth0
 ip address 100.64.0.26/29
 label-switching
 enable-ldp ipv4
!
interface eth1
 ip address 100.64.0.18/29
 label-switching
 enable-ldp ipv4
!
interface eth2
 ip address 100.64.2.9/29
 label-switching
 enable-ldp ipv4
!
interface eth3
!
interface eth4
!
interface eth5
!
interface eth6
!
interface eth7
!
router ospf 1
 ospf router-id 100.127.0.4
 network 100.64.0.16/29 area 0.0.0.0
 network 100.64.0.24/29 area 0.0.0.0
 network 100.64.2.8/29 area 0.0.0.0
 network 100.127.0.4/32 area 0.0.0.0
 cspf disable-better-protection
!
bgp extended-asn-cap
!
router bgp 8675309
 bgp router-id 100.127.0.4
 neighbor 100.127.0.3 remote-as 8675309
 neighbor 100.127.0.3 update-source lo
 neighbor 100.127.0.1 remote-as 8675309
 neighbor 100.127.0.1 update-source lo
!
line con 0
 login
line vty 0 39
 login
!
end

 

MikroTik PE-1

 

# jul/17/2018 17:33:30 by RouterOS 6.38.7
# software id =
#
/interface bridge
add name=Lo0
add name=bridge-vpls-777
/interface vpls
add disabled=no l2mtu=1500 mac-address=02:BF:0A:4A:55:D0 name=vpls777 
    pw-type=tagged-ethernet remote-peer=100.127.2.1 vpls-id=8675309:777
/interface vlan
add interface=vpls777 name=vlan777 vlan-id=777
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/routing bgp instance
set default as=8675309 router-id=100.127.1.1
/routing ospf instance
set [ find default=yes ] router-id=100.127.1.1
/interface bridge port
add bridge=bridge-vpls-777 interface=ether3
add bridge=bridge-vpls-777 interface=vlan777
/ip address
add address=100.64.1.2/29 interface=ether1 network=100.64.1.0
add address=100.127.1.1 interface=Lo0 network=100.127.1.1
add address=100.64.1.10/29 interface=ether2 network=100.64.1.8
/ip dhcp-client
add disabled=no interface=ether4
/mpls ldp
set enabled=yes lsr-id=100.127.1.1 transport-address=100.127.1.1
/mpls ldp interface
add interface=ether1 transport-address=100.127.1.1
add interface=ether2 transport-address=100.127.1.1
/routing bgp peer
add name=IPI-MPLS-1 remote-address=100.127.0.1 remote-as=8675309 
    update-source=Lo0
add name=IPI-MPLS-3 remote-address=100.127.0.3 remote-as=8675309 
    update-source=Lo0
/routing ospf network
add area=backbone network=100.64.1.0/29
add area=backbone network=100.64.1.8/29
add area=backbone network=100.127.1.1/32
/system identity
set name=MIkroTik-PE1
/tool romon
set enabled=yes

 

 MikroTik PE-2

 

# jul/17/2018 17:34:23 by RouterOS 6.38.7
# software id =
#
/interface bridge
add name=Lo0
add name=bridge-vpls-777
/interface vpls
add disabled=no l2mtu=1500 mac-address=02:E2:86:F2:23:21 name=vpls777 pw-type=tagged-ethernet remote-peer=100.127.1.1 vpls-id=8675309:777
/interface vlan
add interface=vpls777 name=vlan777 vlan-id=777
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/routing bgp instance
set default as=8675309 router-id=100.127.2.1
/routing ospf instance
set [ find default=yes ] router-id=100.127.2.1
/interface bridge port
add bridge=bridge-vpls-777 interface=ether3
add bridge=bridge-vpls-777 interface=vlan777
/ip address
add address=100.64.2.2/29 interface=ether1 network=100.64.2.0
add address=100.127.2.1 interface=Lo0 network=100.127.2.1
add address=100.64.2.10/29 interface=ether2 network=100.64.2.8
/ip dhcp-client
add disabled=no interface=ether1
/mpls ldp
set enabled=yes lsr-id=100.127.2.1 transport-address=100.127.2.1
/mpls ldp interface
add interface=ether1 transport-address=100.127.2.1
add interface=ether2 transport-address=100.127.2.1
/routing bgp peer
add name=IPI-MPLS-1 remote-address=100.127.0.1 remote-as=8675309 update-source=Lo0
add name=IPI-MPLS-3 remote-address=100.127.0.3 remote-as=8675309 update-source=Lo0
/routing ospf network
add area=backbone network=100.64.2.0/29
add area=backbone network=100.64.2.8/29
add area=backbone network=100.127.2.1/32
/system identity
set name=MIkroTik-PE2
/tool bandwidth-server
set authenticate=no
/tool romon
set enabled=yes

 

 

 

Preview: Networking Field Day Exclusive with Aruba (HPE) – The 8400 core switch

 

aruba-game-has-changed

Back to Silicon Valley!

As a network type, it’s hard not to be excited when heading to a Networking Field Day event. I joined then NFD club by attending NFD14 and have been hooked ever since.

Not only is it an honor and a privilege to be invited to an NFD event, the personal relationships that are forged in the larger TFD community are some of the most valuable I’ve ever had in my career.

This go around we’ll be visiting Aruba (A Hewlett Packard Enterprise Company) in Santa Clara to deep dive on the newest addition to the Aruba product line – the 8400 core switch.

A new face in campus town – the Aruba 8400

It’s been a while since anything exciting happened in the world of campus networking. It’s a steady segment for most vendors but nothing disruptive has really happened in the last few years.

And that’s not incredibly surprising. For better or worse, as long as campus networks aren’t broken in most enterprises, they are often neglected in favor of the data center and cloudy pursuits.

Aruba is touting the 8400 to increase automation and visibility in the campus core – both are areas that network engineering teams have traditionally struggled to implement.

Couple that with a brand new API enabled NOS that has built-in analytics and Aruba may have a serious claim on the ‘game changing’ campaign it has been running since announcing the 8400 in June 2017.

The 8400 quick specs:
  • 8 slot chassis (for linecards)
  • Provides up to 19.2 Tbps switching capacity (8.571 billion packets per second)
  • Supports a maximum of 256 10GbE (SFP/SFP+) ports, or 64 40GbE (QSFP+) ports, or 48 ports 40/100GbE (QSFP28) combination
  • Full 8400 data sheet is here

First impressions:

What I like

Speeds/Port Density – The speed/port density specs for the 8400 read more like a data center switch than a campus core which means even the largest campus networks will have plenty of available ports with up to 100 gig if needed.

Security – Encryption at wire speed is becoming more and more of an issue as new security and compliance requirements force network teams to treat private links that were previously trusted as untrusted. The availability of MACSEC on linecards is big plus.

Automation – In reading the product literature, one of the differentiating factors listed is the ability to automate manual tasks like the provisioning of network switches to support wireless access points. This is a task that can be fairly daunting in a network with a large number of switches and no automation. I’ll be interested to see how the ‘zero touch’ provisioning for APs that Aruba describes actually works.

Visibility/Troubleshooting – Enhanced visibility and troubleshooting tools are a welcome feature for any engineering team. Aruba has developed a Network Analytics Engine that is listed as being at the heart of this set of capabilities. Onboard network analysis modules have been tried before by other vendors with varying degrees of success and so it will be interesting to see what Aruba’s take is on built in analytics.

Virtual Switching Framework – As a designer of networks, i’m a big fan of leveraging link aggregation in my designs for path redundancy coupled with switches than can support multi-chassis LACP.  The 8400 supports Aruba’s Virtual Switching Framework which allows both chassis to work together in similar fashion to a switch stack which allows for a single LACP channel to contain links in two different chassis. While this isn’t a groundbreaking feature, it’s critical to competing in the campus core market.

Complete REST API – Aruba describes the REST based API in this blog post as having access to “every network function and state, both persistent and ephemeral, within the switch.” This opens up a world of possibilities for integration and automation into enterprise applications as well as automation/orchestration engines.

Initial questions I have for Aruba:

Code maturity – The Aruba OS-CX network operating system seems to be the heart and soul of the new switch. As with any new NOS, one of my first questions is around interop and bug testing. What interop testing has been done and what are the results from current field deployments?

Software licensing and support – Software/feature licensing and support can be a source of frustration fro enterprise clients. Understanding the software and support model that Aruba uses will be one of the initial questions that I have.

Depth of the L3 feature set – As much as we try to avoid complexity in the core, sometimes advanced features in OSPF and BGP are needed such as dynamic routing within a VRF or a complex set of REGEX values to build a route map for a BGP peer. One of my goals in attending this NFD is to better understand the capabilities of Aruba’s routing stack in OS-CX.

To disaggregate or not

Often the opinions we have on new technology are shaped by our daily work. As someone who is frequently engaged in whitebox integration, disaggregation has become more and more prevalant in my daily work.

I suspect the decision by Aruba to use a chassis to offer port density rather than a disaggregated leaf/spine architecture stems from the lack of demand by enterprises to use leaf/spine in the campus.

Chassis is what everyone is comfortable with and expects to implement when designing the campus core. As such, nobody in the world of disaggregated networking has taken aim at the campus from a software standpoint.

That said, it will be interesting to see if a small leaf/spine core is considered for future hardware iterations of the OS-CX family aimed at campus deployments.

More to come!

As I write this, I’m enroute to #NFDx and am looking forward to the presentation by Aruba so that we can deep dive and really understand what makes the 8400 tick and the problems Aruba is trying to solve.

Please tag me @stubarea51 on twitter with the #NFDx hashtag if you have questions you’d like to ask Aruba about the 8400.

Stay tuned!