ISP Design – Building production MPLS networks with IP Infusion’s OcNOS.

Moving away from incumbent network vendors


1466540435IpInfusion interivew questions


One of the challenges service providers have faced in the last decade is lowering the cost per port or per MB while maintaining the same level of availability and service level.

And then add to that the constant pressure from subscribers to increase capacity and meet the rising demand for realtime content.

This can be an especially daunting task when routers with the feature sets ISPs need cost an absolute fortune – especially as new port speeds are released.

Whitebox, also called disaggregated networking, has started changing the rules of the game. ISPs are working to figure out how to integrate and move to production on disaggregated models to lower the cost of investing in higher speeds and feeds.

Whitebox often faces the perception problem of being more difficult to implement than traditional vendors – which is exactly why I wanted to highlight some of the work we’ve been doing at integrating whitebox into production ISP networks using IP Infusion’s OcNOS.

Things are really starting to heat up in the disaggregagted network space after the announcement by Amazon a few days ago that it intends to build and sell whitebox switches.

As I write this, I’m headed to Networking Field Day 18 where IP Infusion will be presenting and I expect whitebox will again be a hot topic.

This will be the second time IPI has presented at Networking Field Day but the first time that I’ve had a chance to see them present firsthand.

It’s especially exciting for me as I work on implementing IPI on a regular basis and integrating OcNOS into client networks.


What is OcNOS?


IP Infusion has been making network operating systems (NOS) for more than 20 years under the banner of its whitelabel NOS – ZebOS.

As disaggregated networking started to become popular, IPI created OcNOS which is an ONIE compatible NOS using elements and experience from 20 years of software development with ZebOS.

There is a great overview of OcNOS from Networking Field Day 15 here:


What does a production OcNOS based MPLS network look like?

Here is an overview of the EVE-NG lab we built based on an actual implementation.



Use case – Building an MPLS core to deliver L2 overlay services

Although certainly not a new use case or implementation, MPLS and VPLS are very expensive to deploy using major vendors and are still a fundamental requirement for most ISPs.

This is where IPI really shines as they have feature sets like MPLS FRR, TE and the newer Segment Routing for OSPF and IS-IS that can be used in a platform that is significantly cheaper than incumbent network vendors.

The cost difference is so large that often ISPs are able to buy switches with a higher overall port speeds than they could from a major vendor. This in turn creates a significant competitive advantage as ISPs can take the same budget (or less) and roll out 100 gig instead of 10 gig – as an example

Unlike enterprise networks, cost is more consistently a significant driver when selecting network equipment for ISPs. This is especially true for startup ISPs that may be limited in the amount of capital that can be spent in a service area to keep ROI numbers relatively sane for investors.

Lab Overview

In the lab (and production) network we have above, OcNOS is deployed as the MPLS core at each data center and MikroTik routers are being used as MPLS PE routers.

VPLS is being run from one DC to the other and delivered via the PE routers to the end hosts.

Because the port density on whitebox switches is so high compared to a traditional aggregation router, we could even use LACP channels if dark fiber was available to increase the transport bandwidth between the DCs without a significant monetary impact on the cost of the deployment.

The type of switches that you’d use in production depend greatly on the speeds and feeds required, but for startup ISPs, we’ve had lots of success with Dell 4048s and Edge-Core 5812.

How hard is it to configure and deploy?

It’s not hard at all!

If you know how to use the up and down arrow keys in the bootloader and TFTP/FTP to load an image onto a piece of network hardware, you’re halfway there!

Here is a screenshot of the GRUB bootloader for an ONIE switch (this is a Dell) where you select which OS to boot the switch into


The configuration is relatively straightforward as well if you’re familiar with industry standard Command Line Interfaces (CLI).

While this lab was configured in a more traditional way using a terminal session to paste commands in, OcNOS can easily be orchestrated and automated using tools like Ansible (also presenting at Networking Field Day 18) or protocols like NETCONF as well as a REST API.

Lab configs

I’ve included the configs from the lab in order to give engineers a better idea of what OcNOS actually looks like for a production deployment.










MikroTik PE-1



 MikroTik PE-2





Preview: Networking Field Day Exclusive with Aruba (HPE) – The 8400 core switch



Back to Silicon Valley!

As a network type, it’s hard not to be excited when heading to a Networking Field Day event. I joined then NFD club by attending NFD14 and have been hooked ever since.

Not only is it an honor and a privilege to be invited to an NFD event, the personal relationships that are forged in the larger TFD community are some of the most valuable I’ve ever had in my career.

This go around we’ll be visiting Aruba (A Hewlett Packard Enterprise Company) in Santa Clara to deep dive on the newest addition to the Aruba product line – the 8400 core switch.

A new face in campus town – the Aruba 8400

It’s been a while since anything exciting happened in the world of campus networking. It’s a steady segment for most vendors but nothing disruptive has really happened in the last few years.

And that’s not incredibly surprising. For better or worse, as long as campus networks aren’t broken in most enterprises, they are often neglected in favor of the data center and cloudy pursuits.

Aruba is touting the 8400 to increase automation and visibility in the campus core – both are areas that network engineering teams have traditionally struggled to implement.

Couple that with a brand new API enabled NOS that has built-in analytics and Aruba may have a serious claim on the ‘game changing’ campaign it has been running since announcing the 8400 in June 2017.

The 8400 quick specs:
  • 8 slot chassis (for linecards)
  • Provides up to 19.2 Tbps switching capacity (8.571 billion packets per second)
  • Supports a maximum of 256 10GbE (SFP/SFP+) ports, or 64 40GbE (QSFP+) ports, or 48 ports 40/100GbE (QSFP28) combination
  • Full 8400 data sheet is here

First impressions:

What I like

Speeds/Port Density – The speed/port density specs for the 8400 read more like a data center switch than a campus core which means even the largest campus networks will have plenty of available ports with up to 100 gig if needed.

Security – Encryption at wire speed is becoming more and more of an issue as new security and compliance requirements force network teams to treat private links that were previously trusted as untrusted. The availability of MACSEC on linecards is big plus.

Automation – In reading the product literature, one of the differentiating factors listed is the ability to automate manual tasks like the provisioning of network switches to support wireless access points. This is a task that can be fairly daunting in a network with a large number of switches and no automation. I’ll be interested to see how the ‘zero touch’ provisioning for APs that Aruba describes actually works.

Visibility/Troubleshooting – Enhanced visibility and troubleshooting tools are a welcome feature for any engineering team. Aruba has developed a Network Analytics Engine that is listed as being at the heart of this set of capabilities. Onboard network analysis modules have been tried before by other vendors with varying degrees of success and so it will be interesting to see what Aruba’s take is on built in analytics.

Virtual Switching Framework – As a designer of networks, i’m a big fan of leveraging link aggregation in my designs for path redundancy coupled with switches than can support multi-chassis LACP.  The 8400 supports Aruba’s Virtual Switching Framework which allows both chassis to work together in similar fashion to a switch stack which allows for a single LACP channel to contain links in two different chassis. While this isn’t a groundbreaking feature, it’s critical to competing in the campus core market.

Complete REST API – Aruba describes the REST based API in this blog post as having access to “every network function and state, both persistent and ephemeral, within the switch.” This opens up a world of possibilities for integration and automation into enterprise applications as well as automation/orchestration engines.

Initial questions I have for Aruba:

Code maturity – The Aruba OS-CX network operating system seems to be the heart and soul of the new switch. As with any new NOS, one of my first questions is around interop and bug testing. What interop testing has been done and what are the results from current field deployments?

Software licensing and support – Software/feature licensing and support can be a source of frustration fro enterprise clients. Understanding the software and support model that Aruba uses will be one of the initial questions that I have.

Depth of the L3 feature set – As much as we try to avoid complexity in the core, sometimes advanced features in OSPF and BGP are needed such as dynamic routing within a VRF or a complex set of REGEX values to build a route map for a BGP peer. One of my goals in attending this NFD is to better understand the capabilities of Aruba’s routing stack in OS-CX.

To disaggregate or not

Often the opinions we have on new technology are shaped by our daily work. As someone who is frequently engaged in whitebox integration, disaggregation has become more and more prevalant in my daily work.

I suspect the decision by Aruba to use a chassis to offer port density rather than a disaggregated leaf/spine architecture stems from the lack of demand by enterprises to use leaf/spine in the campus.

Chassis is what everyone is comfortable with and expects to implement when designing the campus core. As such, nobody in the world of disaggregated networking has taken aim at the campus from a software standpoint.

That said, it will be interesting to see if a small leaf/spine core is considered for future hardware iterations of the OS-CX family aimed at campus deployments.

More to come!

As I write this, I’m enroute to #NFDx and am looking forward to the presentation by Aruba so that we can deep dive and really understand what makes the 8400 tick and the problems Aruba is trying to solve.

Please tag me @stubarea51 on twitter with the #NFDx hashtag if you have questions you’d like to ask Aruba about the 8400.

Stay tuned!