Doing this design allow us to build a full site with redundancy and all features being virtual except the physical cables in to the SWC switch layer. But everything else happens between virtual routers with virtual recources allowing us on the fly recource allocation and then the L2 is VxLAN private tunneling in the XCP-NG SDN controller.

The future is bright for all of us smaller ISPs with one or two gen older hardware with refurb warranty for virtualization of routers and players like mikrotik bringing the L2 100GbE to us smaller players at very good prices!

Great write up and thanks for reinforcing my decision on our design as the right one.

Same here. Seems like ROS6 and 7 are not compatible!!! Huge also failure with a border router upgraded from 6 to 7 and cisco neighbors when it comes to IPv6. Seems like compatibility of ROS7 is not good even with MT ROS6. I hope they fix this soon as new routers can only work with ROS7 and can not be downgraded to 6.

Would actual hardware be expected to handle the packet fragmenting better, ideally at least a few gbps before adding encryption?

Is there a possibility of have the Configs for routers and Switches in lab for to see results in a lab environment please?

best regards!
TWR

How did add the second nic ?

How to setup ROS7 to share received ibgp routes to Ros6 routers.

any help on this is really appreciated.

There is no default gateway needed here because his servers are all on the same network, so its layer 2 which is what vlxan is for. Gateways are only needed if you need to reach a destination outside your network, eg routed. If his example included that, then the gateways would be at your leaf.

I’ve run into this issue as well and have commented about it in this thread (which I believe is yours)

https://forum.mikrotik.com/viewtopic.php?t=180876

we have seen this issue as well. What version and platform are you running? Do you have a PCAP you can provide

Thanks for sharing this example configuration and comparison with other equipment

If I want to import 2.5 million routing tables, do I need to establish 5 EBGP?

I advised them several times (albeit unsolicited), via forum and support tickets, the merits of adopting an open source routing engine; but I’m just the little guy, and it’s fallen on deaf ears. It doesn’t make any business sense what they’re doing. From a technical perspective, the only thing that explains it is sunk-cost fallacy and perhaps a few of the internal developers have outsized political clout.

Maybe I’m wrong; but even if I am, it’s still disappointing.

thx M

I was able to add second nic in the int file in networks. All works well – Thank You!

Congratulations to all involved in that project.
I have 5 openwrt on VirtualBox and 2 IPv4-BGP-Global-Internet-Table-VM-v1 running different tables, all working like a charm.

I’m learning a lot, thank you for great help to community.

Regards from Brazil.

Should interface “ether2” and “vlan777” be added to bridge vpls-bridge-vlan-777 instead of Lo0?

We have core router in data center and a /24 public .
We have multiple locations via different NLD links with unique vlan id.
We are using /30 for each location.

So can we just use vpls over vlan and bridge the vpls tunnels together at data center.and use /24.

This because of RouterOS in virtual, so you cant set MTU 1500.

Cheers,
Adam

interface GigabitEthernet5/0/4
switchport trunk encapsulation dot1q
switchport mode trunk
switchport trunk native vlan 100
switchport trunk allowed vlan 100,200

end

@Neal What else would you run inside of VLANs?

Hi Poyozon.

It means using your command “wget data.ris.ripe.net/rrc00/latest-bview.gz” can make this .ova update with latest routing table?

Thanks

OSPF uses much more CPU than other routing protocols, and in the case of having PPPoE servers you should perhaps consider going BGP on that MIKROTIK.

Would that affect the MTU settings also on v6.47 and above or it’s been taken care of? thanks again.

How will one configure vlan on BGP based VPLS, since the VPLS is added dynamically in this config scenario?

Set the pseduowire type to tagged and bridge it to a physical interface.

wget data.ris.ripe.net/rrc00/latest-bview.gz

Is VXLAN would be the swiss knife in front of L3-MPLS / EVPN and current WAN-based enterprises connectivities ?
Does it have chance to be really largely adopted ?

Thanks

> The initial release of VxLAN is based on unicast and multicast to deliver Layer 2 frames.

I can’t see how to configure that, is it possible, or is it only multicast at present?

It can handle BUM traffic in more efficient way (encapsulated inside IP multicast) instead of copying such traffic multiple times to other endpoints. It requires properly working multicast routing to work. Unfortunately it doesn’t solve CE multihoming issues as pure VXLAN is still based on hardware mac learning on VTEPs. Such problems (and many others like anycast gateways or LACP distributed across IP fabric) can be solved if used in conjunction with EVPN based control plane. I hope mikrotik guys are working on this.

We’re an ISP based in Australia and use primarily Microtik for our routing – we’ve recently identified what appear to be some hard limits on the throughput of Microtik CHRs (about 2M pps). We were looking forward to your presentation at MUM Europe regarding breaking the 100Gbps barrier with CHRs and were wondering if you’d be presenting in a different format or be open to having a call to discuss at some stage?

We have alternatives with TNSR or customer DPDK deployments, but would prefer to stick with a Microtik platform if possible.

I believe my colleague Philip Loenneker may also have reached out to you on LinkedIn.

Thanks,

Brett Henderson | Manager – Cloud Platform | TasmaNet
40-50 Innovation Drive, Tas 7010, Australia
M: +61 466 150 343 | Email: brett.henderson@tasmanet.com.au
http://www.tasmanet.com.au

VxLAN solves a number of scale and loop avoidance issues that VPLS has. It also does not require LDP or other MPLS signalling and can work over IP.

Also, you can use a 1500 byte MTU, you just have to adjust the IP MTU on the transit links. I was building the lab quickly and just lowered the tunnel but you can certainly use a larger MTU if the equipment supports it.

sorry, ment vpls. not vtp, too late in the evening. 😉

The mtu size seems even worse. ;-/

In this case, the VPC has a statically assigned address but DHCP could be used.

Unfortunately, there is no equivalent command at this time

in R3-Tower-1, is ether1 getting dynamic IP address from the VPC?

Thanks!

Great Work – keep it up.

force ipsec to use aes encryption, because it has a hardware acceleration on ccr devices

It would be nice if the article was updated to mention this since your tests show up in searches and it seems people are having issues reproducing this outside of a lab setting.

http://forum.mikrotik.com/viewtopic.php?f=1&t=112545

Great post, thank you very much!!!

I’m building network with similar setup, could you please tell me what Intel SFP+ did you use for Intel x520-DA2 10 Gbps PCI-E NIC?

Regards,
Igor

Hi,

Is there any updates with that kind of test?

Im really interested,.how this rojter can handle ddos (udp,.syn flood etc).

vi bgp_LAB1.sh (bgp_LAB2.sh etc)
————-
#!/bin/bash

echo “Starting the service”
cd bgp
./bgp_simple.pl -myas 1234 -myip 3.3.3.1 -peerip 3.3.3.2 -peeras 4321 -p ISP5-NorthAmerica-NewYork-Jan-2016

————-

Make it executable.

#chmod u+x bgp_LAB1.sh &1

Run it and send the process to the background. I needed to disown and exit to keep it from jumping back to my screen. Don’t know why it kept doing that for me.

#./bgp_LAB1.sh &1

#disown -h
#exit

I bet someone with real perl knowledge could figure out making it multi-core.

http://stackoverflow.com/questions/1961682/how-can-i-make-my-perl-script-use-multiple-cores-for-child-processes

You need put command “sudo” before.

haha, April fools!

You’re welcome Akshay! Glad it was helpful for you

Nothing is advertised.
Please advise on this

Re Intel X520, it is possible to use unsupported, eg Mikrotik SFPs,

echo “options ixgbe allow_unsupported_sfp=1,1” > /etc/modprobe.d/ixgbe-options.conf
depmod -a
update-initramfs -u

I can’t find original link that led me to it but it does work.

Also, have you tried ProxMox? I’ve moved away from ESXi completely, it’s got brilliant features and way more options…

I probably should post an update with some tips that I’ve discovered after working with this VM for a while. You are correct that if you SSH into the VM and issue the commands in a terminal session versus running it in the VM console window, the table usually loads in 15 to 20 minutes instead of hours. This appears to be exactly as you described – a consequence of displaying over 500,000 prefixes in the console window. Also, be sure to set your BGP peering timers to 600 keepalive and 1800 hold to avoid the peering going down unexpectedly. If you want to be able to test how quickly a router can take a full table in, I typically peer a VM or another physical router to the BGP Table VM and then peer the router that i want to test to the intermediate peering point to get more accurate results for speed of convergence.

Thanks Leo and thanks for sharing your work…I’ll be sure to check it out!

All of this by using some simple Python scripting as below:
https://ccie49534.com/2014/11/15/generating-dummy-static-ip-prefixes-with-python/

Cheers,
Leo

Nevermind I’m an idiot who doesn’t know how to use Linux.

Unfortunately the program bgpsimple does not have an IPv6 version and development stopped in 2011.

MikroTik does RFC testing and publishes the numbers on their website….this was intended to be more of a real world performance test.

Unfortunately, I don’t have the config from that test anymore, but considering the devices were directly connected in a lab, you might want to use two test devices and directly connect them with your current config and see if the speeds improve. If they do, then you know there might be an issue with your provider.

You can select LDP participation by interface in MikroTik. Just remove the default all and add only the interfaces that need LDP.

http://wiki.mikrotik.com/wiki/Manual:MPLS/LDP

Casey

thank you

do you know what is command in MikroTik for mpls bgp forwarding?

We have situation with MikroTik where we cannot do mpls bgp forwarding without LDP.
We need to set interconnection with other AS and we cannot shutdown mpls LDP in MikroTik.

Can you help us with some guidance?

Best Regards!

Did anyone ever perform RFC benchmarking for layer-2 using JDSU testsets or similar, through Mikrotik’s EoIP?
We’re using RB2011il-rm’s, and are getting bit errors and LOF and out-of-sync’s.

SD

Lyma

+1
I’m also interested in a test showcasing edge/peering scenario and PPS throughput.

All fair points and we plan to do testing with smaller packet sizes, but remember there are many different use cases for routers and ISP edge/peering routers are but one use case. Enterprises and Data Centers frequently use 9000 MTU on core network segments and especially on storage networks which are the one of the biggest growth areas in network engineering.

I realize that your test setup might have trouble generating this load, but even just a partial “we were able to generate 10 Gbit/s of small packets and the router did fine with that” is better than nothing. Right now we have a test with MTU 9000 at 80 Gbit/s. That is approximately 1 million packets per second. Or just 0.5 Gbit/s with 64 bytes packets. If that is all it can do, any kid with a 1 gig FTTH connection can kill it with flood ping! We need numbers…

Sadly – this continues to be an issue with Mikrotik.

The system at present only uses ONE CORE – even if you were to purchase the $3500 72 Core Tile Top of the Line Mikrotik – you would still have the one core issue…

Hoping they fix this in version 7 as they have hinted…

Mostly because our lab is used to virtualize different vendors to plan/validate network designs for our day to day work. Just having Linux servers for load testing isn’t as practical as having a VM Ware Hypervisor that we can segment with a Vswitch and real switch. I wish we had enough time to put into development on the server side but most of our work is all Network Engineering/Architecture so ESXi makes sense for us because we can rapidly spin up just about any environment we are working on. Thanks for the feedback!

Regards,
H.

Dobby, but it does have M.2 slots. Two of them

Can You test its over PPTP server with 2 Simple Queues for upload and download and 2 PCQ with 1Mbits upload and 1 Mbits download.
How many connections it will handle of pptp like pppoe?

Note I don’t use ifconfig – see this serverfault post.

An example (must be root):

(Example assumes you put your router on 192.168.200.2/24 on VLAN 200 and 192.168.201.2/24 on VLAN 201)

Creating two VLANs on a physical interface (e.g. eth0.200, eth0.201)

ip link add link eth0 name eth0.200 type vlan id 200
ip link add link eth0 name eth0.201 type vlan id 201

Creating a network namespace

ip netns add iperfserver

Binding one VLAN to a namespace

ip link set eth0.200 netns iperfserver

Bringing the VLAN up, giving the VLAN an IP address, and giving the namespace a default route

ip netns exec iperfserver ip link set eth1.200 up
ip netns exec iperfserver ip addr add dev eth1.200 192.168.200.1/24
ip netns exec iperfserver ip route add default via 192.168.200.2

Running iperf server in the namespace

ip netns exec iperfserver iperf -sD

Then you can set up your Linux routing normally:

ip addr add dev eth1.201 192.168.201.1/24
ip route add 192.168.200.0/24 via 192.168.201.2

And finally run iperf:

iperf -t 999 -i 1 -c 192.168.200.1

Cheers,
Tim

Thanks! And you’re welcome. Glad the info was useful for you. 🙂

We have designed solutions that scale beyond 100,000 VPN tunnels across multiple CCRs. From a tunnel standpoint, you can easily get 10,000+ connections per CCR using OpenVPN, after that it’s just a matter scaling as many CCRs and switches as you need to get the bandwidth at the physical layer. As far as IPSEC goes, you can get about 7.5 Gbps of hardware acceleration using a CCR.

I’m not aware of anyone who has it in production yet. We have several customers evaluating it as a potential core router, but haven’t actually put any with live traffic. Although, if the testing we have done is any indication, these routers will be able to handle insane amounts of traffic.

We are working on finishing part 3. We ran into some unexpected issues with our ESXi servers that are generating the load and hit a speed bottleneck in the PCIe bus at 27 Gbps per server even though they are capable of 40 Gbps. Once we have sorted that out, we will release the results of our testing.

nearly ~3200 € for an router that comes not sorted with one or two
miniPCIe or M.2 slots would be not really funny, also another phy
likes a free programmable Xillinx FPGA or a free PCIe slot to hug up
the entire number of LAN ports with perhaps a HotLava PCIe NIC
is a pity and really sad. I am really horny to get my hands on this
router and test it here in Germany also, but having some more options
for a larger user database, log or syslog storage would be really awesome.

But ok let us really see what they are have fiddled out.

You have a sharp eye nz-monkey 🙂 The first review was done with the pre=production unit, and MikroTik is sending us some pictures of the actual production model inside and out. Once we have a production model in our lab, there will be subsequent reviews with performance test metrics. Our ESXi lab hosts currently have 40 Gbps of capacity and we have parts on order to increase it to 80 Gbps so we can perform a proper load test.

Agree 100% this is a very exciting router from MT and opens up a world of possibilities.

e.g. does not have m2 slots and different phy

This is a very exciting product from Mikrotik.