LAG vs. ECMP discussion on real network deployments

We discussed LAG (Link Aggregation Group) and the ECMP (Equal Cost Multipath) on real network deployments with the Service Provider/Telco Engineer engineers on my slack group.

I thought it was good discussion so you can see what others are doing and the reasons of their deployments.

In this talk, three people involved. Myself , Engineer 1 and Engineer 2. Since the participants may not want to expose their networks publicly and I didn’t ask their permission, just I used Engineer 1 and Engineer 2, instead of their names. Engineer 1 started the discussion. Nice one, hope you find it informative. Enjoy..

Engineer 1

Hello guys, I hope you are all good.

I wanted to spark a little discussion about using LAGs vs ECMP in your networks. What are you using and what was the driver? Personally I use bundled interfaces.

It’s a bit easier to manage, less IP addresses is used, IGP LSDB is smaller, if member link goes out of service it’s hidden from IGP view.

Load balancing depends on platform, but I think it’s as good as in case of ECMP. Obviously with bundles we are restricted to pair of the same devices.

The downside is that it’s very hard to troubleshoot if LAG member link is malfunctioning, because traceroute always look the same. Also, if we need BFD support it may require micro-BFD which is not always available. With ECMP links we don’t have that problem. Any other gotchas?

Oh, and QoS sometimes is a pain on bundles

Engineer 2

I prefer ECMP rather than LAG anywhere possible

BFD and link issue are easier to run with individual links

And for the LAG, it may cause congestion when one link member is down

For us, the IGP cost is fixed (manually) on the link. When the link member goes down, the cost remains the same

Hence the LAG link is more prone to congestion.

Engineer 1

Well, with LAG you can rely on min-links, so if too many member links go down entire LAG will go out of service

Engineer 2

Yes. But if there are only 2 x 40G in the LAG, we would have big problem.

Orhan Ergun

Depends on the place in the network as well in the datacenter mostly LAG and MLag is used , without even BFD, failure detection is much faster due to physical link detection and if you will run routing protocol, and this is the case in any large scale DC anymore , LAG reduces the number of adjacency as you stated above but on WAN , mostly ECMP is used

Those who have large WAN networks , may not have always ECMP available , in that case neither LAG nor ECMP available but everyone use traffic engineering

MLAG might be a solution if links will be terminated on the different devices but , again , on the WAN , almost no one uses it , MLAG I see on the DCI (Datacenter Interconnect ) and in the Campus designs

Also, business types might dictate certain topologies , for example retail enterprises mostly have hub and spoke hierarchy , in that case , from the spoke site when there are two links toward Hub location, which is generally HQ or Datacenter , most people use ECMP , regardless of underlying service , be it MPLS VPN , Internet , P2P Ethernet and so on

Also, since Engineer 1 is asking your deployment model , requirements and the constraints for it , please share what you have and why you have when you have a question, you can expect the others to do that then.

Engineer 2

We used to have lots of LAGs and frankly, they work fine.

Occasionally, there is issue like the one I shared above that led to congestion

We counter by putting sufficient BW

We used LAG mainly because it is so easy to use. Just make sure the link quality is good, add it to the LAG, and it is done.

Orhan Ergun

You use it on the WAN ?

Engineer 2

Yes, @orhanergun

In SG context, WAN is around 70km

Orhan Ergun

Thats okay , but what is the topology , I mean layer 3 topology , not the optical one

Engineer 2

The core is partial mesh ; while PE has dual links to two diff P

Orhan Ergun

How LAG works between PE and two different Ps ? do you have MLAG (Multi Chassis Link Aggregation Group) ?

Engineer 2

Oh no. The LAG is between a PE and a P. Both PEs will have two LAGs to two P. And use ECMP if the destination has equal cost

Engineer 2

We run BFD for fast detection and failover

It is that simple ?

Orhan Ergun

On LAG bundles do you have micro bfd ?

Engineer 2

Yes.

Orhan Ergun

Ok I wont ask the devices

Engineer 2

Interestingly, we use two vendors and the BFD works flawlessly

Orhan Ergun

Interoperability is fine ?

Engineer 2

Yes.

Orhan Ergun

Couldn’t you do fast failure detection at layer 0, layer 1 ?

Engineer 2

In some cases, we have DWDM in between. So L1 detection isn’t enough

Orhan Ergun

Ok

you have SDH too ?

Engineer 2

No, we do not use SDH

SDH serves as our access/last mile

Orhan Ergun

Still you might need fast failure detection at your last mile for business customers

Are you using APS in that case ?

Engineer 2

Oh, I don’t have access to that part of the network

But as far as I know, for SDH, they can provide 50ms for switching/protection.

Orhan Ergun

Then they have

Engineer 2

We also have a metro-e network for access

And they use TE FRR if I am not wrong

Orhan Ergun

Are you using G.8032, REP , ERP etc ?

Engineer 2

But it is not part of my network

Orhan Ergun

Any fast failure mechanism for ME ?

You mean, you have TE FRR in the access network ?

Engineer 2

That’s what I understand

Created by
Orhan Ergun

Orhan Ergun, CCIE/CCDE Trainer, Author of Many Networking Books, Network Design Advisor, and Cisco Champion 2019/2020/2021

He created OrhanErgun.Net 10 years ago and has been serving the IT industry with his renowned and awarded training.

Wrote many books, mostly on Network Design, joined many IETF RFCs, gave Public talks at many Forums, and mentored thousands of his students.  

Today, with his carefully selected instructors, OrhanErgun.Net is providing IT courses to tens of thousands of IT engineers. 

View profile

Daniel Lardeux
Daniel Lardeux Senior Network Consultant at Post Telecom

I passed the CCDE Practical exam and Orhan’s CCDE course was very important contributor to my success. I attended the CCDE course of Orhan Ergun in July and it was exactly what I needed, Orhan is taking the pain to break down the different technologies.

Roy Lexmond
Roy Lexmond Senior Network Designer at Routz CCDE #20150017 & CCIE R&S; #26557

After I attended Orhan Ergun’s CCDE course I passed the CCDE practical exam.I really enjoyed the course a lot ...

Nicholas Russo
Nicholas Russo Network Consulting Engineer (CCDE/CCIEx2), Cisc

I signed up for Orhan’s CCDE training. This training is very technically detailed and the use-cases, quizzes, scenarios, and mind maps are all great resources in the overall training program. Orhan teaches his students to think like a network designer ...

Slide Heading
Slide Heading Network Systems Engineer at Conscia A/S CCIE #42544 (SP) & CCDE #20160015

Orhan is forcing you to take off the implementation hat that most of us have been wearing for many years, instead he is providing a new fancy design hat, which makes you see and deal with the issues presented ...

Kim Pedersen
Kim Pedersen CCIE in RS and SP (#29189) CCDE#20170021

I’ve used Orhan’s self-paced CCDE training material. If you are interested in knowing how all the technologies go together in a coherent design i can highly recommend it.I also enjoyed the Quizzes which helped pick out my weak spots in selecting ...

Laurent Metzger
Laurent Metzger 3xCCIE/CCDE Senior Network Architect

Hi Orhan. I passed the CCDE exam on February 22. I read everything that you put on your Self Paced CCDE Training course and it was very helpful in my success. Thank you very much.

Martin J. Duggan
Martin J. Duggan Network Architect at AT&T;, Ciscopress Author CCDE #20160006 & CCIE#7942

I attended Orhan’s April 201610 days CCDE Bootcamp. I am CCDE now !

You can tell Orhan has a great deal of experience, it really comes through when he presents his design case studies and the CCDE Practical scenarios.

Muhammad Abubakar
Muhammad Abubakar Lead Network Architect – CCDE #20160016 2xCCIE #26693 2xJNCIE VCIX

Your excellent CCDE materials and amazing Bootcamp helped me tremendously through my learning journey.Also thank you very much for being available whenever I have a design question or a complex design topics. I can’t compare your design skills ...

Jennifer Pai
Jennifer Pai Network/Security Engineer at KNET Technology

Thanks Orhan very much for this course. It helped strengthen my “Network design mind”.

Ruslan Silyayev
Ruslan Silyayev Solution Architect at R.I.S.K Company

Training by Orhan is not a CCDE preperation training only. It will be useful for engineers which are dealing with design. You want to pass CCDE exam or learn network design, then don’t look at anywhere else!

Sameer Meher
Sameer Meher Solutions Architect at 23 Wards/Japan

Orhan Ergun’s CCDE course was really very good. CCDE Level Intelligence was delivered very well and with very useful case studies and the scenarios, I am thankful to Orhan for all his help!

Ken Young
Ken Young Senior Technical Architect Province of Nova Scotia, 2xCCIE #41597 | CCDE #20170047

If anyone wants to understand network design and architecture, also pass CCDE exam , I recommend you to attend Orhan’s online courses! I am a CCDE now but learning is a journey, we will be together in your other courses too Orhan!

Matt Cross
Matt Cross Technical Architect at Heartland – CCDE #2019::7

Orhan did an excellent job of filling in the gaps of knowledge that I had that took me to the finish line of the practical exam CCDE. The community of people that Orhan facilitates are both engaging and supportive of the journey to CCDE. Orhan ...

Shiling Ding
Shiling Ding Sentinel Technologies – CCDE #2019::12

Just passed the CCDE Practical exam! I attended Orhan Ergun’s CCDE training program , used Orhan’s Instructor Led and Self-Paced CCDE training and Online CCDE Practical Scenarios during my CCDE journey. Orhan’s CCDE In Depth book is an excellent summary ...

Abelardo Basurto
Abelardo Basurto Solutions Architect at Cisco Systems – CCDE 2018::6

Hi everyone, I’ve just passed the CCDE Exam. My Number is CCDE 2018::6 I attended to Online CCDE Bootcamp of Orhan. I want to thank Orhan not only for the great book and bootcamp, but also for his commitment, availability and willingness to assist the ...

Hady Mohamed Abdellah
Hady Mohamed Abdellah Network Architect Hamad International Airport Qatar – CCDE 2018::1

Hi guys, I’m so happy that I passed the exam. I’ve already got my number CCDE 2018::1. Thanks to Orhan for being the best CCDE instructor in the world. I highly recomend Orhan’s CCDE Training and In-Depth-CCDE ...

Bryan Bartik
Bryan Bartik Sr. Systems Engineer at CompuNet – CCDE 20170059

Hi Orhan I passed CCDE Practical exam on November 2017 ! I really enjoyed your materials and quizzes and use cases. They were definitely helpful in my preparation. Thanks a lot !

Giedrius Trapkauskas
Giedrius Trapkauskas Network Solutions Architect at Liberty Global – CCDE 20180004

I attended Orhan’s CCDE Training in Istanbul and it was very helpful in my preparation. I passed the exam recently and I want to say Thank you Orhan! For those who want to pass the CCDE exam, definitely start with ...

Alaa Issa
Alaa Issa Sr.Solutions Architect – CCDE#20180033 3xCCIE ( Collab|DC|Security )#27146

I registered to Orhan’s training in Feb 2017. From that time, I attended Orhan’s training several times. The depth of knowledge which Orhan has is amazing, and how to present such consistent knowledge to the ...

Mazin Ahsan Design Lead Engineer | Solutions Engineer | CCDE License # 20160030 | CCIE Licence # 23892

I passed the CCDE Practical Lab exam on November 17,2016 from supplications of elders and dedication from my Sensei Mr. Orhan Ergun I took different CCDE bootcamps in the past. Orhan has the most depth and expertise ...

Jeff Patterson CCDE# 2018::11

Hi Orhan I wanted to pass along my appreciation for the outstanding training material. I used the online CCDE training provided by Orhan as well as the In-Depth-CCDE book and passed the exam in February 2018. Thank you Orhan!

Mehdi Sfar
Mehdi Sfar Network and Security Architect / CCDE #20210003 | CCIE R&S; #51583

I signed up for Orhan’s CCDE Self paced Course. This course, along with the CCDE In Depth book, helped me for my CCDE Practical as well as Written exams. It pushed me to ask the "WHY" questions and allowed ...

Related courses

Cisco CCDE v3 Training

71:39:34 Hours
225 Lectures
Expert

$1246

Network Design Fundamentals Course

02:28:37 Hours
17 Lectures
Beginner

$50

Layer 2 Network Design Training

02:16:18 Hours
17 Lectures
Intermediate

$40