Orhan Ergun 19 Comments

Seamless MPLS architecture can be used to create large scale MPLS networks, reduce operational touch points for service creation, reduce overall complexity and enable flexible service creation points in the Service Provider networks.

Seamless MPLS architecture is best suited to the very large scale service provider networks that have 10s or 100s of thousands access nodes and very large aggregation networks ,but still want to have a predictable,proven control planeIP traffic increases rapidly due to video, cloud, mobile internet, multimedia services and so on. To cope with the growth rate of IP Traffic, capacity should be increased but at the same time operational simplicity should be maintained.

Seamless MPLS architecture is also called as Unified MPLS but there are small differences between them.

 

Let’s look at what typical service provider networks look like.

service provider topology

Figure 1 : Typical Service Provider Network

source : cisco.com.

Most of the service provider that may benefit from the seamless MPLS ,  have business, residential, mobile and/or wholesale access networks.

Access, Pre aggregation, Aggregation, Edge and Core are the common hierarchies in the large scale service provider networks.

You have probably always dealt with them as Edge (PE) and Core (P) devices that you will see as ABR and P in my topology drawings throughout this post.

Edge devices provide a service and they are the service initiation point. It can be MPLS L3VPN PE in the case of Business MPLS VPN or BNG for residential/business broadband etc.

Typical core nodes are in the rage of 10s, edge nodes 10-100s, aggregation nodes, 100s-1.000s, pre aggregation nodes 1.000-10.000s and access nodes are 10.000s to maybe 100.000 nodes.

I will explain you how these amounts of nodes still run end to end MPLS with single IGP with the seamless MPLS architecture . We will extend MPLS all the way down to the Access nodes.

Access nodes can be DSLAM, OLT, Cell Site Routers and so on.

It is common to see 10s of thousands of cell site routers in large mobile operator networks. Or for the Tier 1 Service Provider DSLAM, OLT and Cell site routers can be 100.000 of access device.

 

Business Requirements for Seamless MPLS

Reduce the operational touch points 

The picture below summarises the MPLS evolution in the Service Provider Networks.

service provider mpls

For some service providers, MPLS might be just in the core, but some of them extend MPLS to the Aggregation domain,and some may have MPLS up to the access device but not using seamless MPLS architecture.

The dashed line illustrates PWs between the domains. As you can see, there is no end-to-end LSP but LSP is stitched even in the 3rd picture, which has MPLS all the way down to the access domain.

Let’s look at the picture below to better understand service creation points today in most service providers networks.

service provider service creation

If you want to provision,let’s say a MPLS L2 VPN service, you need to provision an access device, Vlan is configured between access and aggregation. The Sonet/SDH, ATM access environment has already started be consolidated to Ethernet and IP/MPLS in the access network.

If MPLS is enabled on the aggregation and core network, then PW is provisioned between the aggregation and edge network. PW is stitched and end to end provisioning is completed.

With the seamless MPLS, service provision is ideally achieved by configuring only two nodes.

Reduce the OPEX 

If MPLS is used end to end from access nodes in one aggregation area to another access node in another aggregation area, a unified and predictable control plane reduces the troubleshooting time as well.

It is predictable because the MPLS control plane is robust,mature and highly available. If you don’t use MPLS in the access networks, you would deal with Spanning Tree, G.8032,REP protocols in the access layer. I’ll talk about these in a separate article.

Availability 

Many of the access layer protocols for the ethernet have good resiliency characteristics. When the MPLS is introduced to the access networks, this ability should be maintained.

Fast convergence in the seamless MPLS architecture is provided through IP FRR and/or MPLS TE-FRR, BGP PIC, Egress PE FRR.

I will show you how they work later in this article.

IPv6 Support

Seamless MPLS architecture supports IPv6.

But in order to understand the applicability,use case, and requirements that I’ve covered so far this was needed.

 

Let’s take a look at Seamless MPLS Architecture.

Seamless MPLS Architecture

end to end mplsOur target with a seamless MPLS solution is to have an end to end label switch path from the access node in one aggregation domain to another access node in a different aggregation domain.

In this topology although I show two access and two aggregation domains, Seamless MPLS architecture can scale easily 100 Aggregation domains, 10.000 aggregation nodes in those domains.

As you can see from the below picture, we will divide the MPLS domains and connect them through BGP as a hierarchical manner and create an Inter- domain MPLS LSP.

inter domain mpls lsp

Intra-Area LSP is created in each domain. IGP runs only on the Aggregation and Core domains.

For the inter-area LSP, all the nodes in the network have to reach loopback interfaces of each other. This is done via BGP.

We have a Single BGP Autonomous System with the seamless MPLS architecture.To have an end to end LSP, BGP + Label which is also known as RFC3107 is used.

seamless mpls control plane

In the picture above, you can see control plane operation in a seamless MPLS solution.

PE-1 and PE-2 represent access nodes. P devices are aggregation nodes, ABRs are Edge nodes that connect aggregation to the Core domain.

We use separate IGP areas to minimise IGP size in each domain. Assuming 10.000 aggregation nodes and every aggregation domain consist of 1.000 aggregation nodes, this network would have 10 IGP area + core backbone area.

seamless mpls different IGP

As you see from the picture, we have three IGP areas (area is a logical concept in this picture, it can be IS-IS level or OSPF area)

The core MPLS domain is placed in a different area t0 the aggregation network areas. If the IGP protocol is IS-IS, then the aggregation domain is placed into an IS-IS level 1 domain and the core IGP is placed into an IS-IS level 2.

Between the areas, routing reachability is achieved through BGP. Nothing is redistributed in IGP. Within the area reachability is achieved through Intra Area IGP.

IBGP sessions is created between PE and ABRs and between ABRs. Since all the sessions are IBGP, we need Route Reflector, otherwise we would have scalability issues. We will have 1000 nodes in each aggregation domain and we would have 450000 IBGP session if you wouldn’t use Route Reflector.

These route reflectors are also an Area Border Router or Level 1/2 routers.

Alternative IGP design, to run separate IGP process on the ABRs for all attach aggregation domain.

Because if you have a OSPF with same process, even if you have NSSA in the aggregation, those routes would be sent into the core if you don’t filter them. Creating an NSSA areas and filtering might be seen as complex.

For the IS-IS case, if you put Aggregation network into IS-IS Level 1 domain, nothing is sent except ATT bit in L1 LSP from level 2 domain into level 1 but from level 1 to level 2 route propagation can be prevented with filtering.

On the ABRs, we redistribute IGP into BGP to send the loopbacks of the routers to the other domains. BGP +Label (AFI/SAFI 1/4) is sent between IBGP neighbors.

One might ask , what about the access devices ?

We have DSLAMs, OLTs, Cell Site Routers , do all those devices have to support IGP ?

Answer is no. They will have static default route towards aggregation routers. But if you want to have end to end LSP , those devices have to support at least light weight LDP implementation.

LDP label advertisement can be done in two ways. Downstream Unsolicited or Downstream-on-Demand.

If Downstream Unsolicited label advertisement is enabled on the LSR interface, router sends a label binding for the prefixes to its neighbor without waiting neighbour’s request.

If the Downstream-on-Demand label advertisement mode is enabled router asks label information when it is needed.

For the access node, downstream-on-demand label advertisement is used. Aggregation nodes and access nodes label binding information for the necessary loopbacks.

One caveat of having default route on the access node is LDP specification. Normally LDP nodes install label information into LFIB for the /32 prefixes. RFC 5283 which is Inter area LDP extension can be used to overcome this.

Aggregation nodes should have static route for the access nodes loopbacks and those static route should be redistributed into the Aggregation domain IGP for the Intra domain (Transport Label) LSP.

Access device loopbacks , same as aggregation domain loopbacks, redistributed from IGP into IBGP on the aggregation nodes. In our topology aggregation node is the ABR but for the larger scale , we have separate Aggregation nodes in addition to ABRs.

Let’s see end to end packet flow with the dataplane interaction.

Screenshot 2015-03-02 09.59.06

At the headend access node, we have 3 labels in the stack. Top most IGP label comes from LDP to carry the traffic to BGP next hop which is ABR1, ABR1 assign a BGP label for the PE2 loopback.

When the packet comes to the ABR1, ABR1 swaps its own BGP label with the BGP label assign by ABR2.Both ABR1 and ABR2 is BGP route Reflectors but we also set BGP next-hop self on this RR to achieve scalability.

If ABRs would be a route reflector PE devices would have end to end Transport LSP which is created by LDP  for the loopbacks of every other PEs. But since each IGP domain is separated and not redistributed to each other, this is not possible.

When BGP next-hop self is configured on the Route Reflector, since the bgp next hop will be a Route Reflector which are also an ABR, every PE will have only two labels for the Route Reflectors. ( I assume for the redundancy we have more than one RR/ABR ).

When ABR1 swaps the BGP label, ABR1 pushes new transport label which is assigned by LDP  to reach ABR2.ABR2 is the next hop for the remote PEs which are access devices in our topology.

 ABR2 is the penultimate hop from the BGP point of view and it does PHP for the BGP assigned label. It pops the BGP label, push the transport label assigned by the local LDP.

PW label as you seen doesn’t change end to end.

With seamless MPLS if you want to create an MPLS Layer 3 VPN service end to end, it can be done by configuring only two PEs. No PW stitching,no different LSP segments, no VLAN traffic engineering and so on.

Conclusion :

Seamless MPLS architecture is suitable for very large scale deployments such as Tier 1 SP , Mobile Operators which has thousands of cell site routers and so on. 

Otherwise bringing BGP + Label into design might be seen as complex.

There is no Standard or Informational RFC for the Seamless MPLS architecture but all the element especially for the availability requirements, are supported by Cisco. If other vendors have similar solution,let me know in the comments box below.

3 labels are enough to have LSP hierarchy for Seamless MPLS to support 10s of thousands of devices in the network.

But definitely it reduces the services touch points so to have end to end service, instead of configuring,managing,troubleshooting 5 to 10 device, seamless mpls ideally achieves to service creation with only two devices.

Seamless MPLS doesn’t need any protocol extensions but for the sub 50 msec or sub second convergence it requires IP FRR, BGP PIC, Egress PE FRR mechanisms.

Having separated IGP areas is must. Only the loopback interfaces of PE,Aggregation and ABR devices are known.

P devices and transit link information is not required. At least not all the devices have to know P devices and link information. Still for the monitoring purposes  they can be carried to the management system.

Do you think that your company can benefit from Seamless MPLS ? 

Let’s talk as always in the comment box below.

 
0.00 avg. rating (0% score) - 0 votes
  • Alvaro Arriola

    Hey Orhan, excellent coverage of a kind of mysterious topic. I was in a Huawei workshop a few months ago and they mentioned the seamless MPLS design along with their new services. I was kind of let down with the fact that they didn’t go into detail, they just mentioned it as an alternative for CSC. I think some characteristics of the seamless MPLS architecture are shared with the CSC architecture. Thank you for this post, I wasn’t able to find any good coverage on this topic.

    • Thanks Alvaro. I see some resources on Juniper and Huawei relate. It is good to see they have a vision and architecture support. This is nice,hierarchical and for very large deployments. I will cover how it can work with MPLS-TP , and the differences etc.

  • Chinar

    Great BLOG! KUDOS to you Orhan for bringing this so neat!!

    Seamless MPLS is a very robust architecture for 4G or new upcoming ISP architectures. Ideally, if instead of community based-redistribution at Pre-Aggregation Nodes, if the BGP-LU was extended all the way till CSR routers (PE1 in your diagram), then it would be ideal E2E Seamless MPLS with BGP-LU (RFC 3107) stitching inter-domains.

    It’s 1 complex network architecture, but when understood in depth the details, it’s incredibly interesting to see the scalability an ISP can achieve with Seamless-MPLS and BGP-Labelled-unicast architecture.

    Rgds,
    Chinar

    • @Chinar , welcome to the site. Thanks for commenting. You said ” if the BGP-LU was extended all the way till CSR routers (PE1 in your diagram), then it would be ideal E2E Seamless MPLS with BGP-LU (RFC 3107) stitching inter-domains “. PE1 is access node in the diagram, so probably supporting BGP on all access device type in SP would be hard, even it would, then definitely RT constraints, ORF type of filtering should be there. Probably from the access nodes static routing is enough for many service. But wouldn’t it be nice to provide traffic engineering on the access node or starting from access node ? 🙂

      • Chinar

        Orhan,

        TBH, I haven’t played around much with Traffic Engineering to comment on it. What I meant was routers would need 4-Label stack to support Seamless MPLS till the end. 4 labels constitute LDP-Label , BGP-Label, VPN label and even LDP -FRR convergence mechanism label. If the access routers (PE) support 4 label stack, seamless MPLS is a very good design model to be considered and even implemented.

        • Hi Chinar, Somehow I didn’t respond this. Not you don’t need 4 labels during a ” traffic on a protected link “. You are still doing end to end labeling. You just don’t need to do it with BGP between access and aggregation node. Access and aggregation is using static route for each other and DOD label advertisement is used for labeling.

  • Pingback: OSPF Design Challenge - Network Design and Architecture()

    • Grzegorz Wypych

      Orhan wrote: “On the ABRs, we redistribute IGP into BGP to send the loopbacks of the routers to the other domains”

      This is not requirement, you can and you should redistribute loopbacks of RR/ABRS to IGP domains. From core to aggregation IGP and filter using prefix-list.

      • As you said RR=ABRs loopbacks. I say routers , which mean PEs. Scalability..

  • Hello Orhanergun
    i have a question..
    in your graph PE1-P-ABR, the P between PE1 and ABR is really a P or a PE?, in other words in which routers VPNs exist and in which no?

    In a network with 1000 PEs how many inter areas IGP you recommend create?

    • Hi Andres, In that picture PE-P-ABR is explained in the context of one area. So inside an aggregation domain,P device is just doing label swapping based on Transport label of Intra area. ABR is doing pop and swap operation based on 3107.

      For your second question;

      Seamless MPLS idea is to bring edge role to the Access node.

      So service can be created at the access node; DSLAM,OLT,Metro ETH switch etc.
      Then 1000 PE is not really a target number of devices. If it 1000 device though, I would put probably in one area only.

      Consideration how those devices,links are reliable, how they are connected so LSA number should exceed the MTU of a link since you don’t want to deal with fragmentation.

  • Orhan, great article! I’m a big fan of seamless MPLS, but have only deployed it on smaller scale networks. Very interesting to see how you scaled it out to very large environments. Would like to see something similar, but incorporating end-to-end TE.

    • I just explain this, seamless mpls is okay but actually everything made simpler with another architecture.

      I will write a comprehensive article about it , something similar to Network Complexity article but hope I can find a time since 2 CCDE classes will start in April at the same time

  • Sunil

    But here setting the next hop self on the RR, we are putting the RR in the data plane , I think an additional overhead on the RR. There must be some way to prevent this.

    • Thanks for reading and commenting Sunil. You need to do that and you want to do that ( Similar argument I did in my network complexity article-check that one as well, you want network complexity,you need complexity)

      You need to do that , because as I stated in this article, IGP domains are restricted.We don’t advertise those each other. Or if you run different IGP processes, you don’t redistribute IGP processes to each other.

      When you do next-hop-self as you know you put RR in the data plane. You create a new inner label for the loopbacks with 3107.

      But this help to IGP to find the BGP next hop. If you wouldn’t do that, for the Intra domain LSP, IGP domain had to be end to end.

      By doing next-hop-self you hide the complexity beyond the ABR from individual aggregation domains ( Thus you want that )

      By the way, I will be writing an article to compare Seamless/Unified MPLS with the past and current Service provider architectures to show actually how can we achieve scalability with different ways but my both CCDE and Pre-CCDE classes started this week so I need to postpone the long articles a little while.

      Cheers

  • Zachary Mokua

    how can you introduce seamless MPLS with automation in a multi-vendor environment; assuming your core network is from cisco {management by cisco prime} whereas the aggregation and access are huawei (with U2000 management)?

  • I’m waiting for your article of comparison between Seamless MPLS/Unfiied MPLS , Orhan.

    Wanna see how different is Seamless from Unified.
    I thought Unified MPLS is a cisco term for Seamless MPLS.
    If I’m not wrong, Huawei calls it H-MPLS (Heirarchical MPLS model or something) using RFC-3107.

    Cheers

    • Hi Chinar,

      There are couple differences, there is a book which explains all the variations very well. You may want to take a look at that as well. Yes always 3107 is used , not just Huawei

      Cheers,
      Orhan

      • dinag

        which book plz