The Case for Optimized Multi-layer Networks

And a big thanks to Cisco & DT for providing the proof points!

The IP-over-DWDM versus ‘optimized multi-layer transport’ debate dates back nearly 20 years. Every so often, it gets hyped up again (and then dies back down) until the next wave. Even my learned colleagues here at ECI have written about it numerous times over the years here, here and here.

Consistently, each side has its own set of supporters - i.e. the transport camp & optical vendors are big believers in the need for multi-layer transport while the IP/Ethernet crowd and the router vendors are staunch supporters of IP-over-DWDM. Over the years, the benefits cited by each side have been relatively consistent:
 

As my colleague has recently discussed here, the proliferation of 400GZR/ZR+ will all but bury the density issue for IP-over-DWDM. So, now that the biggest obstacle to IP-over-DWDM design is behind us, the path towards single layer design is open. Or is it?

Cost as a driver of single versus multi-layer networks

One of the latest in the line to speak about the need to move to IP-over-DWDM design was Cisco, who spoke at NANOG 77 last October. True to its game, Cisco’s conclusion was that IP-over-DWDM had more benefits than competing network designs, including optimized multi-layer transport (which Cisco calls Bypass):
 

However, the best thing to come out of this presentation (to my great delight!) is that the presenter provided a test case and Cisco calculations for each type of network design. Finally, we can go beyond the theoretical, and analyze a practical use case with quantitative measures and for that, I thank our colleagues as Cisco.

Interestingly, the one KPI that was missing in this presentation was -you guessed it, the cost. Surely Cisco could use this same use case to prove the #1 advantage of an IP-over-DWDM network: that it is cheaper to build a single-layer network than a multi-layer network? Well, since Cisco didn’t do it, I took the information provided and ran the numbers myself using published price information.

Calculating the costs

In the use case of a regional network with 55 sites, the speaker presented three possible network designs:

• Hop by hop – (otherwise known as IP-over-DWDM) Single-layer, single hop end to end network
• Hollow core – The other extreme which is based on a one-to-one mapping of traffic demands to end-to-end wavelengths
• Optimized bypass – An optimized multi-layer network

For each design, Cisco calculated the numbers of interfaces required based on a 400G interface solution.

My pricing assumptions:

• We took the data provided by Cisco – as is
• All pricing was based on Cisco’s current products (5508 platform and NCS2006 platform) and publically available price lists
• My calculations took into consideration 36x40G card and 100GB transceivers instead of the 10X400G line card and the 400ZR transceivers used as the basis for the use case, since these products are not currently available
• DWDM solution calculation was based on full featured CDC ROADM for every node
• Line card numbers are based on a 1:10 card to interface ratio, although this is likely the minimal number required

I focused only on the hop by hop versus optimized bypass, and so used the following configurations:

• Hop by hop – 55 routers and 55 optical 4 degree mux/demux with amplifiers
• Optimized bypass - 55 routers and 55 WDWM 4 degree CDC ROADMs

The results:

Less complexity as a driver for IP-over-DWDMResults are estimated and based on list price before discount, of course. I am sure the calculations can be improved, however I believe the ratios will be the same.

Now let’s consider IP-over-DWDM’s #2 advantage – the fact that a one-layer network is more simple/less complex than a multi-layer network.

If I’m not mistaken, there is one operator that tried to light the fire under ‘minimalistic, converged IP and optical layer network’ (in other words IP-over-DWDM) and that’s Deutsche Telecom with its Terastream project. Initially envisaged, back in 2012, Terastream was “Conceived as a way to radically simplify the design of telecom networks”.  Yet up until now, seven years later, as far as I know, project Terastream is far from mainstream at DT. When interviewed about the situation and status of the project, Axel Clauberg, one of Terastream’s biggest champions, told Light Reading here:  "The industry is clearly not where it was supposed to be," he said. "Maybe I was too optimistic in 2012".

Meanwhile complexity has received an unfairly, bad rap for many years. Complexity in itself isn’t all bad – automated complexity is everywhere today and usually provides a better experience than the simple, manual processes being replaced.  I mean just take a look at how mobile phones optimize the complex process of signal processing and roaming (as opposed to fixed line phones or even vs older generation mobile phones), and how today’s car engines optimize for performance by introducing much more complex sensor and control mechanisms.

Indeed, nobody would argue that today’s networks are not complex– they are capable of many things. That complexity is what drives both flexibility and the freedom to develop and optimize services. Granted complexity is difficult to manage manually, but this is precisely why we have automation, SDN, and (soon) full-blown AI-driven networks.

Can the IP-over-DWDM debate finally be laid to rest?

In the end, optimized multi-layer networks have significant benefits, irrespective of CAPEX. For example, they are much more flexible and resilient (i.e. ROADMs can switch degrees as needed automatically, without capacity degradation, thus optical restoration is done automatically). They are much more future proof due to optical transceiver technology. Also optical networks are easier to groom and enable easier optimization.

Also, remember that IP-over-DWDM still relies on optical fibers. As opposed to data centers, where fibers are abundant and easy to install, carrier networks utilize fibers that are often more scarce, difficult and expensive to install - thus they often require more than simple, optical interfaces. Even in the hop-by-hop design presented above, there was a significant amount of optical equipment, which (in spite of its cost) only provides static, manual configuration. The likely outcome of such a simplistic design is locked bandwidth, which requires truck-rolls to unlock.

Perhaps, as I said, I am just like the rest of the’ optimized multi-layer transport’ supporters. I urge you not to take the point of view above at face value. If you have another point of view, or better yet, if you believe my case is not as ‘open and shut’ as I say, share! And if you’d like to see my detailed calculations, contact me.