Advances in data centre-related infrastructure are largely driven by increased network demand for higher bandwidth, faster service and ubiquitous access. Because of these trends, very high fibre-count cables (more than 3,000 optical fibres) become essential for connecting hyperscale data centres within a campus over distances of less than 10km. Another application space that is currently a topic of active interest in the industry is metro data centre interconnect (DCI). In this scenario, ultra-low-loss fibres can extend network reach, ultimately allowing data centres to be moved to the areas with cheaper real estate and closer proximity to renewable energy sources. The study presented in this article highlights how innovative optical fibre and cable products could reduce the cost of connecting data centres across a wide range of distances.
Distributed data centres
Today the internet has touched almost every single aspect of our lives, such as e-commerce, social networking, online games, and video streaming. Enterprise and government organisations are increasingly placing their mission-critical workloads into the cloud to ensure ubiquitous access to content and services, anywhere, anytime.
Cisco’s Global Cloud Index 2015–2020 predicts that as newer applications such as virtual reality and the Internet of Things (IoT)emerge, global data centre traffic will increase by more than threefold over that period.
In the face of rapidly increasing demand for higher capacity, cloud networks have been migrating to a distributed regional architecture. In a more traditional campus model, data centre capacity expansion is achieved by adding more servers within the same data centre. However, in today’s world, data centre operators often prefer to build a second data centre, either within the same campus or on a geographically separate campus, and then connect it to the primary data centre (Figure 1). Such an approach alleviates the constraints of limited space and/or high land costs, which is particularly pertinent in dense metropolitan areas such as London or Los Angeles. It also allows the operator to power the new data centre from a different electrical substation and thus limit the impact of a power failure.
In many cases, setting up a second data centre represents the only viable approach to store and process the expanding amount of data being generated by customers around the world. For example, Amazon at its AWS Re:invent 2014 conference, revealed that the optimum size of an AWS data centre is 25–32MW with 50,000 to 80,000 servers; an increase in the number of servers beyond this would necessitate the construction of a new data centre facility.
The existence of multiple, physically diverse data centres creates the demand for low-cost, high-bandwidth, and low-latency data centre interconnect (DCI). As the name implies, DCI links data centres either within campus, metro, or even long-haul areas, with link lengths ranging from a few hundred metres to several hundred kilometres. According to ACG Research (2H-2015 Worldwide Optical Data Center Interconnect Forecast, April 2016), up to 70 per cent of the DCI links around the world are 80km or less, and these will be the focus of our analysis.
Transceiver technologies compared
The maximum capacity that can be achieved for a DCI link depends on the available transceiver technology, which has been evolving rapidly over the last few years. Many transceivers used today are still operating at or below 40Gb/s, but there is an increasing trend to adopt 100Gb/s transceiver technology as the ‘work horse’ for DCI. Furthermore, 400G modules are being discussed at international standards meetings and conferences, such as the IEEE P802.3bs 400Gb/s Ethernet Task Force, which provides physical layer specifications for link distances up to 10km.
High-speed DCI transceivers can vary significantly in terms of price. Typically, higher-data-rate transceivers rated for longer distances are more expensive. In Figure 2 we have mapped a number of existing and emerging 100G and 400G DCI transceiver formats alongside their estimated market pricing. To perform such an estimation, we used data presented at public forums or published over the last 12 months. We also cross-correlated this data with the difference in technical implementation of these transceivers, by evaluating the number of optical components (e.g. lasers and receivers), bandwidth, type of modulation format, and wavelength grid (i.e. CWDM versus DWDM).
Our study determined that 100G single-wavelength transceivers based on four-level pulse amplitude modulation (PAM4) could represent a cost-effective replacement for some existing 40km- and 80km-rated transceivers. A contributing factor to 100G PAM4 transceiver cost-effectiveness is the use of direct-detection technology, spectrally-efficient modulation (two bits per symbol), and modest-bandwidth electronics. The next couple of years will be pivotal in determining future 100G PAM4 success and rate of adoption. Hence the conclusions are:
- Cost-effective 100G CWDM4 transceivers will continue to dominate the 2km space in the foreseeable future.
- The 10km space will become more fragmented and more competitive: the ‘workhorse’ 100G-LR4 form factor will continue to exist, but will be challenged by emerging CWDM-based 100G 4WDM-10 transceivers.
- In the 40–80km space, 100G PAM4 could displace incumbent 100G ER4 and 100G coherent quadrature phase-shift keying (QPSK) transceivers due to their lower costs.
Innovation continues: equipment vendors are actively working on higher-data-rate modules for future data centre connectivity needs. The IEEE has started work on a 400G standard based on PAM4 technology for applications up to 10km. Several coherent modulation formats are also being considered for 400G metro DCI connectivity, including single-carrier 400G coherent transmission based on 64-state quadrature amplitude modulation (64-QAM). Despite its current high cost, 400G transceiver pricing is likely to decrease in the future as the technology matures.
Advanced optical cables for DCI
Shifting focus to the optical cabling, we determined that advanced optical cables and advanced optical fibres could provide significant benefits for on-campus (2–10km) and metro-DCI (40–80km) applications, respectively (Figure 2).
Our reach-price transceiver analysis indicates that, for distances up to 10km, data centre operators will use cost-effective, four-wavelength 100G transceivers operating in the 1310nm window. These transceivers require a dedicated fibre pair to achieve 100Gb/s of bi-directional capacity, and a further increase in capacity would require lighting up a second fibre pair. With the data traffic between a cloud provider’s on-campus data centres now regularly in the order of 100Tb/s, servicing this traffic requirement with this transceiver type would necessitate the use of at least 2,000 fibres, 1,000 per direction (Figure 3). Data centre operators may also overprovision fibre on day one to provide some protection against future data traffic growth; with 50 per cent capacity over-provisioning, the number of required fibres in this case would reach 3,000.
Fibre count requirements will be potentially lower for 400G PAM4 transceivers; however, those transceivers are currently in the early stage of their development and standardisation. The successful adoption will be contingent on their availability from multiple vendors and price, once they become commercially available.
An additional constraint for high-fibre-count cables is their ability to fit within the existing ducts, which in North America typically have an inner diameter of 2 inches (about 50mm). This condition is valid irrespective of the transceiver technology used and has increasingly placed a focus on increasing fibre density in the cables serving the DCI space. Cable manufacturers have responded by producing higher density cables with higher fibre counts and smaller diameters to overcome these challenges.
Advanced optical fibres for DCI
In the 40–80km DCI space, high-speed transmission modules are typically designed to operate over generic singlemode fibre types that typically have 0.20dB/km attenuation. Existing technologies, like 100G coherent QPSK, are resilient enough to maintain adequate transmission performance even over an aged fibre plant. However, any increase in modulation density leads to higher required optical signal-to-noise ratio (OSNR), causing a rapid decline in transmission distances.
For the 64-QAM format on which single-carrier 400G coherent optical transceivers are based, the maximum transmission distance is reduced by a factor of about 10 when compared against 100G QPSK. Therefore, the use of ultra-low-loss optical fibres with 400G 64-QAM transceivers is an interesting avenue to explore in order to extend the reach of high-capacity DCI.
Calculations that we carried out using a Gaussian-noise model showed that 400G 64-QAM transceivers can achieve 70km single-span (unrepeatered) transmission using a generic singlemode fibre with attenuation of 0.20dB/km. But by selecting ultra-low-loss fibres with attenuation of 0.17dB/km, the reach can be extended to about 90km.
Two critical areas for consideration when choosing a location for a new or secondary data centre, are real estate cost and access to energy sources. It should be noted that a cloud-scale data centre built today may consume several hectares of land – equivalent to several football pitches. Selecting a location with lower land costs of $100 per m2 can deliver $1 million of savings for a data centre occupying a single hectare.
The declining cost of renewable power generation, along with tax incentives available in many jurisdictions, has made renewable energy an increasingly attractive option for data centre operators. Indeed, high availability of cost-effective renewable energy sources has led many firms to locate operations in the Pacific Northwest in the US (hydropower), Iceland (geothermal), and Sweden (hydro and wind power).
Assuming the primary data centre is in downtown Los Angeles, the use of ultra-low-loss fibre could increase the radius of potential placement for the second data centre by 20km, reaching more rural areas with lower land costs (Figure 4). This also corresponds to a 50 per cent increase in the coverage area, and hence a substantially wider choice of locations for a new data centre facility. In the Los Angeles area, for example, many of the grid-supplied renewable electricity source options are located more than 70km from the city centre. Thus, the deployment of ultra-low-loss fibre may provide data centre owners with the opportunity to build nearer to renewable energy sources and help them achieve their committed goals for clean energy.
In conclusion, as global data centre traffic continues to grow, more data centres will be built and the need for high-capacity links to interconnect those centres will increase. Optical fibre and cable innovations will help deliver a bright and agile future for the cloud and all its applications, as DCI and transceiver technology evolve to keep massive amounts of data seamlessly flowing around the world.
• Hao Dong is market technology development manager, Corning Optical Communications, based in New York
