As the pandemic underlines the value of the internet more than ever, its underlying technology is making one of its biggest transitions for years. Companies are actively developing solutions that will enable the current 12.8TbE switching capacity of network switches to double every 18 months. This is because the powerful switch application specific integrated circuits (ASICs) doing their processing are speeding up rapidly.
Likewise, transceiver data rates are set to increase from 100 to 800Gb/s by 2022 and the number of transceivers per switch will grow from 32 to 64 or 128. The resulting density of components will be difficult to integrate, and will generate a lot of heat that is hard to dissipate.
Such advances will therefore require components that focus on energy efficiency, explained Erman Timurdogan, director of optical communications and process design kit development at Analog Photonics. That will reduce thermal losses enough for coolers to keep up, he said.
Multiple choice
To increase efficiency, advanced complementary metal-oxide-semiconductor (CMOS) nodes and multiple chiplets with different optimised functionalities, are being combined in multi-chip modules (MCM). ‘However, transferring the data out is getting harder,’ Timurdogan said. ‘There are only two electrical solutions: increase socket size to allow more connections, or increase the communication data rate in each connection. A large socket increases cost and decreases manufacturability, and copper connections are lossy at high data rates.’
There is a solution that does not suffer from these drawbacks: place optical fibre connections close to switch ASICs. That idea has driven two slightly different concepts with slightly different application focuses: On-Board Optics (OBO) and Co-Packaged Optics (CPO). Both move away from the pluggable module standards to which the optical communication industry has become accustomed.
The challenge for the companies delivering the resulting products is finding the right technical strategy to realise these concepts. And while development is ongoing, the industry is yet to agree on the necessary product formats.
In current module formats, electrical signals travel around 10 inches to the front plate for conversion to an optical signal. Hamid Arabzadeh, CEO of photonics developer Ranovus, explained this won’t be possible once Ethernet switch ASICs reach 51.2TbE aggregate capacity. Density of retiming ICs or digital signal processors (DSPs) needed to recover signal loss in traces and connections can’t keep up, while also meeting cost and power consumption requirements of the Ethernet switch systems.
Arabzadeh explained that OBO and CPO ‘take the photonic and electronic components of the module and co-package them with the Ethernet switch ASIC, creating an optical input/ output (I/O) using silicon technologies’. He adds that OBO can use Silicon Photonic (SiP) components. Alternatively, it can use PICs comprising two or more non-silicon-based photonic components in a single photonic device. But Arabzadeh argued that CPO can only use SiP due to density, cost, power consumption requirements, and reliability challenges of running in a hot environment.
Confidence boost
Ranovus’ Odin IC is designed to contain electronics and SiP on a single die for module, OBO or CPO applications. These different applications exploit the same IP, Arabzadeh says, allowing companies to try Odin ‘in a module form factor to gain confidence in the mass deployability of OBO and CPO’.
Odin includes several miniaturised, power efficient and cost-effective components on a single CMOS chip. It has demonstrated operation with eight optical channels of 100Gb/s per fibre pair, potentially enabling 25.6TbE switches. Arabzadeh revealed that it will be in deployment for Ethernet CPO in the second half of 2022. Ranovus also has a strategic collaboration with IBM, TE Connectivity and Senko Advanced Components to create an ecosystem to design and manufacture multi vendor solutions for CPO.
Currently, hyperscale data centre companies like Facebook and Microsoft lead CPO requirements, Arabzadeh said, but other sectors will use it when they have the same capacity requirements. ‘Larger contract manufacturers need to enter into this space to develop their capabilities and reduce the cost of fabrication,’ he added.
The integration of CPO on router and transport platforms allows the continued disaggregation of transport and switching, added Helen Xenos, senior director, portfolio marketing at Ciena. That separation allows each function to perform optimally. ‘Because of the significant benefits of this approach, CPO can extend beyond the initial intended application of switch-to-switch or switch-to-router within the data centre,’ Xenos said. ‘At the same time, CPO is optimised for all interfaces to be of the same type. This is much more common in hyperscale data centre architectures than in service provider architectures, where there are multiple rates, reach and service types. As a result, CPO offers the highest initial value for data centre deployments.’
Ciena is a member of the Optical Interconnect Forum, which launched the Co-Packaging Framework Implementation Agreement umbrella project. One of that project’s objectives is, said Xenos, ‘to identify the key co-packaged applications and their requirements, to widen the application space for CPO.’
Best of both worlds
In the first generation of the CPO implementation that the industry is working towards, the switch inputs/outputs remain electrical, Xenos noted. The lead switch interface proposal for traditional networking CPO applications is a 112G extra-short reach serialisation/deserialisation (Serdes) interface (XSR). ‘For a future generation of CPO, one consideration is to replace the Serdes interface and move directly to an analogue interconnect between the optics and switch,’ she said. ‘This would further reduce power dissipation and enable scaling to higher bandwidth.’
With in-house experience in both SiP and also InP photonic integration, Ciena is able to produce and supply the optical engine for the CPO solution, she said. ‘This development would reuse many of the building blocks that we already deploy in our coherent pluggable offering,’ Xenos said.
Analog Photonics is also considering the shift from modules to CPO designs. ‘To reduce the complexity of CPO, 51.2Tb aggregate switch bandwidth can be divided into 16 optical engines at 3.2TbE,’ said Timurdogan. But scaling PIC chipsets to 3.2TbE at a 100G signalling rate needs 32 channels. Using single laser wavelength channels would require 32 fibres each carrying 100G, which would not increase bandwidth density per fibre. ‘Instead, 32 laser wavelength channels can be used in a single fibre,’ observed Timurdogan. ‘However, this solution will not offer any backwards compatibility with existing multi-source agreements and IEEE Standards. We decided to form an 800G building block that is compliant with these standards and can scale to 3.2TbE. That is why we decided to use eight fibres and four laser wavelengths using the 400GFR4 format.’
Timurdogan also highlighted that electronic ICs for transceivers consume most power in enhancing signals and correct optical losses. The company therefore introduced ultra-low loss fibre-to-chip couplers to reduce signal degradation and minimised the drive voltage of transmit photonics to work with energy-efficient CMOS drivers. It also integrated an athermal, low-loss multiplexer and demultiplexer to mix and separate multiple wavelength channels integrated with SiP transceivers on the same platform, and increased the sensitivity of the receive photonics to reduce the optical power needed. The company intends to have a 3.2Tb chipset ready by the fourth quarter of next year.
While many of the specifications are ready for OBO and CPO products, Timurdogan highlighted the many current unknowns. He noted that CPO products will be harder to reach once installed, and may need to be pre-configured and sold together with CPO transceivers. ‘This makes the business model blurry, requiring switch maker, CPO transceiver and box companies to work closer than before.’
Good combination
Intel Corporation’s Robert Blum agrees that 100Gb/s channels are likely to be part of co-packaged optics systems, but anticipates them being combined with 50TbE or 100TbE switches starting in late 2023 or 2024. ‘That’s where you can realise the benefits,’ he said. ‘When we did the analysis, we see about 30 per cent lower electrical power for the combined switch, and also similar improvements in cost.’
Intel’s journey to this stage starts with 1.6Tb/s silicon PIC engines it has developed. ‘We have hybrid lasers that are part InP, part silicon, and we use InP photodiodes,’ Blum said. ‘On the pluggable modules today, we use different modulators that are much larger. For the photonic engine, we made ring modulators that are much smaller and integrated everything onto a single chip.’
On the packaging side, it’s another big development to put a photonic engine with the driver chips, the TIA and the receivers. Then, in March, the firm demonstrated a fully-functional 12.8Tb/s Ethernet switch based on CPO passing live traffic, exploiting four of its 1.6Tb/s engines. ‘You have the switch ASIC and you have to put the photonics around the entire chip because the I/O is so dense,’ Blum explained.
This demonstration is a ‘stepping stone’, he continued. ‘The data rates that we would likely use for switches at 50TbE will be higher, using a 3.2Tb/s or 6.4Tb/s PIC, scaling that to the next switch.’ After Intel attains that performance level it will ask the rest of the industry for commitments from hyperscale data centres to deploy these switches. ‘That’s an ask for the entire ecosystem to make that happen,’ Blum admitted. ‘It’s not something that Intel or any other company would do in isolation.’ Blum also wants agreement on format standards. ‘To communicate between your photonic chip and the switch ASIC, you need some kind of short reach Serdes’ he said. ‘What do you put on the photonics? What do you put in a switch ASIC? That needs to be aligned across the industry.’
Chicken and egg
Andrew Rickman, CEO of Rockley Photonics echoed this outlook. ‘There is a chicken-and-egg problem,’ he said. ‘Production deployment of CPO needs substantial investment in technology and field trials, but this requires some form of financial commitment from the end-users.’ There is a clear need for agreement across the industry on numerous factors, he added. These include the interfaces between the ASIC and the optics, packaging and form factors of the CPO assembly, the engines, the external laser source and on the format of the overall supply chain.
In 2020, Rockley demonstrated a 25.6Tb/s OptoASIC switch. Its OptoASIC devices combine one or more PICs and ICs within a single package, such as a switch ASIC and the surrounding optics. They use OptoEngines to convert between optical and electrical signals for eight duplex channels each, a figure that is likely to be more when commercialised. ‘Internally, the OptoEngine comprises a transmit PIC, receive PIC, and corresponding transmit and receive ICs required to interface the main ASIC’s Serdes channels with the PICs,’ said Rickman. ‘The transmit PIC incorporates modulators to imprint the electrical signal on the outgoing optical signal. Optical power may be provided by lasers either integrated directly on the PIC, or coupled in from an external laser source. The receiver PIC incorporates photodetectors to convert the incoming optical signal back to the electrical domain.’
CPO will also benefit users of high-performance computing systems and specialised systems for machine learning, AI or big data processing, Rickman observed. ‘Generally, any use case where bandwidth is getting constrained by power, density, or cost can benefit from CPO,’ he said. The fact that silicon-photonics-based integration can enable high-volume OBO and CPO products reflects the supply chain’s maturity, Rickman added. That’s thanks to silicon photonics’ key advantage of using volume manufacturing to drive down cost. However he emphasised again that adoption is the main hurdle. ‘CPO needs to overcome substantial inertia to move an entire industry away from the entrenched model of pluggable optics,’ Rickman said. ‘This migration has important ramifications for vendors and users of data centre optics.
