As befits its status as a computing giant, Microsoft’s musings about the internet’s future are poised to reshape the optical communication industry. In August 2015, following discussions with colleagues and suppliers, Brad Booth, principal engineer at Microsoft in Redmond, Washington, helped found the Consortium for On-Board Optics (COBO). The move answered a question: looking at network speeds of 400G and beyond, will networking equipment that uses faceplate pluggable optical modules continue to make sense?
The width of the electrical connector on pluggable optics modules conventionally used in telecom and datacom environments constrains how many units a user can plug into a faceplate. One option is therefore to create a newer pluggable module specification, like the recent QSFP double density (QSFP-DD). Even then, when considering powers of 10W and higher needed for 400G and the consequent thermal issues, Booth recalls the realisation that ‘it’s probably not going to fit in traditional modules’
‘A traditional QSFP28 module was only designed for about 3.5W,’ Booth pointed out. ‘If a switch that has 32 ports of QSFP28 today was to upgrade to QSFP-DD consuming around 10W each, that means I’ve got a faceplate with 320W of power density. How do I move the resulting heat away? How do I guarantee a high level of reliability? One way is that you don’t block the faceplate with optics – instead you move the optics inside.’
Microsoft therefore started talking to vendors about moving optical components from pluggable modules directly onto line cards containing network switching electronics. ‘A lot came and said “We can create the on-board modules that you’re looking for”,’ Booth said. ‘That’s good but we’d like to buy from multiple different vendors, so that drove discussions to create an industry forum to enable standardisation.’
Today, as COBO chair, Booth has started work on draft standards, white papers, and best recommended practices with representatives from companies from across the industry and organisations like IEEE. Microsoft recognised that it needed to take a leadership role to propel any move away from pluggable transceivers and toward on-board optics it might desire.
‘If we wanted to do it for ourselves, we probably could have, but it didn’t make much sense,’ Booth emphasised. ‘We’d be better off having the whole industry being able to use something going forward, whether it be for networking equipment, industrial applications, aerospace or whatever. The industry’s not going to create a consortium for Microsoft to stand on the outside and tell everybody to make this happen for us, so let’s step up and do it.’
Practical challenges
On-board optics should help companies like Microsoft retain and perhaps increase system density, explains Adam Carter, chief commercial officer at San Jose, California-headquartered optical component and subsystem vendor Oclaro. ‘With MTP optical connectors, for example, if you have on-board optics you can have many more channels coming to a small connector on the front panel,’ he said. ‘That footprint would enable companies like Microsoft and some of the web 2.0 players to reduce the total number of systems that they would have to deploy in major networks. That would give them economic benefits.’
Other potential advantages that could make board-mounted modules particularly useful for high-speed networking equipment include using different cooling options, including liquid cooling. Another benefit for companies like Microsoft is that instead of buying switch and optical modules separately and assembling them, they could come pre-assembled and verified.
Microsoft would like the option of exploiting COBO on-board optics when it reaches 400G inside its data centres, says Booth, which will be in ‘probably about two years’. The hardest aspect of COBO’s work towards achieving that deadline is deciding which connector should be standard, because different companies want to provide input, Booth observes. Other activities include considering how to take the optical signal from a board-mounted module to the faceplate and highlighting best practices and recommendations on how to cable the systems.
However, there are disadvantages, stresses Carter. ‘Say you have 140 customers coming out of various channels on a line card and one goes down, so one customer gets cut off – you have to take that whole line card out. So how do you deal with that?’ ‘You have to basically return the equipment,’ added Mark Nowell, distinguished engineer at networking equipment pioneer Cisco in Ottawa, Canada. ‘I don’t think any equipment supplier is going to provide warranties if a customer starts pulling components on and off the board.’
Microsoft has been very open that this isn’t a concern for its applications, but it might be for smaller companies, who may also be concerned about up-front costs, Carter emphasises. ‘Pluggable optics allow you to “pay as you grow”,’ he said. ‘It’s not clear that on-board optics would allow that. Microsoft tends to populate systems fully before it deploys them anyway, so there aren’t any empty slots. With other customers and market segments this might not be the ideal model to go forward, so there are a lot of discussions around that.’
Even in the ‘hyperscale’ world that Microsoft inhabits, not everyone is aligned to COBO yet. ‘It’s predominantly driven by Microsoft, but it doesn’t necessarily mean it’s going to be used by Google, Amazon and Facebook’ the Oclaro executive observed. ‘That’s where, from an industry perspective, we have to be a bit careful that we don’t get fragmentation of potential solutions going forward.’
While Oclaro was a founding associate member of COBO, these and other open questions mean that the company isn’t completely committed to developing a product that meets its standard. ‘The mechanical footprint that’s been developed doesn’t fit in a QSFP-28, which is obviously a very big disadvantage for us as a transceiver supplier,’ Carter revealed. He added that there’s been a lot more interaction on this aspect recently, and that he’s keen to see where that leads.
Carter’s ideal scenario is that a COBO standard will enable a new building block that could also serve as the optical subassembly in pluggable modules. ‘I’m hoping that COBO designs a footprint that we would be able to use in conventional transceiver packaging. That way we leverage economies of scale in our operations, which has benefits not only for the hyperscale guys but also all the way across the industry. People have maximum choice, whether or not they want to go with on-board optics if they have a huge density requirement or they can have a pluggable module and use a pay as you grow scheme.’
Oclaro would want to use its longer-range singlemode solutions in the COBO footprint, although other suppliers may also be able to incorporate shorter-range multimode technology. ‘We’ve been looking at various laser structures and approaches you could use to do the arrayed-type devices you would need,’ Carter explained. ‘Not necessarily edge emitters but also not vertical-cavity lasers. We already use the background technology on the receive side in some products today. But on the laser side, it’s a lot more complex to make sure you’ve got multiple singlemode channels all working to the same specification reliably and consistently. That’s why we’re a COBO member – to keep abreast of what’s going on, but also bring knowledge to the table about how semiconductor lasers, particularly on the singlemode side, would work in what would be pretty harsh environments from a temperature point of view.’
Crossover coming
In reaching for 400G data rates, it’s also important to consider silicon switch devices and their electrical signalling, because they use the optical modules to get data onto and off the line card, according to Nowell. ‘The signalling to do 400G is based on eight electrical lanes,’ said the Cisco executive, who was also a founding member of COBO. ‘High-volume pluggable optics modules are optimised for four lanes or one lane. The big question is how would you achieve 400G? Can you do it in the same pluggable density that you have with 100G, 40G or 10G? The electrical lane count can affect the connector and module size. When considering the technologies to support 400G interconnects, you also need to consider the space, extra power and thermal challenges in order to figure out whether different form factors are needed.’
This consideration led Cisco to establish the QSFP-DD multi-source agreement (MSA) together with Lisle, Illinois-headquartered interconnect provider Molex, with Nowell as a co-chair. ‘The switch chips that are going into 400G require eight lanes at 50G each and QSFP, which is the workhorse pluggable form factor for the industry, doesn’t support that,’ he commented. ‘With QSFP-DD, we came up with a way of not changing the form factor from the perspective of the faceplate while meeting these other challenges. You can still build networking switch configurations capable of supporting networking requirements, with the 32 ports of QSFP-DD, but we’ve modified the electrical connector at the back so it now supports eight lanes instead of four. If you have a double density connector on your equipment, you can also plug in a regular QSFP module into that slot, so the customers have backwards compatibility and the flexibility they are used to.’
The QSFP-DD MSA had to take steps to resolve the same heat problem that COBO faces. ‘We made some modifications, but not to the basic module form factor,’ Nowell said. ‘We optimised the cage. Standard QSFP allows you to cool up to 3.5W, and with this cage and form factor we think we’re able to cool up around 9W.’
With a foot in two camps on how to progress towards 400G networking, Nowell underlines that cost is almost the only criteria that matters. He points out that the adoption of on-board optics isn’t new. They have been used in core router multi-chassis products for service provider network applications, which Cisco has supplied since 2004.
‘Different generations have toggled back and forth between embedded and pluggable for the inter-chassis interconnects,’ Nowell explained. ‘The situation depends on what the lowest-cost solution was at the time. That solution was a high-end product so the system had built-in redundancy and the customer engagement in who fixed failures was very clear. In this new world where COBO is trying to go, the goal is to make it a commodity, to actually trying standardise it. That’s a necessary goal if this technology is going to take off in the industry and be widely adopted in low cost and high volume.’
The prospects for adoption look hopeful, if the response to COBO’s activities so far is anything to go by. Several interested parties from applications beyond network switches have expressed a strong interest, according to Booth. Consequently, COBO is moving from its original status as an unincorporated entity to becoming a non-profit organisation. ‘We’ve got one project right now which is focused on data centre networking,’ Booth confirmed. ‘We will probably take what we learn there and carry it over to industrial applications or telecom-based optics. It’s starting to build up momentum now where we’ll be seeing more people trying to take it in other directions.’
Meanwhile, back in the network switch world, Booth believes that people will struggle to find alternatives to the on-board optics approach. ‘We might be able to do pluggables for some 400G generations, but everybody’s realising they don’t have a roadmap past 400G,’ he said. ‘What happens when we hit 800G, when we hit 1.6T? Everyone’s like, ‘Whoa, we’re running out here!”’
Nowell agrees that on-board optics will be an important long-term solution for high-speed networking. ‘If I look forward 20 years in the future we’ll most likely be doing this everywhere,’ he said. ‘I don’t know when it will happen, but I’m confident that a crossover will occur away from pluggables. Is there going to be a complete crossover? I’ve learned never to bet against technology. With QSFP-DD, absolutely I think that people will want to try to take that beyond 400G. Can they do it? We’ll have to see. The form factors have kept up in every generation so far. But I do think COBO is starting its work at the right time.’
- Andy Extance is a freelance science writer based in Exeter, UK
