[Tech Note] Why does AI still have to wait with tens of thousands of GPUs? How silicon photonics and CPO are building the ‘highways of light’

AI doesn’t wait. The foundations of every industry are already cracking. This isn’t a story about one sector. Manufacturing. Finance. Healthcare. Data centers. The entire industrial order is being rebuilt around AI — and it’s happening now. Knowing about AI is no longer a competitive advantage. You need to see through it. That’s why our Newsroom is putting AI — and the semiconductors that power it — under the microscope.

In [Tech Note], Professor Younghyun Kim of the School of Semiconductor Convergence Engineering at Hanyang University ERICA breaks down data bottlenecks in AI factories and explores silicon photonics and co-packaged optics (CPO) as next-generation technologies that could help overcome them.

The competitiveness of AI semiconductors is no longer determined solely by the processing speed of individual compute chips. The performance of an entire AI factory depends on where data resides, the paths it takes, and how quickly it reaches its destination. In the AI era, the data highway is shifting from electrical signals to light.

Imagine a vast industrial complex filled with hundreds of factories, each equipped with state-of-the-art machinery capable of producing goods in the blink of an eye. But what happens if the roads connecting these factories have only one lane in each direction? Trucks carrying raw materials line up outside warehouses, while finished products cannot reach the next stage of production on time. The factory machinery may have become faster, but that does not necessarily mean that production speeds increase at the same pace. That is because the production speed of the entire industrial complex ultimately depends on the capacity of its roads. A similar situation is unfolding in today’s large-scale AI data centers, often referred to as “AI factories.”

▲ No matter how many computing devices an AI factory has, GPUs will be left waiting for data if the data transfer pathways are too narrow.

Graphics processing units (GPUs) excel at performing the vast number of calculations required for AI simultaneously. However, they cannot begin computing until the necessary data arrives. They must retrieve training data and “weights” — the values learned by an AI model during training — from memory. When multiple GPUs work together to train a single model, they must also continuously exchange computation results. As GPU speeds become faster, the data transfer capabilities of the surrounding memory and networks are growing increasingly important.

Until now, copper interconnects have been used to connect the semiconductor components within servers. Copper is easy to work with and efficiently transmits signals over very short distances, such as between chips on the same package substrate. The challenge arises as transmission speeds increase and distances grow. Electrical signals weaken and interfere with one another as they travel through copper wires. Repeaters* and signal correction circuits are needed to restore degraded signals, requiring additional power that, in turn, generates more heat and increases cooling demands.

* Repeater: A device that receives a weakened signal during transmission, amplifies or regenerates it, and retransmits it.

▲ Copper is efficient over short distances, such as within semiconductor packages. However, as transmission speeds and distances increase, signal loss, heat generation, and repeater power consumption also rise, making optical interconnects that carry multiple wavelengths through a single fiber increasingly advantageous.

To reduce signal loss and power consumption, optical interconnects that transmit more data using light are gaining attention. A single optical fiber can carry light at multiple wavelengths simultaneously. By using wavelength division multiplexing (WDM)*, which assigns different data streams to different wavelengths, transmission capacity can be increased without adding more optical fibers.

* Wavelength division multiplexing (WDM): A technology that transmits different data streams simultaneously through a single optical fiber by assigning each stream to a different wavelength, or color, of light.

Although optical fiber is gaining prominence, this does not mean copper is going away. Electrical connections through copper remain advantageous within chips and over very short distances. Optical interconnects have already become standard for longer-distance connections, such as those between racks and buildings in data centers, and are now moving into shorter-distance applications.

The key is not a competition between electricity and light, but a division of roles. The goal is to build “highways of light” that can transmit more information more efficiently, starting with connections where copper interconnects face increasing signal loss, power consumption, and heat generation.

If optical fiber is the highway, infrastructure is also needed to put data onto that highway, direct it to the right destination, and retrieve it at the other end. Silicon photonics brings this infrastructure onto a small chip. Inside the chip are microscopic pathways called optical waveguides, which guide light through the principle of total internal reflection, enabled by differences in refractive index between the waveguide and surrounding materials. Alongside these waveguides are modulators and photodetectors fabricated using semiconductor processes. Modulators encode information onto light by changing its intensity or other properties according to electrical data, while photodetectors convert incoming light back into electrical signals to retrieve the information.

▲ Silicon photonics directs light entering through an optical fiber into waveguides within a silicon photonic integrated circuit (PIC), where integrated components such as ring resonators, modulators, and photodetectors control the light. Within this small chip, electrical signals can be encoded onto light, while incoming light can be converted back into electrical signals to retrieve the information.

The significance of “silicon” goes beyond the material itself. The microfabrication and mass-production techniques developed by the semiconductor industry can also be applied to optical devices. This makes it possible to fabricate numerous optical pathways repeatedly with high precision in a compact area and integrate them close to electronic circuits.

However, controlling light and generating light are two different challenges. Silicon is not well suited to efficiently generating light through electrical current. For this reason, communication lasers use compound semiconductors such as indium phosphide (InP). One approach is to place these lasers outside the silicon photonics chip and supply light from an external laser source (ELS). This allows lasers to be positioned away from electronic chips that generate substantial heat, while their modular, replaceable design also simplifies maintenance.

How close, then, should these optical chips be placed to the electronic chips that process data? One answer to this question of placement is co-packaged optics (CPO).

A data center switch acts as a traffic controller, directing data exchanged among numerous servers to its intended destination. Along the front panel are rows of optical transceiver modules that convert electrical signals into light and vice versa. These “pluggable” modules can be inserted and removed as needed. This design makes it easy to replace individual faulty modules and benefits from extensive operational experience. However, electrical signals must still travel across the circuit board from the application-specific integrated circuit (ASIC), the brain of the switch, to the optical modules on the front panel. As switch capacity and port speeds increase, this seemingly short electrical path becomes a growing challenge for power consumption and signal integrity.

Co-packaged optics (CPO) places the optical engine, the core component responsible for converting electrical signals into light and vice versa, directly next to the switch ASIC. Both components are positioned on a common package substrate, which serves as a shared platform for mounting and electrically connecting chips. It is like relocating a logistics terminal from a distant location to the same site as the factory itself. The distance that electrical signals must travel becomes shorter, while optical fiber handles the longer transmission path. CPO can increase bandwidth density — transmitting more data within a limited switch area — while also reducing the power otherwise spent compensating for high-speed electrical signal loss.

▲ In the pluggable architecture on the left, electrical signals travel from the switch ASIC to the front-panel optical modules. In the CPO architecture on the right, the central switch ASIC and surrounding optical engines are placed on a common package substrate, significantly shortening the electrical connection distance.

CPO requires the co-design of optical devices and the electronic circuits that drive them and process their signals. Silicon photonics offers the advantage of close integration with electronic chips by leveraging existing semiconductor manufacturing infrastructure and advanced packaging technologies. This also supports system-technology co-optimization (STCO), an approach that jointly optimizes everything from devices and manufacturing processes to packaging and systems.

However, supplying laser light to the chip and precisely connecting optical fibers remain technically challenging. Optical devices must also operate reliably alongside ASICs that generate substantial heat. Because a defect in a single component can affect the entire package, it is also important to improve yield — the proportion of manufactured products that meet quality requirements — and design architectures that make defects easier to detect and repair.

Despite its technical challenges, the reason AI companies are turning their attention to CPO is clear: To fully harness the computation power of GPUs, network efficiency must also be improved. Whether GPUs are tightly interconnected to function as a single large computer or connected across multiple servers to scale up computing capacity, adding more GPUs inevitably places a greater burden on the interconnects linking them. In response, global tech companies such as NVIDIA and Broadcom are accelerating commercialization by introducing network products incorporating CPO technology.

Results demonstrating the technology’s effectiveness are already emerging. At the 2025 European Conference on Optical Communication (ECOC), Meta presented evaluation results for a CPO switch system. In the tests, power consumption in the optical interconnect portion was approximately 65% lower than that of the comparable pluggable architecture. Extended operation also confirmed connection stability under the specific test conditions*.

Alongside CPO, alternative approaches are being developed to retain the advantages of conventional architectures. Linear pluggable optics (LPO) maintains the removable optical module design while reducing power consumption by eliminating the digital signal processor (DSP) traditionally used to compensate for signal distortion. Near-packaged optics (NPO), meanwhile, places optical engines close to the switch chip without integrating them into the same package.

▲ LPO retains the pluggable optical module architecture while eliminating the DSP inside the module, whereas NPO places optical engines close to the switch chip but outside its package.

CPO, LPO, and NPO are not sequential stages of technological development; rather, they are distinct approaches selected according to performance, power efficiency, cooling, and maintenance requirements. Pluggable architectures offer the advantage of easy replacement, but as large-scale AI systems demand greater bandwidth and power efficiency, CPO becomes increasingly important because it shortens electrical connections to the package level.

As these approaches continue to coexist, CPO adoption is expected to expand first in applications where its power-saving benefits and operational reliability have been demonstrated under real-world conditions.

High Bandwidth Memory (HBM) and CPO reduce the time GPUs spend waiting for data at different points in the system. HBM stacks multiple DRAM dies and places them directly next to the GPU. Within the same package, HBM connects to the GPU through short, wide electrical pathways using an interposer*. CPO, by contrast, improves data transfer efficiency across the network that connects multiple GPU systems by converting electrical signals into light close to the switch chip. Even if HBM supplies sufficient data to GPUs locally, overall performance cannot be fully realized if communication between GPUs becomes a bottleneck. This is why both memory and networking capabilities must work together.

* Interposer: An intermediate substrate used to interconnect multiple chips (e.g., CPUs, GPUs, and memory) within a single package.

▲ HBM reduces the distance that data must travel within the same package as the GPU, while CPO improves data transfer efficiency across networks connecting multiple GPU systems.

In addition to delivering data quickly, it is also important to make efficient use of memory capacity distributed across multiple systems. When memory is allocated separately to individual systems, some may run short of capacity while others have unused resources. One approach to addressing this imbalance is memory pooling, which combines memory resources into a shared pool that multiple processors can access as needed. Compute Express Link (CXL) is an interconnect standard that supports memory expansion and pooling. In this context, the role of light could extend beyond GPU-to-GPU communication to the pathways through which processors access shared memory pools. Using silicon photonics-based optical interconnects for these connections could enable longer transmission distances and greater bandwidth while potentially reducing power consumption.

However, this does not mean that HBM located directly next to the GPU must also use optical connections. A hybrid architecture is possible: Data requiring fast access can be stored in HBM connected through short, wide electrical pathways, while data requiring additional capacity or shared access can reside in optically connected memory pools. Optical interconnects, however, do not make remote memory accessible at the same speed as HBM. It is therefore essential to evaluate where different types of data should reside, when they should be moved, and the latency and power consumption involved in those transfers.

Looking ahead, the competitiveness of AI semiconductors will depend not only on the raw computing speed of individual GPUs but also on how closely GPUs, memory, and networks are co-designed and integrated with the optical, packaging, and system technologies needed to implement them. Electrical and optical connections will coexist, each serving the applications where its advantages are most effective.

Ultimately, the most powerful AI factory will not be the one with the fastest individual GPU, but the one where numerous GPUs and memory devices can work together without waiting for one another. This is precisely why semiconductors need light.

Disclaimer: The opinions expressed in this article are solely those of the author and do not necessarily reflect the official position of SK hynix.