Semiconductor Engineering examines how open-weight AI models affect chip architecture, noting the focus is shifting from peak TOPS to memory bandwidth and capacity, sustained throughput, numerical format flexibility, and deployment security.
In its AI Ecosystem series, SK hynix notes that as AI shifts from pretraining to inference and agentic AI, the system bottleneck is moving from GPU compute to where data resides and how it flows. The memory wall proposed by Wulf and McKee in 1994 is becoming reality: AI accelerators now reach petaflops of compute, but memory read speeds cannot keep up, leaving many compute units idle waiting for data. The response is to increase memory bandwidth or move memory closer to the processor.
TrendForce analysts explain three paths tech giants are taking to compress the KV Cache through architectural and algorithmic innovation, with Google and DeepSeek respectively reworking attention mechanisms via GQA and MLA.
In the first half of 2026, the core constraint of AI inference has shifted from compute to memory, as surging KV Cache demands meet limited memory supply and become the main bottleneck. For Llama 3 70B FP16, model weights are about 140GiB, but the KV Cache reaches 160GiB at 64K context and balloons to 320GiB at 128K, about 2.3 times the weight size.