Breaking the Memory Bottleneck Part 2: How Tech Giants Shrink the KV Cache Footprint

Part 1 of this series looked at capacity expansion: how Enfabrica, Penguin Solutions, Marvell, and Meta are leveraging CXL to increase memory capacity for KV Cache. But expanding physical memory capacity is just one way to fight the bottleneck. Another way is to lower how much capacity KV Cache actually requires, through innovations in model architecture and algorithms.

For instance, researchers have rethought how many Key-Value vectors a model actually needs to keep around. Alongside that, a separate line of work has asked whether those vectors need to be stored at full precision at all. But even when KV Cache’s capacity requirements are lower, it can still be slow to read. Some companies are tackling this bandwidth problem directly, rethinking how models generate tokens in the first place.

Behind the paywall, we break down:

How Google and DeepSeek reworked attention mechanisms via GQA and MLA
How NVIDIA’s KVTC and Google’s TurboQuant quantize KV Cache down to a fraction of its size
How Google, Meta, and DeepSeek are racing to fix the decode bandwidth bottleneck, from MTP to DiffusionGemma
TrendForce View: what solving the KV Cache bottleneck means for memory demand

Related report: Overcoming the Memory Bottleneck: CXL Expansion and KV Cache Compression Innovations