Rethinking Memory For AI Inference

Key Takeaways:

• Moving compute closer to memory can reduce data movement, power consumption, and latency.
• Different AI workloads require different approaches to memory architecture, interconnects, and processing.
• Software support, standardization, and thermal management remain major barriers to wider adoption.

The shift from AI training in data centers to AI inferencing at the edge will require a fundamental rethinking of where and how data is processed and stored, how it is prioritized, and what needs to be traded off in terms of power and performance to stay within a limited power budget.

At the center of these efforts is a recognition that the explosion of data — especially in physical AI where data is being live-streamed from sensors — requires much of that data to be processed as close to memory as possible. This has been a topic of discussion since the inception of the Internet of Things, but it really began gaining traction with AI due to the flood of data and concerns about the amount of power required to move and process it. In battery-electric vehicles and drones, for example, the less energy spent on processing data, the greater the range per charge and the more energy that can be allocated to critical and not-so-critical functions.

The challenge is understanding what data needs to be kept, what can be discarded, and what needs to be acted upon. Those are very different priorities. Making those choices requires advanced processing, and the quickest path to results using the least amount of energy is to minimize the distance data needs to travel between processing and memory.

“The realities of these LLMs getting into physical devices are causing everyone to rethink how they architect their memory,” said Frank Ferro, group director for product management at Cadence. “Even with conventional memory — without even talking about processing in memory — how do I partition these models to maximize the performance both of the overall processor and memory subsystem?”

Solving those issues will require several new approaches, such as processing near memory (PNM) and processing in memory (PIM), as well as improvements across multiple technologies and approaches. Compilers and operating systems will need to catch up, and standards will need to be established.

“AI in the data center and all these large language models have been the focus of everyone’s attention for the last couple of years, and now these models have been trained and pushed out into the field, into the edge of the network and physical AI,” Ferro said. “The reality is you have limited compute power and limited memory.”

AI inferencing at the edge presents different challenges than training AI models in AI data centers.

“Training almost always involves highly variable memory access patterns, and mostly involves the use of interpreted code — for example, Python to PyTorch — which necessitates the complexities of coherence across layers of memory,” said Steve Roddy, chief marketing officer at Quadric. “Once inference begins, the use model changes dramatically. Whether you are looking at big LLMs running in a data center, or models running locally, such as smaller LLMs or VLAs (vision-language-action) in cars, the model is known before you run it. That means the data access patterns are known ahead of time. This is where ahead-of-time compilation can start to take great advantage of the various at-memory, processing-in-memory, or in-memory-compute (IMC) options. The system knows exactly where, when, and how the compute is going to occur. There is no need for one element of a compute solution to guess if something manipulated the data while it was in another memory subsystem, because the entire system orchestration was planned out hours, days, or weeks before, when the AI inference model was compiled.”

Certain stages of the AI inference pipeline benefit from near-memory solutions more than others.

“Pushing the compute into the DRAM or near the DRAM has interesting implications,” said Sharad Chole, chief scientist and co-founder of Expedera. “You’re moving your communication pattern — my activation goes to memory, rather than my memory goes to activation — and that changes how much communication you need to do. These sort of solutions are okay for low-batch decode. For example, if you have batch one and you are trying to do decode and you are not reading all the weights, as soon as we move to large-batch decode, which is cost-optimized, or prefill, which is large-batch execution, the activation communication becomes the challenging part. These are the two ways to think about it. In the end, you have to break your execution into parts that are either storage-sensitive, in terms of how much data you need to store, or bandwidth-sensitive, or batching-sensitive.”

Either way, bringing compute closer to data can reduce the time and the energy spent moving it, and processing in the same memory package (custom HBM), or on the same memory die (LPDDR6-PIM), provides two fundamental advantages. “One of them is performance,” said Kos Gitchev, product marketing director at Cadence. “If we’re looking at a CPU or a core sitting inside the SoC, and every time you need to do an operation you’ve got to go out to memory, there’s a transaction that says, ‘Okay, get me some data from the memory that’s going to hit a cache somewhere in the SoC.’ That will eventually go to memory. It will come back to the cache and eventually come back to the processor. This operation really takes some time.”

So while chip architects can get compute running quickly on an SoC, the SoC has to wait for data to arrive. “Processing in memory to a large degree addresses this,” said Gitchev. “Now you can say, ‘I have this chunk of memory data in the memory, go do something with it, and I’m just going to go read the results. I’m going to skip this whole step of reading it, compute again, and putting it back. I’m just going to do a read, and that’s all I need.'”

Swipe left to view all columns →

Fig. 1: Compute-in-memory vs. processing-in/near-memory — where the math happens and what each term means. Note that CIM and IMC are used interchangeably in hardware to describe architectures that perform computational tasks directly within the memory array, but IMC also has a separate software meaning. Source: Semiconductor Engineering

Reducing data movement
Ultimately, the entire system needs to be optimized to make this scheme work.

“The framing has shifted from, ‘We need more memory,’ to, ‘We need fabric that keeps memory fed without starving other traffic,'” said Nandan Nayampally, chief commercial officer at Baya Systems. “More recently it’s, ‘How to architect around scarcity itself.'”

Others agree. “AI systems are increasingly limited by the ability to move data efficiently between compute, memory, accelerators, chiplets, and dies,” said Andy Nightingale, vice president of product management and marketing, at Arteris. “This makes the interconnect and memory architecture as important as the memory device itself. This reinforces the need for scalable NoC, chiplet, cache coherence, and system mapping technologies that help customers optimize data movement before silicon.”

Interconnects, caching schemes, and packaging all play increasingly significant roles at the edge, but choosing the right combination while also keeping costs down is a challenge. So while new versions of high-bandwidth memory are enabling more processing in the base die of a multi-die stack, an LPDDR6-PIM that places slices of compute alongside the DRAM banks may be more attractive, depending on the cost.

“Near-memory computing is still using an external channel interface,” said Hoshik Kim, senior vice president and fellow at SK hynix.[1] “But by putting processing units near memory data, we can actually get rid of or remove a lot of data movement and have higher energy efficiency. Also, we can offload the compute function from the CPU or GPU, so that we can have better utilization of the host CPU or GPU.”

Typical use cases involve memory controllers inside the base die. “We are integrating some useful near-memory processing units to offload simple MAC operations,” said Kim. “Because, if you think about inference workload operations, most function computation is a simple MAC operation. So we could put MAC operations inside the base die to expand for and then work with the GPU for a more complex, compute-intensive operation.”

Multi-year process
Some of the necessary technologies are available today. Others will take years to develop, or improve enough to the point where they can make a difference.

“We are looking at in-memory applications in three to five years, maybe around 2030,” said Vishal Chandrasekar, director of product management at Ayar Labs. “You want to be able to connect up many GPUs so that you are able to have different GPUs access the memory of others. People are trying to ramp up the amount of memory per GPU, but you’re limited by packet size and HBM bandwidths and the like. Compute in memory is one path, which makes it so the base die of your memory now has compute capabilities of some kind.”

Still, that technology is not quite ready for commercialization. “It’s a little aggressive to say that we’re already seeing it out in the market,” said Cadence’s Ferro. “There might be some very specific embedded applications that are using a type of processing in memory, but as a broad usage, it’s still pretty early.”

Processing-in-memory solutions are slowly moving from research to commercialization. For example, Samsung announced the architecture and performance benchmarks for its LPDDR6-PIM chip aimed at low-power, mobile on-device AI applications.

“LPDDR-PIM is the one that has got the most attention within JEDEC,” said Randy White, memory solutions program manager at Keysight. “There is active development and discussion around LPDDR-PIM, but there are challenges for both the hardware and software. From a hardware standpoint, you’re mixing logic process with DRAM process, and they’re optimized for different metrics, so that innovation itself makes the manufacturing process more complex and expensive. You have to be very selective and make sure you know your application can sustain the added cost of that.” Applications for LPDDR6-PIM will range from fixed-node edge servers optimized for performance to mobile phones and tablets optimized for efficiency.

Trickier than hardware?
The software challenges may be even more difficult. “How do you manage workload skew and load balancing?” White noted. “Let’s say you take a checkerboard-style memory array or core, and every 10 columns or so of memory, you stick a vertical slice of logic silicon. If you were to take that PIM die, put it under a microscope, you’d see a checkerboard with slices repeated every certain number of rows. That by itself has some timing constraints. While you’re accessing one row, you have to wait for calculation to an instruction to be complete on the compute engine. So it’s not just a matter of fetching data and balancing timings between different bank accesses. Now you have memory access coupled with waiting for compute data. The whole point is that I don’t have to move data as much. I’m moving data between adjacent cells, and that’s great. From a hardware standpoint, you get lower power, etc. But for the engineer who’s writing the BIOS, who has to create the scheduler for synchronizing, that has just exploded in terms of complexity.”

PIM architectures need to be considered in terms of the impact on the rest of the system. “The hardware has to be designed and optimized so you have the right number of compute coefficients and things like that,” said White. “Then the memory has to be segregated in a way that you don’t have too fine resolution, so you have the right coarse and fine trade-offs in terms of separation.”

Fig. 2: What must be solved for near-data computing to achieve its potential. Source: SK hynix/DAC

Additionally, memory programming models are still in flux, particularly with the rise of non-traditional memory systems such as PIM, CIM, CXL-attached tiered memory, and heterogeneous memory architectures.

“I don’t think the industry will settle on a single programming model and coherence implementation,” said Steven Woo, fellow and distinguished inventor at Rambus. “Over the last 40 years, different programming models, coherence protocols, and consistency models have been implemented that provide different trade-offs to systems designers and application developers. The diversity of architectures and coherence implementations is driving the need for more high-performance libraries to be developed that allow programmers to call common functions that are implemented in ways that are optimized for different systems. High-performance libraries maintain a level of abstraction that improves program portability and performance scalability without burdening the programmer with understanding all details of the underlying hardware.”

Compiler and memory controller considerations
Software innovations are needed to support new memory architectures.

“The question is no longer whether a PIM is viable or makes sense,” said SK hynix’s Kim. “It’s a reality. The problem is how to make a software compiler framework and system software. All of the ecosystem needs to follow up.”

Cadence’s Gitchev agreed. “The real challenge is that currently the systems are designed so that the compilers, operating system, and everything works with a processor core or an accelerator sitting inside your SoC, and everything is designed for this processor to do all the work,” he said. “That’s the infrastructure. Now, essentially, you can offload some of that processor work and say, ‘I need to do this operation, and I don’t really need the processor to do it.’ There’s a whole infrastructure behind that to be able to do that. There’s a software aspect. The operating system and the processor need to understand, ‘These operations I can send to memory, they’ll do it, and I’ll just get the results, but these operations I still have to do locally because the memory I have may not be able to do it, or my system is not as efficient.'”

There’s also the need for standardization inside the chip. “How do you connect these things together?” said Gitchev. “Today we don’t really have a good way to say, ‘Go and do this operation in memory,’ as far as the type of transactions that move from the processor to the SoC fabric to memory. This infrastructure is not there. We need the hardware and protocol to be able to tell memory to go do this stuff, and then we need the software and the operating system to support the system to be able to do this and decide what’s efficient.”

This comes down to two streams of data. “I’ve got a stream of data that’s storing and reading and writing,” explained Ferro. “Then I’ve got a stream of data that’s compute, and that’s not native to the way operating systems and processors are built. You now need cooperation with the processor companies, the interface between the processor and the fabric, the system, and then of course the operating system. All those pieces need to get adjusted. It may or may not be a big adjustment, but without standardization, everyone’s system is going to be highly customized.”

Home agents and memory controllers
Near-memory and in-memory architectures introduce distinct pressures to traffic management and system coherence.

“In traditional hardware coherent systems, home agents are responsible for tracking where shared data is being used throughout the system and making sure processing elements have correct copies of the data,” explained Rambus’ Woo. “Hardware coherency eliminates the need for programmers to manage the updating of data across processing elements, simplifying programming. Synchronization primitives like locks and barriers, and software coherence give applications an additional level of control over how and when data is updated. Together, these hardware and software mechanisms enable efficient algorithms to be designed that maintain a level of hardware abstraction that hides the details of the underlying hardware, easing the task of programming and managing the way data is updated to maintain coherence in systems.”

That means software engineers increasingly need to understand the hardware, and vice versa.

“Achieving good, scalable performance as problem sizes grow and the number of processing elements increases is becoming more difficult,” said Woo. “While home agents remove the burden of managing coherence from the programmer, the messages sent by home agents to maintain coherence can become performance bottlenecks. Programmers increasingly need to understand the underlying structure and functions of the hardware to achieve good performance. Algorithms are often modified to shard data in ways that maximize the independence of tasks and minimize the amount of coherence traffic traveling in the system. In some systems, utilizing no hardware coherence mechanisms results in the best performance, but complicates the task of programming.”

The memory controller faces a twofold challenge. “One part is the memory subsystem, which consists of a PHY and a controller,” said Cadence’s Gitchev. “The controller itself is where these commands need to actually do the work. But now you have a different set of commands. Before that, you fundamentally had two different commands — read data, write data, and that’s about it. The control is a lot more complicated than that, but at a high level, that’s what it does. It reads data and writes data in the most efficient way possible. Now a different challenge is coming in where we need a different set of commands and rules to say, ‘Hey, now can you go take this data and multiply it, and take this data inside the memory and divide it, or do an operation.’ It’s not a significant challenge for a controller to do that. We add more commands, and we’ll figure out how that works, and try to make it efficient.”

The second challenge is harder. “The processor needs to be able to send those commands,” said Gitchev. “It needs to prepare a memory inside the memory subsystem to say, ‘If I’m going to multiply this data by that data, do I have the right data in the right place? Do I have the results in the right place? This operation is complete.’ The controller manages some of that, but it also has to have very close collaboration with the overall system.”

Thermal, physics, and CIM
Processing near memory can help reduce data traffic, but it could amplify existing physics problems, such as thermal issues.

“The memory shortage is partially due to current state-of-the-art design and use of available memory,” said Satish Radhakrishnan, head of global field applications and solutions at Vinci. “Innovative design can help reduce demand per unit and improve utilization. The most viable solutions to increasing memory utilization — including higher HBM stacks, custom HBM, near-memory, in-memory compute, and chiplet-based disaggregation — all share one property. They trade a bandwidth problem for a physics problem. Stacking memory closer to compute concentrates heat in exactly the places it’s hardest to remove. Heterogeneous material stacks and micron-scale interconnects create thermo-mechanical stress and warpage behavior that doesn’t respect the boundaries between die, package, and board. Heat doesn’t stop at the chip, and neither do the constraints that come with it.”

For this reason, memory companies tend to announce new thermal technology with their new architectures. For example, SK hynix announced an integrated HBM (iHBM) solution that embeds integrated cooling elements (ICEs) within the HBM package. Micron’s recent roadmap included vertical HBM (vHBM) with thermally aware design and a heat spreader over the PHY. Samsung is tackling heat issues with a range of solutions, including hybrid copper bonding (HCB) and a heat path block (HPB), which it plans to include in HBM5. [2]

SK hynix’s Kim proposed in-memory compute to reduce heat in the HBM base die. “If you put DRAM on top of logic, DRAM is very sensitive to temperature,” he said. “Logic is very prone to hot spots. Basically, the hot spots happen at the bottom of memory. Thermal can be a real challenge, and you need to resolve these issues. In-memory computing can be an answer. It has extremely low power consumption, so you can probably put in-memory computing at the base die or the bottom of memory.”

In practice, IMC/CIM could work with custom HBM4 or HBM4E because the base die is built on a foundry logic process. SRAM is the native memory, meaning that SRAM CIM macros can be built there. CIM does multiply-accumulate inside the SRAM array, so weights aren’t repeatedly moved between memory and a separate MAC unit. Data movement is where most of the energy goes, which is why SRAM CIM macros report far better energy efficiency, in operations per watt, than conventional digital accelerators. Less energy per operation means lower power density, so fewer hot spots under the DRAM.

Conclusion
Software support remains the missing link between near-memory hardware and its promise for AI. For compilers and AI frameworks, processing-near-memory and in-memory architectures are emerging targets that could let developers offload mathematical operators directly to the memory subsystem, but the software needed to support that shift is still catching up.

The available compute — and the limits on it — differ by architecture. LPDDR6-PIM is limited to MAC units or simple ALUs suited to matrix-vector multiplication and element-wise math. A custom HBM4/HBM4E logic-process base die can add other memory-control functions. But that flexibility comes with a thermal ceiling — how much compute fits depends on the heat it adds beneath the DRAM.

“We have spent a lot of time over the last two years talking about custom HBM and how to use that base layer,” noted Cadence’s Ferro.

AI is driving the push to bring compute closer to memory, but proximity alone will not deliver the gains. Broader adoption depends on matching workloads to the hardware, managing heat, and giving compilers and operating systems the tools to decide what to offload. Moving the compute is only part of the solution; making the system use it efficiently is the next test.

References

1. DAC 2026 Research Special Session: “Near-Memory Processing is Becoming a Reality” (SK hynix)
2. Cooling design emerges as battleground for next-gen AI memory (Korea Times)

Related Reading
How Will The Custom HBM Business Work?
There is no one-size-fits-all — yet.