Agentic AI 改变 CPU 与 GPU 的算力平衡

Key Takeaways:

• Agentic AI, reasoning models, mixture-of-expert architectures, and reservoir computing are changing how AI workloads use compute resources.
• Agentic AI is particularly popular, and because it requires more CPU, CPU supply is coming under pressure.
• Mixture-of-experts models are becoming very popular, enabling specialized queries that require fewer parameters to load.

Convolutional neural networks (CNNs) and large language models (LLMs) have dominated commercial AI for some time. Proprietary recommender models for companies such as Meta have also had significant societal impact even though only one company employs each such model. GPUs perform the heavy lifting for many of those types of models. But agentic AI needs more orchestration, and that’s a job for CPUs.

When AI’s only job is to run a model, it turns to GPUs or dedicated neural processing units (NPUs). Agents still use these models, but each is only part of the solution. Those agents must make decisions based on what the models return, and that requires if-then-else logic, and that’s what CPUs are good at. GPUs, not so much. According to some industry estimates, agentic workloads could alter the ratio of CPUs to GPUs from 1:4 to as low as 1:1.

Newer model types are also emerging, most of which are still based on LLM fundamentals. Unlike agentic AI, reasoning models, mixture-of-expert (MoE) models, and reservoir computing still rely on GPUs. But they’re also ripe for use by agents.

Pixels and parsing
For several years, practical, commercial AI meant CNNs and recurrent neural networks, or RNNs. The former were good at analyzing camera input, while the latter were proficient at analyzing sound and language. The second “L” in LLM means “language,” but LLMs do much more than language today. The notion is that some piece of text is abstracted into a token, and now practically anything can be a token.

The benefit of LLMs over RNNs is that they can maintain an arbitrarily long context, making them better at picking the right words when interpreting and generating text. In addition to requiring fewer resources, RNNs can analyze incoming frames of sound immediately without waiting to establish context, improving real-time response.

The challenge with early LLMs was that they had one job — to predict a likely next word in a sequence. That word had to make sense syntactically, but not necessarily semantically, and it certainly didn’t check facts to ensure correctness, hence the frequent “hallucinations.”

One way to improve that was through retrieval-augmented generation (RAG). Instead of just figuring out a word that could maybe work, models could access “embeddings” that contained real facts, helping generate text that was both syntactically, semantically, and factually correct.

Secret agent man
Today’s hot topic is, of course, agentic AI. An agent is a new entity, not one that replaces or modifies an LLM (or any other model). It effectively represents a higher level of abstraction, acting as an orchestrator of models. It’s not a model itself, although it must be trained.

“There is some new research out there that says agentic inference is going to dwarf everything else put together before it,” said Ed Doe, CEO and co-founder of Delos Data.

The whole point of an agent is that it does more than find an answer to a question. It does something with that answer. What it does will likely depend on the answers it’s received so far. In other words, it can send tokens to a model and, when it receives the result, decide what to do next, embarking on a sequence of decisions and actions.

A well-trained agent is meant to handle a set of tasks without tight supervision. “If, as a leader, you have to tell your team every step to accomplish, you’re going to be more consumed with every step of the way,” explained Rob Knoth, product management director, digital and signoff group at Cadence. “But if you say, ‘I need you to go over there and get this work done,’ then you can go focus on another task, occasionally looking over to make sure that things are going well.”

This is a new space. “Since last year, agentic AI took off,” noted Satadal Bhattacharjee, global head of cloud and AI infrastructure silicon at Arm. “Now you can say, ‘I want to go to Costa Rica for 10 days. I need hotels, car rental, and entry tickets for these parks. Here is my budget. Go and reserve all of that.’ You give it an outline, just like to a travel agent, then the agent calls APIs to take action on your behalf.”

A new job for CPUs
An agent’s decisions typically involve if-then-else branching, and that’s not what GPUs are good at. CPUs do far better at such decision-heavy code, so they’re becoming a more important part of the computing cluster. For typical LLMs, they mainly serve as hosts, setting things up and managing communication through PCI for things like scale-out. That’s a relatively light CPU load, allowing it to handle host duties for multiple GPUs at the same time. The ratio of CPUs to GPUs will vary by system, but it’s around 1:4 for typical LLMs.

With agents, CPUs will be far busier doing actual work (rather than just enabling the GPUs to do work). So while the GPU has been the belle of the ball for a long time, CPUs are getting renewed respect as the CPU: GPU ratio moves closer to 1:1. That will also raise demand for CPUs, which can nudge prices upward.

“The challenge used to be how many AI accelerators we can get per CPU,” said Ron Lowman, strategic marketing manager for IP at Synopsys. “With agentic AI, there’s more CPU load demand, and therefore the ratios have changed.”

Bhattacharjee agreed. “Demand for CPU has risen a lot because everyone is looking for a lot more compute and in a power-efficient way to run these agents,” he said.

“In the last 18 months, everyone is clamoring over agentic AI,” said Yong Meng Sua, CTO at QCI. “This is why CPUs are getting very expensive right now.”

“The CPU is back, but is being used in a very different way than it was used in the rest of the data center,” Doe said.

Agentic queries are also more compute-intensive because, instead of just one query, sending a request to an agent typically results in many queries as the agent completes its series of tasks.

More advanced agents must also understand sequencing, and that can be programmed. “There is a programming part as well,” Arm’s Bhattacharjee said. “You can run JSON scripts to tune it and sequence it. There’s a lot more that can be done with agents.”

Show your work
The other challenge with early models is that humans can’t understand them. Models are a big cloud of numbers, and no single number can realistically be tied to anything physical or factual. The weights achieved during training aren’t sacrosanct. Training starts with random weights that get refined over many passes. Start with a different random assignment and the training will evolve to a different — and yet still correct — model.

So how do you prove that your solution is correct even if it is one of many possible correct models? There’s no deterministic way to trace how a basic model performs its calculations, making it hard to certify systems that must, for example, be safe. And the notion of safety is expanding as frontier models behave badly and escape their sandboxes, for example.

Reasoning models use additional inferencing before generating answers to queries, and in some cases they provide an accounting of how they arrived at an answer. “Reasoning is where it uses its knowledge of the topic, the context from previous discussions that you had with the model, and your knowledge of the person that is asking,” said Bhattacharjee. “So reasoning is where the model now does a next level of thinking and comes up with a better response. It’s more holistic in terms of covering a wider spectrum rather than just answering the question that is being asked.”

Reasoning models are also subject to training, so if a particular step in the reasoning was incorrect, that feedback can improve performance on future queries.

The reasoning element involves calculations better suited to GPUs than CPUs. “Reasoning is typically part of model inference and runs primarily on GPUs or other AI accelerators,” said Bhattacharjee. “Thinking involves generating additional tokens and neural-network computation that is best done by accelerators.”

Reasoning models have been commercially released. “OpenAI’s O-series models and DeepSeek-R1 are examples of commercially deployed reasoning models,” added Bhattacharjee.

Crowdsourcing (or cloudsourcing?)
LLMs are enormous models, and running each token through the entire model is resource-intensive. Two approaches attempt to tame the extent of these models.

The simple one is a small language model (SLM). Instead of trying to capture all of the internet and human knowledge in one model, it focuses on a specialty and is very well trained on that specialty, but on nothing else. An SLM is a self-contained model that can run independently.

Then, an increasingly popular technique segregates portions of individual model layers as experts in a mixture-of-experts (MoE) model. On a given layer, one can establish multiple experts that are independent of each other. Each has its own set of weights, making it better at dealing with certain tokens.

“Some of the model’s dense feed-forward network (FFN) layers are replaced by multiple separate FFNs, or ‘experts,’ together with a learned router,” explained Bhattacharjee.

“Earlier we had this scaling in large language models where if you just added more parameters, you got better performance,” explained Priyank Shukla, director of product management, interface IP at Synopsys. “With a mixture of experts, you not only have one large parameter model, but you have expert models, so you don’t need 10 trillion parameters for the whole process. You split it, and you have, let’s say, 1 trillion, which will do, let’s say, legal work. 1 trillion will do math work. And the front-end model will give the query to the right model.”

There’s a generic model that says, ‘Let me understand what question is being asked,’ called the pre-fill phase. “During that phase, it parses out your prompt to understand the context,” said Bhattacharjee. “Then there’s a series of experts that it works with. Once it understands that your question is related to, say, healthcare, there’s an expert trained on healthcare.”

A gating function or router decides, for a given token, which expert to send it to. Theoretically, it could send it to all experts, but typically only one or two get a shot at it. Training happens in the usual way, but the router is also trained by watching which experts perform best on different tokens during training. That allows it to direct tokens appropriately during inference.

The context reflecting everything that’s happened so far in the inference process is stored in the key-value cache (KV cache). It gets larger as processing continues and drives future decisions. This will be sent to the selected expert.

“We have to transmit the KV cache from the first model, which interacts with the relevant mixture of experts,” said Synopsys’ Shukla.

The router also executes on the accelerators. “The routing or gating function is part of the model itself and generally executed on the accelerator alongside the model,” Bhattacharjee noted.

Finally, the segregation of experts happens at training. “The experts are trained jointly and aren’t normally manually assigned,” he said.

That distinction becomes clearer when MoE is compared directly with the other specialization strategy: small language models.

SLMs vs. MoE
SLMs and MoEs are distinct in two critical ways. SLMs can process an entire string of tokens, whereas a given expert will handle only certain tokens within a string. SLMs are smaller models and require fewer resources to run. MoEs make bigger models. An ordinary LLM would have only one expert (on everything), while an MoE provides multiple experts for the same task, making the model larger than its pure LLM equivalent. But it’s still likely smaller than what would be necessary if the entire model had to be trained to cover all areas without experts.

Fig. 1: SLM vs. MoE. A specialized LLM can be condensed into an SLM, whereas a MoE model has multiple options for critical parts of the model, with a router that decides which “experts” will handle the next token. Source: Bryon Moyer/Semiconductor Engineering

In addition, an MoE model can have broader coverage, with embedded experts covering a wide range of topics. An SLM covers only a narrow range and is therefore more limited in the material it can consume.

MoE models tend to be larger than regular “dense” models, because one critical feed-forward layer may be replaced by several experts. That means the total parameter count may exceed that of a standard model, but since only a few experts run at any given time, not all of those parameters need to be fetched.

“An MoE model can have a very large total parameter count because it contains multiple experts, but only a subset of those experts is activated for each token,” explained Bhattacharjee. “That sparse activation is the key advantage. It allows the model to increase its overall capacity without requiring every parameter to participate in every inference operation.”

Less clear are the power implications of MoE models. “Although an MoE model may contain more total parameters, only a subset of its experts is typically activated for each token,” said Bhattacharjee. “That means it can require less computation per token than a dense model with a comparable total parameter count. The tradeoff is that routing, memory access, and communication between devices can introduce additional overhead, so efficiency increasingly depends on the design of the overall system rather than parameter count alone.”

Random reservoirs
A rather obscure type of model is called reservoir computing. With a limited remit for handling time sequences, it’s much easier to train than typical models since most of the model is random. The training process affects only the last layer, which takes whatever the random bits produce and maps it to a correct answer.

“It’s a specialized approach for dynamic temporal inputs, particularly for low power, fast adoption, and where you know simple training is valuable,” said Bhattacharjee.

Because most of the model’s contents don’t matter, they can be hard-coded into silicon, with only the last layer accepting trained weights. It also requires far less energy since only weights for the last layer must be fetched.

Not much is heard about this type of AI, but it can be useful for speech or audio recognition, processing sensor data and controlling motors on robots, and modeling sequences, including time-series prediction for applications such as weather and financial markets.

Quantum Computing Inc. (QCI) has a photonic version of a reservoir computer being readied for launch. “You can implement reservoir computing using other methods, but among all the methods, photonics is one of the best choices because of the bandwidth,” said Prajnesh Kumar, quantum technology lead at QCI.

One benefit of photonics is that it can eventually use wavelength-division multiplexing (WDM) to boost productivity. “Say you have 10 wavelengths,” explained Kumar. “You can put 10 different tasks on these wavelengths. Then if your kernel is the same for all of them, you can perform 10 operations in parallel without any sort of interference.”

That way, they’re addressing specific tasks that require high performance and low power. “There are certain future applications such as vertical takeoff vehicles where you need to do trajectory planning,” said Kumar. “You don’t want vehicles to collide. I have also been talking to one of the professors who’s been using our previous-generation product to analyze brain signals and figure out whether we can classify epilepsy versus other conditions based on brain activity. There are other applications, more toward certain military and defense applications.”

Taken together, these specialized approaches broaden the AI landscape rather than pointing to a single dominant architecture.

Many model types
The number of different model types keeps evolving, and that’s unlikely to change anytime soon. Today, CNNs still have a home, while LLMs and agentic AI are the big powerhouses, with MoE now gaining traction. Reasoning models aren’t as widespread but have their niche, as does reservoir computing.

Most of these technologies continue to rely on GPUs (or other NPUs) for the heavy lifting. The big exception is agentic AI, which is putting a premium on CPUs while not removing any of the pressure on GPUs.

It remains to be seen whether new breakthroughs such as attention and transformers occur regularly, but for now, developers are running hard with what is available today to find better ways to use it—with no end in sight.

Reference
1. Michael te Vrugt, “An introduction to reservoir computing,” book chapter to appear in: Artificial Intelligence and Intelligent Matter, Springer, Cham, arXiv:2412.13212v1 

Related Article
Agentic AI Is Changing Data Center Architectures
Standalone GPUs are being replaced by heterogeneous SoCs and chiplets that combine CPUs, GPUs, and NPUs to eliminate memory bottlenecks, reduce latency, and boost efficiency.