Fitting AI Models To Edge Applications

Key Takeaways:

• The first step in right-sizing an AI model for an embedded application is determining its purpose, including the target device functions and product features.
• Designers typically use a mix of compression techniques rather than a single method. Strategies include quantization, distillation, and pruning.
• Small language models (SLMs) and compact edge models require tight system-level integration to ensure optimal efficiency.

Today’s AI industry faces two very different challenges. On one side, enormous AI data centers and AI factories are seeking more efficient ways to process high-parameter large language models (LLMs), with bandwidth and memory as the key bottlenecks. On the other side are smaller AI models running locally on edge devices, where there isn’t enough area, power, or memory for complex, higher-parameter LLMs.

“This has led to a gold rush with people raising huge amounts of capital to build mega data centers, but has also sparked innovation on the edge,” said Ed Plowman, CTO at Imagination Technologies. “People are trying to work out how they can cram an entire container of liquid into a very small cup, and that has been driven by the fact that compute resources are not the main problem at the moment. It’s bandwidth, and that’s why RAM has gotten so expensive. That triggered a whole bunch of researchers to look at what the key problems are and what we can do about it. That triggered innovations around quantization — reducing activations and weights simultaneously — to make the models even smaller but still maintaining the output capability of the model.”

AI models come in many shapes and sizes to fit a wide range of edge devices, from earbuds to phones, home appliances, industrial IoT, robotics, and automotive systems. Designers need to choose a model with the right number of parameters because an oversized model can slow the device down or cause it to overheat. On the flip side, if the model is too small, the product may not be smart enough.

The first step in right-sizing an AI model for an embedded application is to define its purpose. “You have to start from the point of view of what you want the model to do, but then there are a million and one things we can throw at it to make it smaller,” said Plowman. “Quantization and self-compression are very effective tools. Distillation is another approach, and it works better on transformers than on traditional convolutional neural networks (CNNs). All these things are starting to combine to get us to a point where the gold rush for bigger and bigger GPUs is less prevalent. The enablement in embedded systems and the progression towards quite intelligent models being able to run on truly embedded platforms is becoming more realistic.”

Fig. 1: Common ways to compress AI model

The model size must be considered alongside the system. “The hard part is designing a model that is sophisticated enough to solve the problem but small enough to fit the device,” said Magnus Gedda, CTO and chief product officer at Imagimob, an Infineon Technologies company. “On top of that, the model usually shares memory and compute with the rest of the application, so the available resources are smaller than the spec sheet suggests. Optimization techniques like pruning and quantization can shrink a model, but they are applied after design and usually after training, and they cost some accuracy. You need to pick an initial architecture that is the right size from the start so that its optimized version still fits the device and meets the performance target. Getting those three things to line up — problem fit, size, accuracy after optimization — is where most of the effort goes.”

Others agree that system-level efficiency is the central challenge. “Edge AI is ushering in a shift from ‘largest model wins’ to ‘experience per watt,'” said Steve Woo, distinguished inventor and fellow at Rambus. “Techniques such as compression, pruning, and lower-precision arithmetic help shrink models to fit local resources, but deployment remains difficult because memory capacity and bandwidth are often limiting factors. Getting a model onto the device is just the beginning. Making it run efficiently within tight power and latency budgets is where the engineering challenge begins.”

Choosing the wrong AI model can lead to extra work after deployment or a substandard product. “Once you have the optimized model, deployment is not that hard,” said Infineon’s Gedda. “The model is converted to C code, compiled for the target, and flashed to the device. The challenge is using the hardware to its full potential, so that inference is energy-efficient and low-latency. Edge hardware is extremely varied — MCUs, accelerators, NPUs, DSPs — and standards are somewhat lacking. So without the right tooling you end up with a generic, inefficient deployment, or a lot of manual work per target.”

These deployment constraints make model selection an application-specific exercise. The right architecture depends on the functions the device must perform and the workloads it must support.

Functions and workloads determine the correct model
Edge device and physical AI workloads vary considerably across sectors, from medical to industrial, automotive, home, or personal. The specific modalities of the collected sensor data are the primary driver for selecting the optimal AI model type and size.

“Right-sized inference for physical AI is critical,” noted Mick Posner, senior product marketing group director for chiplets at Cadence, in a recent blog. “Intelligent physical systems do not have a one-size-fits-all compute profile. Some workloads require always-on, low-power inference. Others require higher-performance multimodal processing across cameras, radar, lidar, audio, environmental sensors, and system-state data.”

Each hardware function and end-device feature needs to be considered and defined. “The best techniques for right-sizing AI models for edge AI are being able to categorize your functionality so that you don’t need a big model,” said Sathish Balasubramanian, head of product for EDA AI at Siemens EDA. “A 4-billion [parameter] model on a sensor is huge, even though we’re now talking 1 trillion [in the cloud]. Being able to bake that model into less than a billion parameters and have it in each of the sensors is doable. We need to define the highest-priority, must-have function, and train it well in an SLM (small language model). It could even be an ML (machine learning) model or a neural network model. It doesn’t need to be an LLM.”

After defining the must-have functions, designers can start building the AI core, then slowly add more features, including must-have, critical, or nice-to-have.

“Those are the three categories I see people defining in terms of how they do it architectural-wise, and where they fit in what intelligence,” said Balasubramanian. “Once you have determined those categories, the must-haves should be very close to the sensor. You can’t make everything must-have. You really need to prioritize and design it in a way that [the system] is able to be in a self-contained sensor loop with the local compute, local memory requirement, and sometimes even a local backup environment. Everything is cut up. You also need to have at least an hour or two of power envelope available so that the device doesn’t slow down or shut down.”

In the automotive sector, companies are increasingly deploying AI models for certain tasks and functions, though much of that industry is still using CNNs from about three or four years ago, Plowman noted. “They haven’t fully moved to transformers yet, or embraced the latest compression techniques. But certain external bodies have produced image segmentation object recognition systems, which use transformers.”

For IP providers tasked with optimizing these models, one challenge is that they may not get to see the actual model their customers are using. “The automotive guys train their own models, and they’re very secretive about them, for safety, security, and proprietary IP reasons,” said Plowman. “We don’t get to see them. We get a lot of proxies about what they look like or proxy functions that they want to accelerate, but it’s very rare that we get an entire model to play with.”

Fig. 2: IP modules for edge AI and physical AI. Source: Synopsys

For right-sizing, each IP block needs to be weighed against the model size. “Right-sizing a model for the edge is a systems problem, not a single-point optimization,” said Licinio Sousa, senior director of product management at Synopsys. “Techniques like quantization help the model fit, but real-time latency, power, cost, and functional safety all have to be balanced together. As AI models evolve faster than silicon development cycles, systems also need enough flexibility to support not only today’s workloads, but the models that come next.”

Once the model and system architecture are aligned, designers must determine which compression techniques can meet those constraints without sacrificing critical accuracy or performance.

Challenges of various compression techniques
Designers typically use a variety of compression techniques to right-size a model, and each has pros and cons.

For example, distillation enables drastic shrinking but potentially more loss. “You interrogate the model to find and see which bits of it light up,” said Imagination’s Plowman. “You do a similar exercise where you trim things off, but there is always loss from the teacher-student model. The student is trying to refine from the big model the essentials of what the teacher model knows. That’s why there’s potentially more inherent loss, but you retain much more of the reasoning capability. It drastically shrinks the size of the model, but there’s only so far you can go with that without going to quantization.”

Quantization makes models take a roundabout path to their destination instead of a direct one. “When applying an input to a network, there is a starting point for navigation,” said Plowman. “Then, depending on the values that you’ve given it, you need to navigate to an endpoint. Every time you do a layer you’re nudging the transform in a slightly different direction. With a full-width value such as FP32 or FP16, you’re going much more directly to a given point.”

Full-width values travel as the crow flies, while quantization is closer to city-block navigation. “We go up one, over one, up one, over one, but eventually you end up in exactly the same place,” said Plowman. “The trick with quantization is making those small, more angular steps mean the same thing as the direct navigation, but without the overhead power, memory, and other costs of the associated mathematics. It starts off with all the weights in the model, and it starts fitting them into buckets that match the quantization. There are lots and lots of different ways of doing the quantization. That’s a very brutal way of doing it.”

A gentler way is self-compression. “Self-compression techniques understand that, as a model is being trained, certain pathways will be very popular while others will be less popular,” explained Plowman. “You never know what this dimensional shape will look like. It’s random every time you retrain. You can maneuver the space to recreate certain desirable shapes you want every time, or get close to them, but there is an element of randomness. You get certain pathways that become critical as scaffolding for the rest of the system, but they’re not pathways that will matter in the end. We look for the sensitivity to those pathways, then quantize and quantize and quantize them down to zero bits. If nothing changes in the model and you still get the same outcomes, there was likely very small, or infinitesimally small, or no contribution from [those pathways].”

Fig. 3: A quantized transformer architecture. Source: Imagination

For quantization and self-compression, the complexity lies in managing the loss. “You need to make sure you’re not throwing away things that you really need, and that you’ve got good data coverage in the training set for the quantization flow,” Plowman observed.

For example, in automotive, engineers need to avoid losing long-tail data on edge cases or rare training samples that let an autonomous system navigate unusual road conditions.

Conclusion: Trending smaller
Small language models and other compact models suit resource-constrained edge devices, but they require system-level design and optimization to deliver the functions and features a specific product needs.

Ultimately, successful edge AI deployment starts with defining the workload and choosing an appropriately sized architecture — not simply shrinking the largest available model. Compression can help close the remaining gap, but its value depends on preserving accuracy while meeting the device’s power, memory, latency, cost, and safety requirements. As models and hardware evolve at different rates, flexible system design will be essential to support current applications and future workloads.

References

1. Structured vs. Unstructured Pruning: An Exponential Gap (Université Côte d’Azur, King’s College London)
2. Self-Compressing Neural Networks (Imagination Technologies)

Related Articles
The Hidden Challenges Of Edge AI Design
As models evolve faster than silicon cycles, chip architects must balance flexible compute, data movement, and defense-in-depth security.

Can Edge AI Keep Up?
As models evolve faster than silicon cycles, experts weigh how much adaptability architects can afford without sacrificing power, area, or efficiency.