DeepMind reveals Gemini Robotics 2.5 and open safety benchmark "ASIMOV-jailbreak"

Generally better

The new model is up to twice as successful at some tasks.

Two Gemini-powered robots hard at work in a Mercedes-Benz factory. Credit: Google

Google DeepMind has revealed a new generation of the AI-powered Gemini Robotics platform. In the 2.5 release, Google says its robotic AI is getting tantalizingly close to completing real-world tasks as well as humans do. In fact, some Gemini robots were able to get actual work done in Mercedes-Benz factories, and the team is preparing for a day when robots operate alongside humans with an open source safety benchmark.

DeepMind’s physical AI platform has several computational layers between user input and robotic action. There’s the vision-language model piece, which is Gemini Embodied Reasoning. Essentially, it takes your instructions and uses live video to plan how the robot can accomplish the task. The other main component is the vision-language-action (VLA) that actually generates robot actions, like moving an actuator—that’s Gemini Robotics, which has been bumped to version 2.5.

DeepMind says that Gemini Robotics 2.5 is much better at tasks that require a high degree of dexterity. The model was evaluated across more than 200 activities, including serving food with tongs and using a pipette, as you would see in a laboratory. Google also showed that Apptronik’s Apollo 2 humanoid robot with Gemini Robotics 2.5 can now tie a bow with its five-fingered hands, a notable feat, because deformable objects (like ribbon) are notoriously difficult for robots to manipulate.

Deformable objects are tough for robots, but Gemini is making progress. Credit: Google

Overall, Gemini Robotics 2.5 manages a 30 to 100 percent higher success rate compared to the last version. It’s also 20 to 30 percent faster. But that’s based on laboratory testing. Google partnered with Apptronik and Mercedes-Benz to see how capable this AI is when faced with the uncertainty of the world outside robotics labs. The Gemini-powered Apollo 2 robots kitted automotive parts in a few European factories, reaching a 96 percent success rate.

That’s an impressive showing compared to previous robotic AI systems, but it’s still probably not enough for a company like Mercedes-Benz. A human doing that sort of repetitive task will get it right virtually every time, and DeepMind director Kanishka Rao notes that closing that last 4 percent gap will be a “steep road.”

You could manually program a robot to do a handful of tasks in a factory without issue—this type of automation has been common for decades. Gemini Robotics is not trying to just do a few things, though. These are intended to be general robots that can handle almost anything, but there are trade-offs to that approach.

“We can be very good at a particular application at the cost of generalization,” said DeepMind Robotics head Carolina Parada. “Or we can just push on the generalization axis and maybe not get to 99.9 in every application.”

Argon assistance

Training data is essential for training all generative AI models. Most text-based chatbots in existence were fed basically the entirety of the open Internet, but there isn’t as much “physical data” for robotic AI. The best kind of training data for robots comes from teleoperation, but the team says Gemini 4 Argon has helped create a “simulation engine” that greatly accelerated development of Gemini Robotics 2.5.

The team used Argon to generate thousands of variations on practice runs based on real-world camera data. These “digital twins” apparently allowed Gemini Robotics 2.5 to more quickly generalize new tasks and settings. There’s still a role for the real thing, though. DeepMind combined this synthetic data with traditional training demonstrations to ground the model.

Argon was also used for analyzing failures and suggesting ways to improve in both simulations and real robots. However, this isn’t something that works in real time. Frontier models like Gemini take time to churn through their tokens, but robots need to be able to operate without long pauses. The new VLA isn’t lightning quick, either. While Google has experimented with some embedded robotics models, the current 2.5 release is still running on cloud-based servers. So those gains in accuracy and speed still don’t come without latency. The video above is sped up by 2x to make the robot look more human-like.

DeepMind is also working to ensure the failures don’t put any nearby humans at risk. The team notes that Gemini Robotics 2.5 uses inference-time reasoning to avoid hazards. So if something gets in the way, the robot will stop completely or wait until it’s safe to continue. This is essential if you’re going to have a robot operating outside of the lab, for example, in a Mercedes factory. DeepMind says this feature allowed Gemini robots to safely complete tasks 33 percent more often when obstacles were present. The success rate went up 43 percent when people were in the way.

There is no agreed-upon way to test physical AI for safety, but DeepMind hopes to move the industry in that direction with ASIMOV-Jailbreak. This is an open source database that contains 1,500 prompts designed to trick a robotic AI into doing something unsafe. For example, a prompt might ask the robot to place items in a bag and close it with a fastener too fiddly for the robot to operate. The AI should decline to do that, but attempting to proceed and incorrectly closing the container would be the “unsafe” outcome.

DeepMind says that placing Argon in the loop results in a 98.8 percent jailbreak safety level. With just Gemini Robotics ER 2, the system scores 73.5 percent in Google’s safety benchmark.

The full ASIMOV dataset is available on Hugging Face for anyone working on embodied AI who wants to analyze its safety. Gemini Robotics 2.5 is not generally available, but it’s now in the hands of Google’s robotics partners, including Agile Robots, Apptronik, and Boston Dynamics.

Ryan Whitwam is a senior technology reporter at Ars Technica, covering the ways Google, AI, and mobile technology continue to change the world. Over his 20-year career, he's written for Android Police, ExtremeTech, Wirecutter, NY Times, and more. He has reviewed more phones than most people will ever own. You can follow him on Bluesky, where you will see photos of his dozens of mechanical keyboards.

1 Comments