NVIDIA's New Toolkit Lets AI Agents Build Physical AI, Slashes Training Time

NVIDIA has open-sourced its Agent Toolkit, a set of tools designed to cut the cost and time it takes to develop AI for physical systems like robots and autonomous cars. The core idea is to let coding agents manage NVIDIA's foundational libraries directly, breaking down complex projects into smaller, automated tasks. Early results from manufacturing partner Pegatron are already showing a 67% reduction in the time needed to train and deploy models.

The toolkit extends the role of AI agents beyond just writing software code. Instead, they can orchestrate the entire workflow for systems that interact with the real world, from generating synthetic data in simulations to deploying models on edge hardware like NVIDIA Jetson. It gives agents a standardized way to call on NVIDIA’s software stack, which helps streamline development and reduce the need for manual oversight.

A Modular Stack of 'Skills'

To make this happen, NVIDIA has refactored its software into a series of agent-callable modules, or "skills." This enables a full "sim-to-real" workflow that starts in a virtual environment and ends on physical hardware.

The stack includes:

• NVIDIA Isaac: For robotics, agents can automate navigation training and generate data for perception and mobility models.
• NVIDIA Metropolis: A vision AI framework where agents can generate synthetic data for automated inspection or build AI agents to analyze video streams.
• NVIDIA Alpamayo: The company's autonomous driving toolset. Here, agents can reconstruct real-world fleet data for simulation and run reinforcement learning to train and test AV models.
• NVIDIA Omniverse: Agents use these libraries to convert assets like CAD files into simulation-ready formats and optimize large digital twin scenes.
• NVIDIA Cosmos: A set of foundation models that provide agents with physical world reasoning to help them generate controllable synthetic data for training.

Deployment and Early Partner Results

The toolkit’s instructions specify which tools an agent should use and how to validate the results. For security, deployment is managed through the NVIDIA NemoClaw blueprint and the OpenShell runtime, offering policy-based controls for both local and cloud execution.

The tools and skills are available on GitHub, and NVIDIA is offering preconfigured "Physical AI Launchables" on NVIDIA Brev for faster setup. Cloud providers including Microsoft, CoreWeave, and Nebius are also integrating the skills into their platforms.

NVIDIA shared performance metrics from several early partners:

• Electronics Manufacturing:

  • Pegatron: Saw a 67% reduction in model training and deployment time using a Defect Image Generation skill.
  • Delta Electronics: Reported a 17% improvement in detecting soldering defects.
  • Inventec: Cut its data collection for laptop chassis manufacturing by 30%.
  • Foxconn/DeepHow: Increased first-pass yield by roughly 3% by catching production errors sooner.

• Autonomous Vehicles:

  • Li Auto, Afari, and DeepRoute.ai: Are using NVIDIA Omniverse models for neural scene reconstruction, generating thousands of simulations daily.

• Industrial Software & Manufacturing:

  • Cadence, Dassault Systèmes, and Siemens: Are integrating Omniverse skills for inspecting engineering data and building interactive digital twins.
  • SK hynix: Is using the toolkit to build a digital twin of its semiconductor fab, part of its roadmap for a fully autonomous factory by 2030.

• Robotics:

  • 1x, Agile Robots, and Universal Robots: Are using the agent-ready stack to accelerate their development pipelines from data generation to deployment.