Robots are becoming more common in public spaces, but teaching them to perform everyday tasks remains a major challenge.
Researchers at the Massachusetts Institute of Technology (MIT) and the Toyota Research Institute have introduced SceneSmith, a new AI-powered system that automatically creates realistic 3D environments in which robots can safely practice household and workplace activities.
By generating detailed virtual spaces such as kitchens, hotels, offices, and living rooms, the system helps reduce the time and effort needed to train robots before they enter the real world.
Modern robots improve their abilities by learning through experience, much like people do. However, collecting enough real-world training data requires countless hours of testing in different locations and situations. This process is expensive, slow, and difficult to scale as robots are expected to perform more complex tasks.
Researchers have turned to computer simulations to solve this problem. Virtual environments allow robots to practice safely without the risks and costs of physical testing. The challenge has been building digital worlds that accurately reflect the complexity of real-life environments.
SceneSmith: Virtual Training Spaces
SceneSmith addresses this challenge by using three AI agents that work together to design complete 3D environments from simple text instructions. Instead of relying on a fixed collection of digital assets, the system creates new scenes filled with furniture, household items, walls, ceilings, and interactive objects. These environments are then imported directly into robotics simulation software for training.
The system relies on advanced vision-language models(VLMs), which understand both written text and images. According to the research team, each AI agent performs a specific role during scene creation. Together, they generate environments that closely resemble real indoor spaces where robots are expected to work.
One AI agent serves as the designer, creating the overall layout and placing objects throughout the room. A second agent serves as the critic by checking whether the arrangement looks realistic and practical. A third agent acts as the orchestrator, managing the discussion between the first two until the design reaches an acceptable standard.
MIT researchers said this collaborative process allows SceneSmith to create rooms that resemble the work of a human designer.
Lead author Nicholas Pfaff said the system generated more than 1,300 unique scenes while producing highly creative layouts without receiving detailed instructions for every design choice. He said the AI often improvised realistic room arrangements using the knowledge it had learned from internet-scale data.
Users can simply describe the environment they need using everyday language. For example, a request for a garage with a vehicle, workbench, stacked tires, and a ladder leaning against a wall produces a detailed virtual workshop. The completed environment gives robots many different objects to interact with during training.
READ ALSO: AI Hunts for Next-Gen Magnets: $2.7M Project Aims to Beat Neodymium Performance
Smarter Robot Practice
The virtual environments contain significantly more objects than previous simulation systems. Researchers reported that SceneSmith generates up to six times as many items per scene as earlier methods. This higher level of detail exposes robots to situations that more closely match real homes, offices, and commercial spaces.
Robots can practice common activities such as placing dishes in sinks, moving fruit onto plates, opening cabinets, or transferring drinks from shelves to tables. Repeating these actions across many different virtual rooms helps improve the robot’s ability to adapt to unfamiliar environments. Engineers can also test new control software without risking damage to expensive robotic hardware.
The researchers evaluated robot action plans by generating 100 different virtual environments. A vision-language model assessed whether each robot successfully completed its assigned task. Human reviewers agreed with the AI’s assessments more than 99 percent of the time, showing that the automated evaluation was highly reliable.
The team also tested whether robots trained on real-world data could operate successfully within SceneSmith-generated environments. In one example, a robot received instructions to pick up an apple from a bowl and place it on a cutting board. The robot completed the task successfully, suggesting the virtual environment closely matched the real-world settings it had previously learned from.
Researchers also remotely controlled robots through the simulated buildings. The machines opened cabinets, stored bottles, and moved between different rooms without major problems. These tests demonstrated that the environments remain stable during continuous physical interaction rather than serving only as visual demonstrations.
Building Realistic Worlds
SceneSmith creates environments in several carefully planned stages instead of generating everything at once. The AI first produces a basic floor plan, then gradually adds furniture, wall decorations, ceiling features, and smaller household objects. This layered approach helps maintain consistency throughout the scene.
During every stage, the critic agent checks whether each addition makes sense. It can recommend removing unrealistic objects, such as placing a bathtub inside a living room. The orchestrator reviews every decision and can even send the design process back several steps if improvements are needed.
Once the layout is complete, the system assigns physical characteristics to every object. It calculates properties such as weight, friction, and movement so that robots interact with them realistically during simulation. This allows doors, cabinets, and other movable items to behave much like their real-world counterparts.
The researchers compared SceneSmith with existing scene-generation systems, including HSM and Holodeck. They found that the new system consistently produced richer environments with more objects and greater diversity. Examples included private offices, pottery stores, restaurants, bedrooms, hotels, and even themed gaming rooms.
More than 200 participants also evaluated the generated environments. Most users rated SceneSmith’s scenes as more realistic than competing systems in over 90 percent of comparisons. Participants also found that the AI followed written instructions more accurately than previous approaches.
Future Robotics Impact
SceneSmith is not limited to generating complete rooms. The researchers said it can also create entirely new 3D objects from text descriptions. For example, users can request a rolling serving cart, and the system first generates a detailed image, then converts it into a fully functional digital object with realistic physical behavior.
This flexibility removes one of the biggest limitations of earlier robotics simulators. Previous systems depended heavily on fixed libraries of digital objects, making it difficult to create unique environments. SceneSmith instead builds new assets as needed, giving engineers much greater freedom.
The system’s detailed design process requires substantial computing resources. Producing a single fully developed environment can currently take several hours because each AI agent carefully reviews every design decision. The researchers expect future improvements in computing to significantly reduce this processing time.
The team also hopes to expand SceneSmith’s capabilities by adding deformable objects such as sponges and other flexible materials. These items are more difficult to simulate because they bend and change shape during interaction. Larger collections of high-quality 3D models may help support this future development.
Jeremy Binagia, an applied scientist at Amazon Robotics who was not involved in the research, said SceneSmith advances simulation by creating detailed indoor environments directly from simple text prompts.
He noted that the system combines visual realism with accurate physical behavior, generating original digital assets rather than relying solely on existing libraries.
The project was developed by researchers from MIT’s Computer Science and Artificial Intelligence Laboratory and Toyota Research Institute. Support for the research came from Amazon, the US Office of Naval Research, the Toyota Research Institute, and the US National Science Foundation. The team presented its findings during the International Conference on Machine Learning, where the work received spotlight recognition.
As robots move into homes, warehouses, hospitals, and factories, realistic virtual training environments are becoming important. Systems such as SceneSmith may help engineers prepare robots more efficiently, reduce real-world testing costs, and accelerate the safe deployment of intelligent machines across a wide range of industries.
