Skip to content
Patchnotes

Genie 4, robot training worlds and image-to-3D

Issue 43 min read

Google DeepMind's Genie 4 adds sound to generated worlds, new tools train robots in simulation, and several models turn one image into a 3D object.

In this issue

Genie 4 adds sound and dialogue

Google DeepMind’s Genie 4 generates interactive worlds, and those worlds now have sound and dialogue. You explore with keyboard controls and change the scene with text.

It is a research preview for trusted testers, so most people cannot try it yet. A world model predicts what a scene does next as you act in it. That same idea runs through most of the releases below, from robot testing to 3D scenes.

More generated worlds

  • Tencent Hy-Nomas. You steer a generated world with text and camera moves. The 8B model makes up to 60 seconds of 720p video, and a scene stays consistent when the camera returns to it.
  • Mundo. Turns a few images of a place into an editable world you can explore and simulate. It combines a simplified 3D scene, a physics simulator and video generation, and it supports robot testing.
  • OverSeer. Lets several AI-controlled players share one consistent generated world. Fixed camera views act as shared reference points.
  • UniAction. One world model from Stanford that responds to forces, camera movement, hand poses and robot poses.
  • Viggle JST-2. Generates explorable, animatable 3D scenes and characters. A scene takes about a second, and the characters come ready to move.
  • Skyfall AI SKY-001. A world model for engineering design. It breaks a product image into assemblies and parts, plans how to build them, and turns building outlines into floor plans.

Training robots and agents in simulation

These releases supply the practice material. They cover rooms, objects, motion and the workflows that tie them together.

  • SceneBloom. Generates realistic furnished rooms to train navigation agents. It pairs AI layout ideas with geometric checks. The team reports that training in its rooms improved navigation in real scans the agents had not seen.
  • NVIDIA SimReady Assets. 2,677 downloadable 3D objects for simulation, both everyday and industrial.
  • NVIDIA MotionBricks. Generates character movement from a model trained on more than 350,000 motion clips. Reusable controls combine navigation with object interaction.
  • RoboSkills. A toolkit for agents that do robotics and simulation work. It has five workflows, including rebuilding demonstrations in simulation, generating motion and testing robot actions.
  • StreamVibe. Video assistants that notice when you need help as a task unfolds. The release includes a model and a benchmark that tracks changing goals and progress.

From images and video to 3D

Most of these start from a single picture. PURE starts from video.

  • PhysAg3D. Turns a single image into a 3D object with parts that move or respond. A 3D generator builds the object, then an agent proposes interactions and refines the asset.
  • Sperid Labs Idol. Turns a single image into a 3D object built from Gaussian splats. The level of detail is adjustable, and it needs under 8 GB of GPU memory.
  • Hi3D 1.0. The image-to-3D model formerly called Sparc3D, now with code and weights. You can run it locally or try the browser demo.
  • AlignCanvas. Turns images into 3D motion.
  • PURE. Improves 3D reconstruction from video by learning from footage without hand-labeled 3D data. It does better on moving scenes. Code, weights and a demo are out.

The short version

  • Genie 4 adds sound and dialogue to generated worlds, and Hy-Nomas keeps a scene consistent when the camera comes back to it.
  • Robot training now has ready-made rooms, objects, motion and agent workflows to draw on.
  • Single-image 3D now covers objects with moving parts (PhysAg3D) and runs in under 8 GB of GPU memory (Idol).

Topics in this issue