AI can transform a photo of your dog into a VR avatar

Nearly four years have passed since Facebook officially changed its corporate name to Meta, amid promises from founder Mark Zuckerberg that a fully realized digital “metaverse” was just around the corner. Since then, user adoption of virtual reality spaces has plateaued, and Zuckerberg himself has seemingly shifted focus towards AI companions and podcast-playing Ray Bans. For many, simply sitting on the couch at home with a dog by their side remains more appealing than slipping into VR. 

But what if your furry friend could join you?

Researchers from the Ulsan National Institute of Science and Technology in South Korea have developed an AI system that may make that possible. Using a system they call DogRecon, the team was able to take a single photograph of a dog and convert it into a realistic 3D digital model that could be used in virtual or augmented reality applications. Though it’s not quite ready for primetime, DogRecon’s unique weighting mechanisms helped it generate sharper, more lifelike renderings compared to similar AI models, which often produced blurry, distorted images with inaccurate body geometry. The researchers shared their findings in the International Journal of Computer Vision.

The video shows some early examples of how these AI-generated 3D dogs could appear in virtual environments. In the left frame, a pack of puppies (including a shiba inu in green dog booties) slowly march through a backyard garden. On the right, two dogs can be seen walking around a living room. Once the 3D dog avatars are generated, the researchers can control them using simple text prompts.

“With over a quarter of households owning pets, expanding 3D reconstruction technology—traditionally focused on humans—to include companion animals has been a goal,” first author Gyeongsu Cho said in a statement sent to Tech Xplore. “DogRecon offers a tool that enables anyone to create and animate a digital version of their companion animals.”

The researchers did not immediately respond to Popular Science’s request for comment. 

[ Related: A furry antelope robot is keeping tabs on its organic cousins. ]

Transforming 2D photo to 3D avatars 

Though substantial progress has been made in recent years to improve the realism of human avatars in virtual spaces, far less attention has been given to our furry companions. According to the team, one reason for this is that pets, (dogs in particular) are more anatomically complex than we are. Dogs come in hundreds of breeds, each with varying shapes, sizes, and unique physical features. Fido’s four-legged stance also leads to more frequent joint occlusion compared to humans, making accurate 3D modeling more difficult.

All of these extra variables create more opportunities for the model to produce errors when generating dogs in a digital space. However, the researchers argue that the lack of properly modeled pets in virtual reality limits the overall believability of any so-called “metaverse.”

“Without a doubt, human-specific approaches are important, but humans are only a tiny part of the actual biodiversity,” the researchers write.

The team tried to combat these challenges by first using breed-specific statistical AI models to capture variations in body shapes and postures. They then used a second generative AI model to create multi-view images that remain geometrically aligned with the dog’s body shape. By combining these two steps, the system could take a single image of a dog (in this case, real images pulled from the internet) and allow the AI models to essentially predict how the dog would look and move from different angles.

Not all of the AI’s predictions are accurate reflections of what a dog should look like though. To address this, the researchers applied a weighting mechanism to their model, prioritizing outputs that appeared more realistic and filtering out those with distortions or other unusual visuals commonly found in poorly generated AI images. Weighting, in this context, helps set parameters for AI outputs to steer it towards desired results. When they compared DogRecon to other AI models capable of generating video from images, they found that their system consistently produced more anatomically accurate results.

That’s not to say the results were perfect. The 3D models created by DogRecon generally resemble the dog breeds they’re intended to represent, but in the few examples shown in VR and AR settings, they still appear somewhat disconnected and visually jarring. Their movement resembles less the effortless agility of a border collie and more the stiff animation of an early South Park episode.

Still, rudimentary as they may appear, the researchers say tools like this could be welcomed by VR users who want to bring their pup along on their next digital adventure. As for real dogs—they’re still hanging out on the couch.

 

More deals, reviews, and buying guides

The PopSci team has tested hundreds of products and spent thousands of hours trying to find the best gear and gadgets you can buy.