Hands On With Android XR and Google’s AI-Powered Smart Glasses

Naturally, you can work in a mixed-reality environment with a connected Bluetooth keyboard and mouse, and you can put yourself in an immersive environment if you want to focus, or leave see-through mode turned on to make sure your coworkers aren’t taking photos and giggling while you wear a ridiculous headset to get stuff done. It wasn’t clear if you’d be able to connect the headset to a laptop to bring your work into mixed reality, a feature available on the Apple Vision Pro.

Gemini in XR

A tap on the side of the headset brings up an app launcher, and this is where you can toggle on Gemini if you want it to persistently stay “on.” Once it’s on, there’s an icon at the top of the virtual space so that you are aware that everything you say and look at is being registered by Gemini.

In see-through mode, you can walk up to an object and ask Gemini about it—a Googler demoing the headset (before I tried it) walked up to someone else wearing an FC Barcelona shirt and asked Gemini to find the “standings of this team.” Gemini quickly registered the team name and pulled up search results with league standings and scores from recent matches.

You can ask Gemini anything like this and it will answer with visual results displayed in the headset. I asked it to “take me to Peru,” and it opened up a 3D version of Google Maps. I was able to move around and center on Lima, and in cities where Maps already has a lot of 3D models, you can explore areas in greater detail. You can keep talking to Gemini in these experiences, so I asked questions such as when would be the best time to visit and got a prompt answer.

In another example, I peeked inside a restaurant in New York City to take a virtual tour of the space. Google says it can use AI to stitch together images of a venue’s interior and display it so that it feels like you’re there. It did a pretty good job, and I asked Gemini if the place takes reservations, without having to specifically say the name, because I was staring at the name of the restaurant. It does take reservations, but Gemini couldn’t actually make one for me. (That integration might come later.)

Next, I watched a few videos on YouTube, where 2D content looks sharp and colorful. Stereoscopic content was even better; my senses felt surrounded. I watched some hikers walking along a trail and asked Gemini where this all was, and it said, “New Zealand.” I wasn’t able to verify that, but it looked like the right answer. I watched some more spatialized playback of 2D videos as the virtual player added depth and layering to make them feel 3D. I hopped over to the Google TV app and enabled a “Cinema mode” to launch a virtual theater for watching movies and shows, just like on other VR headsets.

Circle to Search, the feature Google debuted earlier this year on Android phones, is also available in Android XR. Just walk up to a physical object near you, press the top button on the headset, and then pinch and draw a circle around the thing you want to know more about. You’ll get a Google Search page with results.

Smart Glasses

Project Moohan very much feels like Google and Samsung catching up to the rest of the VR market, though the Gemini integration gives their efforts a unique layer. However, I will admit I was far more excited to try the smart glasses, where Gemini feels like it could be even more helpful. They didn’t disappoint. I walked over to another room and there were several pairs of glasses in front of me. Some were sunglasses, others had clear lenses. Like the headset, you can get them loaded up with your prescription. Google did not provide a name for the prototype glasses.