There's augmented reality in space now

Astronauts aboard the International Space Station already have humanity's most impressive view of Earth. But a new NASA project is enhancing it even more — through augmented reality.

The program, called Project Sidekick, uses the Microsoft HoloLens headset as a high-tech lifeline to Earth — one that will let technologists on the ground guide astronauts through repairs and experiments. Unlike virtual reality, which immerses users in a lifelike digital world, augmented reality adds to the real world by projecting computer-generated information or images onto what the user sees (think Google Glass). The space station's astronauts received their headsets last December but have only recently begun to test them.

NASA explains that Project Sidekick will help astronauts in two ways. First, it enables them to view an animated holographic illustration on top of the object in front of them — for example, a Sidekick-equipped astronaut could tackle repairs to the space station's notoriously finicky toilet by interacting with a holographic illustration of the toilet complete with specs and instructions from the people who designed it. In that way, the illustration serves as a kind of interactive user's manual that in the future could reduce training and help astronauts with their fixes even when engineers and techs are offline.

Subscribe to The Week

Escape your echo chamber. Get the facts behind the news, plus analysis from multiple perspectives.

SUBSCRIBE & SAVE

Sign up for The Week's Free Newsletters

From our morning news briefing to a weekly Good News Newsletter, get the best of The Week delivered directly to your inbox.

From our morning news briefing to a weekly Good News Newsletter, get the best of The Week delivered directly to your inbox.

Sign up

But a second mode is even more futuristic: Astronauts can let operators on the ground view exactly what they're seeing in space. The remote technologists then guide the astronauts through repairs and experiments in real time using a Skype uplink. Experts can even draw on and annotate the augmented "environment" astronauts see as they peer through the HoloLens.

"Augmented reality can allow an on-the-ground expert to see through the eyes of someone working in space," Columbia University computer science professor Steven Feiner told The Week. Feiner's Computer Graphics and User Interface Lab has designed and built experimental augmented reality systems for the last 25 years, and he and his colleagues know its uses go beyond mere visual fireworks: They have repeatedly demonstrated that people can reduce both the amount of time it takes to perform a task and their overall error rate when they are assisted by augmented reality units.

Because augmented reality lets people interact both with their environment and with experts and objects that aren't physically there, says Feiner, it gives users a crucial advantage. HoloLens-wearing astronauts can access the deep expertise of the technologists who designed and built instruments and experiments on the International Space Station — even when those engineers are hundreds of miles away. And in the future, Feiner adds, remote experts may even be able to add virtual 3D imagery to augmented reality environments that could explain things even better than the expert would in person.

"I'm somebody who believes in this technology and its potential," he says. But there could be a downside — since augmented reality environments are built by people, they're only as good as the information those humans can provide. Feiner says that it's incumbent on technologists and augmented reality users alike to make sure that people aren't put in technically challenging situations without the ability to problem-solve on their own. "We have to enable people to be resourceful when the technology doesn't work," he continues. "You don't put somebody out in the desert without telling them about how to use the stars to navigate…if the rest of their technology fails, you still want them to be able to know which way is north."

HoloLens augmented reality repairs may be the realm of astronauts for now — Microsoft released its first purchase invitations to select developers in late February — but that doesn't mean the rest of us can't follow along while the space station team tests out the technology. British astronaut Tim Peake seems to be a fan; he tweeted that the AR test was "going so well until we found the Alien Invaders game…." As for astronaut Scott Kelly, who tried the visionary goggles a few weeks before returning to Earth, he tweeted a single opinion: "Wow."