This innovative video game can sense your emotions and respond accordingly

Artificial intelligence already pervades 21st-century life, from Siri's directions to Netflix's suggestions of what you should watch next. But how much emotional intelligence is inside computers, cell phones, and video game consoles? In the past, the answer has been "none" — even the most complex deep learning machine is still a machine. That's changing, though, thanks in part to Nevermind, a video game that can sense players' emotions and adjust the experience to fit.

The psychological thriller, which debuted last year, isn't the average first-person shooter game. Instead of being given a gun and told to kill enemies, players inhabit the persona of a Neuroprober, a physician who can enter the minds of trauma victims. As they explore the troubled psyches, Neuroprobers must solve logic puzzles and recover memory fragments to help their patients get better.

The world of the game becomes darker and more twisted as players exhibit more stress — and when they calm down, the game does, too. With the help of optional sensors like Garmin heart rate monitors, Apple Watch, and the Tobi EyeX controller, Nevermind responds to biofeedback like heart rate and eye movements. Now, the latest updates expand the game's capabilities by incorporating emotion recognition software Affdex, which uses data from players' webcams to track their response to emotional distress and further alter what happens on screen.

Subscribe to The Week

Escape your echo chamber. Get the facts behind the news, plus analysis from multiple perspectives.

SUBSCRIBE & SAVE

Sign up for The Week's Free Newsletters

From our morning news briefing to a weekly Good News Newsletter, get the best of The Week delivered directly to your inbox.

From our morning news briefing to a weekly Good News Newsletter, get the best of The Week delivered directly to your inbox.

Sign up

There's a reason that Erin Reynolds, creative director of Flying Mollusk and creator of Nevermind, made a game that freaks out alongside its players. "I'm a big believer in what I call positive games," Reynolds says — games that go beyond fun to teach players a skill or help them become healthier. Her goal is to give back to players by teaching them to calm themselves, even in the face of high-stress gameplay.

The Affdex technology, which can detect subtle facial movements, comes courtesy of emotion digitization startup Affectiva. It was years in the making, says Gabi Zijderveld, VP of marketing and product strategy. It draws on a database of faces of more than four million people from 75 countries — faces whose emotions have been mapped, analyzed, and translated into algorithms capable of pinpointing even the most fleeting feelings.

The company got its start in MIT's prestigious Media Lab, which is known for cutting-edge tech research and provides emotion data and analysis that helps corporations like Unilever determine how best to market and position their wares. But video games are a new foray, says Zijderveld.

"Most artificial intelligence has a high I.Q., but absolutely no emotional intelligence," she notes. "Games are purposely designed to take us on an emotional journey, but they don't sense the emotions of the player."

Reynolds agrees. "Most games that are out there are one size fits all," she says. But with Nevermind, as players' emotions heighten, Affdex measures their responses, then communicates that data to the game. The game, in turn, responds, amping up the pressure and difficulty for stressed-out players and introducing more pleasant, calming elements for those who manage to control their feelings.

Biofeedback information only tells part of the story, Reynolds notes. Heart rate, for example, can show how stimulated a person is, but can't measure whether the heart rate is elevated for positive or negative reasons. That valence, as psychologists call it, can be captured by measuring a person's facial expressions. And layering that emotional understanding on top of the physical data, Reynolds says, is invaluable to game developers.

But gaming is only the start. Zijderveld becomes animated when she talks about other industries that could one day use this technology. Cars, for example, could modify their lighting to reflect a person's emotional state; stores could highlight certain products to people based on their emotions, and hospitals could refine medical care by better understanding how patients really feel. "The future isn't even that far away," she says — with the help of unprecedented processing power and deep learning, companies like Affectiva should become even better at analyzing and even predicting emotional responses.

To do so, they'll need to collect even more emotional data, but both Zijderveld and Reynolds insist that games played in the privacy of people's own homes are not the place to obtain that information. "It's really just about you in our game," says Reynolds. "We use it for the purpose of the game and then throw it away." Zijderveld notes that Affectiva only collects emotional data from consenting participants.

And don't assume that smarter machines will one day replace humans. One day, says Zijderveld, emotionally intelligent technology might facilitate better interaction between humans who increasingly communicate via digital means. Perhaps hyper-personalized products like video games will not just make humans feel more appreciated, but give them chances to practice relating to one another. For now, though, games that teach people to control their emotions are a start.