Enigma of how faces are encoded in the brain

"We've discovered that this code is extremely simple," says senior author Doris Tsao, a professor of biology and biological engineering at the California Institute of Technology. "A practical consequence of our findings is that we can now reconstruct a face that a monkey is seeing by monitoring the electrical activity of only 205 neurons in the monkey's brain. One can imagine applications in forensics where one could reconstruct the face of a criminal by analyzing a witness's brain activity."

Earlier research by Tsao and others used fMRI in humans and other primates to identify the areas of the brain that are responsible for identifying faces. They called these six areas, which are located in the inferior temporal (IT) cortex, face patches. Further investigations showed that these areas are packed with specific nerve cells that fire action potentials much more strongly when seeing faces than when seeing other objects. They called these neurons face cells.

Previously, some experts in the field believed that each face cell in the brain represents a specific face, but this presented a paradox, says Tsao, who is also a Howard Hughes Medical Institute investigator. "You could potentially recognize 6 billion people, but you don't have 6 billion face cells in the IT cortex. There had to be some other solution."

In the current study, Tsao and postdoctoral fellow Steven Le Chang, the paper's first author, found that rather than representing a specific identity, each face cell represents a specific axis within a multidimensional space, which they call the face space. In the same way that red, blue, and green light combine in different ways to create every possible color on the spectrum, these axes can combine in different ways to create every possible face.

The researchers started by creating a 50-dimensional space that could represent all faces. They assigned 25 dimensions to the shape -- such as the distance between eyes or the width of the hairline -- and 25 dimensions to nonshape-related appearance features, such as skin tone and texture.

Using macaque monkeys as a model system, the researchers inserted electrodes into the brains that could record individual signals from single face cells within the face patches. They found that each face cell fired in proportion to the projection of a face onto a single axis in the 50-dimensional face space. Knowing these axes, the researchers then developed an algorithm that could decode additional faces from neural responses.

In other words, they could now show the monkey an arbitrary new face, and recreate the face that the monkey was seeing from electrical activity of face cells in the animal's brain. When placed side by side, the photos that the monkeys were shown and the faces that were recreated using the algorithm were nearly identical. Face cells from only two of the face patches -- 106 cells in one patch and 99 cells in another -- were enough to reconstruct the faces. "People always say a picture is worth a thousand words," Tsao says. "But I like to say that a picture of a face is worth about 200 neurons."

The clinching piece of evidence that cells are coding axes and not specific faces was the finding that for every cell, Chang and Tsao could engineer a large set of faces that looked extremely different, but which all caused the cell to fire in exactly the same way. "This was completely shocking to us -- we had always thought face cells were more complex. But it turns out each face cell is just measuring distance along a single axis of face space, and is blind to other features," Tsao says.

"The way the brain processes this kind of information doesn't have to be a black box," Chang explains. "Although there are many steps of computations between the image we see and the responses of face cells, the code of these face cells turned out to be quite simple once we found the proper axes. This work suggests that other objects could be encoded with similarly simple coordinate systems."

In addition to its implications for studying the neural code, the research also has artificial intelligence applications. "This could inspire new machine learning algorithms for recognizing faces," Tsao adds. "In addition, our approach could be used to figure out how units in deep networks encode other things, such as objects and sentences."