To mark territory or not to mark territory: Breaking the pheromone code

The findings shed light on the evolution of mammalian behaviors—which include human behaviors—and their underlying brain mechanisms.

“How does an individual respond differently to the environment based on experience? How does it distinguish itself from others? These are some of the fundamental questions that a study like this one helps us address,” said TSRI Associate Professor Lisa Stowers, senior author of the new study, which appears in the April 24, 2014 issue of the journal Cell.

The study centered on an important mouse behavior known as countermarking. In the wild, a male mouse typically marks his territory with splashes of urine. “His purpose is to advertise his presence to females and to scare away subordinate males,” said Stowers. But sometimes the mouse encounters a urine mark made by another mouse. In response, he may leave a countermark—to say, “No, this is my territory.” However, if the urine mark has come from a locally dominant mouse, a subordinate mouse will make no countermark and will simply move on.

The findings challenge the traditional view of how pheromones work in animals such as mice. These compounds have until now been thought to trigger behaviors very directly and simply—one compound, when detected, triggers one behavior—so that behaviors critical for survival and reproduction don’t have to be learned.

Chemical Cues

To understand better the chemical cues that guide countermarking decisions, Stowers and her team, including first author Angeldeep W. Kaur, then a research associate in the Stowers laboratory, examined a special set of 21 molecules in mouse urine. Known as Major Urinary Proteins (MUPs), these suspected signaling molecules are linked to testosterone production and are found at high concentrations in the urine of adult male mice.

MUPs are not odorants detected by the main olfactory system; they are pheromones detected by the vomeronasal organ (VNO) at the front of the mouse nose. Humans apparently don’t have a working VNO, but mice, along with many other mammals, do; and because the VNO serves as a more or less direct trigger of behaviors, it can be a handy and powerful tool for studying such behaviors and the neural logic on which they are based.

A male mouse in the wild expresses his own distinctive combination of several MUPs in different ratios—what has been hypothesized to be a pheromonal “bar code” of identity. By exposing mice—and even their VNO neurons in tissue-slice cultures—to single and combined MUPs, Kaur, Stowers and their colleagues were able to confirm that male mice learn their own self code and can distinguish it from the codes contained in urine marks from other mice.

“When a urine mark contains a set of MUPs that are similar to the mouse’s own repertoire, he won’t respond to it, but if we vary the set of MUPs in the mark, he will countermark,” said Stowers.

The experiments revealed, however, that the mouse won’t countermark a spot of non-self urine if it has previously learned to associate the sensed set of MUPs in that urine with a dominant mouse. This response shows that the VNO is capable of activating behaviors in a manner that is more complex and versatile and less hardwired—and thus is more like other sensory systems found in humans.

“Here it is using what we call combinatorial coding, and at the same time it is dependent on prior learning, and both of those features are also found in the main olfactory system,” said Stowers.

On the whole, the findings indicate that even in the VNO, a small number of cueing molecules, working through a small set of neurons, can flexibly control several distinct behaviors, including behaviors such as rage and self/other discrimination that are also basic elements of the human experience.

“We’re using the VNO as a simple model system here, but the implications of these coding schemes go beyond the VNO—we suspect that they are used generally by the mammalian brain,” Stowers said. “And now that we know these signaling molecules, and the neurons and the behaviors they activate, we can start to follow the circuits into the brain to explore how these behaviors come about.”