What computer models can teach us about how the brain creates behavior. Our ongoing coverage of SfN 2017

My sister had an elderly cat named Merlin, who periodically used to wail around 4:30 am until we came downstairs. He wasn’t hungry, he didn’t want to snuggle, and he didn’t want to go outside. I have always been curious about the trigger for this abnormal behavior and whether it was learned or instinctual, so I was intrigued to hear how scientists are using sophisticated tools to decipher the silent language of mouse behavior.

At a Charles River luncheon that coincided with the Society for Neuroscience meeting in Washington, DC, Sandeep Robert Datta, an assistant professor of neuroscience at Harvard Medical School, described how his lab has combined 3D imaging and machine learning to reveal that mouse behavior is a series of reused and stereotyped modules—which they refer to as syllables—with defined transition probabilities. His computational analysis not only provides a way to study the language of normal animal behavior, it can also be a tool to study the effects of drugs and disease mutations, which I’ll discuss shortly.

Unlike early 20th century biologists Nikolaas Tinbergen and Konrad Lorenz, who shared a Nobel Prize for painstakingly studying animal behavior in their natural environment, Datta’s lab is leveraging state-of-the-art technology to objectify mice interaction and to understand how the brain creates behavior.


To do this the lab recorded the mouse’s posture and from there built a 3D pose model of the mouse that they fed algorithms normally used to discover parts of speech in a foreign language. With mathematical models they were able to define 60 syllables and then instruct the computer how to find them in the mounds of data they generated.

When they stacked video frames of each mouse on top of one another it soon became clear that the mice doing very different things, only to snap together and perform something very stereotypical, an almost predictable rhythm to the way the mice moved. The 60 distinct syllables each demonstrated unique repeated motifs of action lasting around 300 milliseconds. Each of these sub-second blocks of behavior appeared to encode a recognizable action, such as darting, sniffing, bunching itself into a ball or assuming the hunter’s position, where it gets down on four legs, stretches and raises its nose.

The lab observed that the mouse’s basic lexicon of around 60 syllables didn’t change at all over time, what did change were the different ways they used the syllables to create new behaviors. The lab has likened this to the way a human can take a finite number of sounds to speak different words.

More recently they have begun exploring the practicality of using their tool in mouse drug studies. In one recent behavioral study, they dosed 1,000 mice a dozen different psychiatric drugs ranging from antipsychotics to antidepressants and found that the behavioral fingerprints created for each mouse reflected the diagnostic criteria of each drug.

They also are leveraging the technology to better understand how the environment, genes and neural circuits influence different patterns of action. For instance, they recently conducted deep phenotyping of three different genetic mouse models of autism spectrum disorder (ASD) to measure differences in behaviour among wildtype and ASD mice. Their findings revealed distinct differences in the mutant strains. They also measured the effects of a psychiatric drug, risperidone on an ASD model absent a deficient form of CNTNAP2 associated with some psychiatric disorders. They learned that it impacted just one of the eight syllables affected by CNTNAP2, a clear indication the drug wasn’t affecting the behavior of the mice.