AI Is Seeing Double

Artificial neural networks are, of course, designed to mimic the function of their biological counterparts. For this to be possible, we must first understand how the brain works. If our assumptions about its structure or function are flawed, our digital approximations of the brain will be similarly flawed. While much progress has been made in this area in recent years, the operation of the brain is still poorly understood.

The brain does not easily give up its secrets. Even state-of-the-art imaging techniques give researchers a very poor, low-resolution view of the brain’s functions. The rough views of large regions of brain activity shown by such scans do not give researchers in the field a deep understanding of the brain’s complex processes, but rather leave them with a lot of guesswork that is mockingly referred to as “blobology.”

Reexamining our assumptions

Given the state of the available tools, we should expect significant revisions in our understanding to be made from time to time. It has long been believed that of the two streams of the visual system, the ventral and the dorsal, the ventral stream was responsible for object detection and the dorsal stream was responsible for processing spatial information. But recent research has upended this assumption, suggesting that the ventral stream also processes spatial information.

Since many object detection models (such as convolutional neural networks (CNNs)) are roughly modeled on the brain’s ventral stream, this research suggests that they might be useful for recognizing spatial information as well. A team of researchers at MIT set out to determine if this was in fact the case, as this could dramatically change the way we apply these models to problems in the future.

To test out their suspicion, the team trained CNNs not on standard object recognition tasks, but instead on spatial tasks — such as estimating an object’s position, rotation, and distance. They then measured how well these models aligned with real neural activity in the brain’s ventral stream.

Two for one special on AI

Surprisingly, models trained on spatial tasks performed just as well in predicting neural activity as those trained on object categorization. This neuro-alignment suggests that the ventral stream might not be optimized solely for object recognition, as has long been assumed. Instead, it may also be tuned for analyzing spatial information, or possibly even optimized for both, as recent work has suggested.

Using synthetic images generated with a 3D graphics engine, the researchers trained CNNs on varying combinations of object categories and spatial latents (such as pose or location). They discovered that CNNs trained to estimate just a few spatial latents could match the neural alignment performance of models trained on hundreds of object categories.

Digging deeper, the team used a method called centered kernel alignment to compare the internal workings of different CNNs. They found that in the early and middle layers of the networks — where the core visual representations are formed — the models trained on different tasks were strikingly similar. This overlap suggests that many visual processing tasks share a common representational base, and that variation in task-specific output might emerge only in the later layers.

Taken together, these findings strongly hint that CNNs are effectively mimicking some aspects of the brain’s ventral stream. Furthermore, they are useful for more than object detection — they also excel at processing spatial information. This greater understanding will allow developers to use these models to their maximum benefit in the years ahead.