How does the body integrate our disparate senses?
I will provide a new explanation for this important issue, but it will turn the brain upside down. Hang on.
First, let’s review the issue for those not familiar with it.
The traditional view of the brain is that of a computer, with sensors running to the brain, and flexors controlled by it. This is known as the input-process-output (IPO) model of the brain. It is still taught in many universities, though its days are numbered. It has many problems explaining human behavior, and this question brings up one of them.
It takes much different amounts of time for various senses to reach the process step, the muscles must benefit from all of them, and the time from an event happening until we respond must be as small as possible, preferably zero or better. Amazingly, the human body is able to meet these exacting requirements, even the one requiring “better”. The IPO model doesn’t even begin to deal with this.
The new view of the brain is that of a prediction engine (called PP for predictive processing). In case you think I’m making this up, please watch Anil Seth’s 2017 TED talk Your brain hallucinates your conscious reality. Dr. Seth is one of many leading edge neuroscientists and philosophers exploring this view.
Careful physiological studies have found how long it takes for the brain to process sensory input. Touch, including heat and cold, takes about 10 milliseconds to reach the brain and another 100 milliseconds to interpret it. Pain can take aa second (1000 milliseconds) to reach the brain, though further processing is limited. Auditory sensation takes about 15 milliseconds to reach the brain and another 200–300 milliseconds to turn it into words. Visual sensation appears in the eye immediately, then takes 25 milliseconds to reach the primary visual cortex (V1), but turning it into faces and objects takes another 300–400 milliseconds. Further processing vision into actions and intentions can take another 200–400 milliseconds.
So when something happens in the world (an event), the brain has to wait for 110 ms for touch, 250 ms for sound, 350 ms for static vision, 600 ms for dynamic vision, and 1000 ms for pain reception. If it doesn’t wait, it doesn’t get all the information. If it does wait, it might be dead from an accident or attack. Of course, this isn’t how the brain works.
It took neuroscientists so long to reach this understanding because the brain’s solution is so fantastic, in the original and obsolete meaning of that term: “Existing only in imagination; proceeding merely from imagination; fabulous, imaginary, unreal”.
The brain maintains a predictive model of the world outside its skull. It does this all the time, and the model is what we experience as reality. What the senses do is correct the model, not drive it. The integration of the senses occurs in the model, as various senses report what is happening. If a sense takes a longer or shorter amount of time to report, the model might get adjusted differently, but there is no confusion about what happened, because the predictive model is all each of us really knows about. Specifically, our memories are of what happens in the model, not in the real world through the senses.
What we traditionally thought of as sensory processing areas, such as the various visual and auditory processing regions and the somatosensory cortex are really two-way streets. Predictions from the model are pushed down through them to the thalamus, and what comes back up are not sensory impressions, per se, but error corrections for the predictions. We’ll talk about what happens when the predictions reach the thalamus, but I’ll say quickly here that when error corrections reach the predictive model, the corrections are generally incorporated into the ongoing model.
When the predictions pass downward through the sensory processing areas, they eventually reach the thalamus. On the way down, they pass through the primary auditory cortex for sound and the primary visual cortex (V1) for vision, predicting what each of them will experience. What reaches the thalamus is predictions for what the raw sensations will be. For audition, the predictions reaching the thalamus are not for words or even for specific sounds, but for loudness and pitch levels in various locations around the body. For vision, the predictions reaching the thalamus are for luminance (and perhaps color) levels across the entire field of vision. The thalamus compares these low level predictions with sensory input that arrives in the thalamus and responds appropriately.
Now I can explain how people can act in an integrated fashion in spite of varying sensory processing times. The model learns how long it takes predictions for each sense to reach the thalamus and throws off predictions for each sense at just the right time to reach the thalamus for some intended “now”. So the predictions for vision are sent first, followed by the auditory predictions, followed by the touch predictions. Then the predictions for the next “now” are sent, and so the brain is predicting sensations every living moment, all arriving at the thalamus at the right moment for the particular “now” for which they are intended.
This explains how humans regularly demonstrate reflexes that appear faster than possible. The baseball batter’s predictive processing uses a myriad of visual cues and previous memories to predict where the pitched ball will arrive over home plate, in spite of there not being enough time between the release of the pitch and its arrival at the plate to process its trajectory and swing the bat. Our predictions are not perfect, but they work often enough that we survive as a species.
The brain doesn’t predict all the senses. Olfaction is perhaps the most primitive sense, so the olfactory bulb reports directly to the olfactory cortex, with only limited connection to the thalamus, with no evidence of top down prediction. It’s also arguably the most complex sense, with an ability to distinguish among one trillion scents, perhaps making prediction hard. We also don’t predict pain sensors, called nociceptors. Nociceptors report to the thalamus through unmyelinated fibers in the spinal cord, so they are slow to report and hard to predict. Proprioceptors provided feedback from muscular activity caused by motor neurons. While they return through the thalamus, they are really part of the muscle control system and not predicted by the model. They do have expected values in the motor cortex, so if they return unexpected values, we experience pain, including phantom limb pain.
This has been the story of how the senses are integrated. The story of predictive processing goes far beyond that, with the primary sensory areas and thalamus working with the amygdala to allow us to navigate an unpredictable world. But that’s a story for another day.
