Apologies in advance, as I haven’t been able to catch up with the wonderful theory videos. If this has been tackled anywhere already, feel free to ignore or link to it.
I’ve been revisiting thoughts and learnings, and pondering on the questions asked about the theory. These often result in answers, explaining that movement mustn’t be physical.
The current categorization of TBP is “sensorimotor learning systems”, pointing out the central role of movement. This is, of course, explained here and by other authors well in context of evolution of brains. However, there’s another central theme, if I remember correctly, already tackled in “On Intelligence”: change.
Since neurons or one could say brains can’t tell movement from virtual movement (“imagination”, etc) as they sit in the skull’s “black box”, the common subject for learning is change, be it due to movement, or due to more sensor-detached brain activity.
I’m not yet sure, how one would communicate it tersely but perhaps collective intelligence comes up with something. At least my perception is that “sensorimotor” can be understood by those who have tackled the subject for long enough but perhaps not by newcomers, used to unidirectional learning approaches. They might misinterpret what sensorimotor means and e.g. converge on a shallow idea that the learning system must have sensors and motors, or it must be “embodied” in the sense of coupling a (virtual) robot to the learning system, which might fall quite short of a meaningful sensorimotor learning system.
A sketch for a sentence (thrown tomatoes welcome): “Sensorimotor learning systems seek to actively cause change in inputs, e.g. by means of movement of sensors, thus sensorimotor”.
I tried looking for the word “change” on the readme site but the search resulted mostly in the contribution process sentences.
listening to music isn’t a passive exercise. The brain continuously predicts changes in the melody. If there would be an unexpected change in the melody, that would cause a prediction failure and potentially learning, e.g. that the record is scratched at a particular point in the song time.
If my memory isn’t lying, the HTM school had direct explanations on the example of melodies. The “movement” in the passive listening to music is the location within the song (easy to grasp: which verse / chorus or simply where in the song), and the imagined model of the other dimensions of the song - pitch, or even more abstract: story, color, mood, etc. Sheet music readers have 2 natural axes: the bars (horizontal) and pitch (vertical), augmented by many more, depending on the listener. One can mentally go back and forth in the song (again, virtual movement or change). Sheet music readers read bars in advance, a mental forward-looking movement.
Another example of how music listening is not passive: mental or real dancing to the music. Nodding the head to the beat or gesticulating anything - this is predicted movement that results in something positive for the listener. Why something positive to the listener? This is, in my view, superbly explained by Lisa Feldman Barrett in her various “How Emotions are Made” sources - since brains are there to help animals survive and reproduce.
You bring up an interesting question, what exactly is sensorimotor learning and sensorimotor inference? Is it about moving our sensors or detecting change? I apologize if I do not understand your question correctly. FWIW here is how I think about this question.
One of the primary goals of the neocortex is to learn a model of the world.
This assumes that the world has persistent structure.
There are two types of structure, spatial and temporal.
Spatial structure, such as the morphology of a mug or the arrangement of rooms in a house is learned by moving our sensors and keeping track of what is observed in a reference frame. How the inputs change depends on how we move.
Temporal structure, such as a melody, by its nature does not require movement to learn.
In the brain there isn’t a hard line between models of spatial structure and models of temporal structure. We have shown that the same neurons that learn spatial structure will learn temporal structure if it exists.
Now to your questions/observations about change
In On Intelligence, I wrote about the importance of detecting changes in our environment. This was not about sensorimotor learning, or changing sensations caused by movement, but how we detect changes to things we already learned. E.g. the altered door.
The main argument in the book was that our brains must have a detailed model of the world. It uses this model to predict its input. If a prediction is wrong, the brain knows it has to update its model. Prediction is not the end goal of the cortex, but it is a ubiquitous property, and therefore we can use this property to figure out how the neocortex works. That is pretty much what we did and continue to do.
We now know in detail how cortical structure learns spatial/temporal models and how neurons make predictions. Once the cortex has learned a model and if an input doesn’t match the model’s prediction, then attention is drawn to area of the misprediction, and cortex modifies its model.
We don’t have a solid theory about how the cortex thinks without acting. This doesn’t seem to be a difficult problem; we just haven’t focused on it yet.
More challenging is understanding how a mechanism designed for sensorimotor learning about the physical world also applies to more conceptual knowledge. We are currently working on this. What is clear is that the cortex uses the same basic principles for representing knowledge about the physical world and conceptual knowledge. The anatomy tells us this. It also makes sense that all knowledge is organized via reference frames. The open question is, is the “movement” component truly abstract or is it still physical. E.g. We learn physics by drawing images of things like atoms and molecules as if they are physical structures. It is hard to learn math without physical analogies. These observations suggest that physicality underlies conceptual knowledge.
The sensory motor model says we move our sensors to understand objects, e.g. when first given a stapler we open and close the stapler to get an understanding of its functioning.
In sortware development it is the same. When given a program to understand (that someone else has written), you manipulate the program just to gain an understanding of its functioning. Like if I make this function call that other function instead, will it still compile, how does the output change.
After having gained an understanding you undo all changes because their purpose was just that, gaining an understanding, not to make improvements to the program.
@jhawkins thank you for the clarification on the current thinking and understanding. I guess, my proposal is jumping a bit ahead of the evidence on abstract thought. I (amateurishly, of course) see hints on abstract thought being related to same mechanisms as sensorimotor learning as you described, e.g. in spatial mnemonics, synesthesia, melody and music examples in general. The example of abstract geometrical models with the molecules you wrote is another superb one. Humans, having dexterous hands, are likely to have experience with ball-like objects like pebbles, resulting in models via analogies that seem to then work in somewhat more scientific rigor.
I guess, one cannot yet claim beyond reasonable doubt that there’s movement in abstract thought via the same mechanism as in the sensorimotor learning model. Which is fine, as it’s arguably better to remain grounded in a good understanding of evidence.
In regards to conceptual knowledge, do we know if in the brain this also has to be represented in a 3d space ? it seems reasonable to think it does but I’m curious if this is known to be highly likely or not. Physics seems like the obvious example where even the more abstract concepts are always grounded in something very physical at human scale ie the abstractions are always of something simple. But i think there could be examples of conceptual knowledge which dont build on material objects whatsoever. Perhaps the overall ‘world model’ which is grounded in such things is utilised here. Regardless it is facinating to imagine how TBT would work for coneptural ideas and abstract non physical spaces.
FMRI studies show that humans organize object types using grid cell like reference frames in the neocortex. In one study the axis of the reference frame corresponded with neck and leg length. This is evidence that cartesian-like coordinates are not always used. Also, the mechanism neurons use to create reference frames suggest that the brain could learn reference frames of more than three dimensions, although there is no direct evidence for this.
Thank you for your reply. I’ve just joined this fourm and i think its amazing that you guys are engaging with people such as as myself who dont even have a formal background in neuroscience. I’m hoping to use monty for my masters project but aside from that hopefully ill have something to contribute at some point.