9 Likes
Finally finished watching this.
Good stuff! Just to ensure I understand what you’re putting out, let me recap what I think you proposing here…
Initial encoding of perceptual information is done pre-cortex. Once received, columns can communicate data locally with other nearby columns to achieve fast inferencing time. However, longer range communications are facilitated over by an intermediatory ‘bus’ layer. This bus layer possesses attentional mechanisms. Finally, the hippocampal complex almost serves as a sort of potential final output node for that data’s throughput. I’m guessing the hippocampus is also providing sequencing of that data stream, time-wise. Is that the case?
As for the bus layer. The thalamus seems like the most obvious biological homologue to what you would be looking for (especially if attentional functions are at play). Is that what you guys ended up choosing to go with emulating?
Beyond the thalamus, have you looked into the cingulate cortex as a secondary bus channel? I’ve been working on a similar framework myself (weird personal project) and the cing. cortex keeps striking me as an intermediary layer worth exploring…
3 Likes
Hey HumbleTraveller!
Thanks for joining our discourse server and for your first post 
The AI bus (or as we now call it “cortical messaging protocol” is for sending information between any cortical columns, close or far apart. It defines a protocol of how information is structured in a common format such that any two columns can communicate with each other, even if they are in different areas of the cortex and model input from different modalities.
You are right that a lot of this communication will be routed through the thalamus and that the thalamus plays a central role here. We have a bit more detailed writeup of how we ended up implementing this here and there is a lot more information on this to come.
The key idea here is that there is one common format in which all cortical columns receive and send information. So we are not thinking of a secondary bus channel. How the cingulate cortex plays into all this is something we still need to investigate further.
- Viviane
5 Likes
Hey,
Thanks for releasing this project to us all. A lot of it is going above my head but I am doing my best to keep up with all of you brilliant people.
One thing that stood out to me is Jeff Hawkins’ comment at the end of the video about how biologically constrained the Thousand Brains Project (TBP) should be.
He alludes to allowing a lot of freedom and departure from accurate theories of the neocortex for the TBP. The tradeoff between how far away from the Thousand Brains Theory (TBT) the TBP will stray is interesting, as well as which principles are the “right” ones to keep and which ones we can let go of.
I see in your TBP document that one of the main long term goals is:
to eventually bring together prior work into a single framework, including sparsity, active dendrites, sequence memory and grid cells
(Citing many cool papers from 2016 to 2019 like this one.)
You also say in the short term goals:
In the initial implementation, many components are deliberately not biologically constrained, […]. In the future, these elements will be substituted with more powerful, albeit more inscrutable neural components.
Can we expect a long term focus on Hebbian learning and the types of network described in earlier papers?
Or do you think more about keeping close to your more recent works that integrate things like active dendrites to backprop-based learning systems.
I am a computer architect and I believe the locality of the learning rules matters a great deal in terms of which computing architecture can allow deploying large scale versions of TBP-style models in the long term. My personal belief that local (e.g. Hebbian) learning rules matter a great deal, so this is why I am so interested in how you think about these things!
Sorry for the long question, I am a very excited by this topic!
2 Likes
Hi Xavier,
thanks for your interest and your detailed questions!
The TBP takes a bit of a different approach than prior Numenta work like HTM or active dendrites. In this project, we start with high-level principles of the neocortex, such as sensorimotor learning, learning structured models using reference frames, the general and repeatable structure of cortical columns, and a common messaging protocol between them. We, therefore, started by defining a messaging protocol that encompasses these key aspects. The idea is that people can implement different learning modules, and as long as they adhere to the messaging protocol, they can be used in this framework. Learning modules could be implementing Hebbian learning or an HTM-style learning mechanism (+something like grid cells to keep track of movement through space). However, our first implementation did not focus on these lower-level neural implementations since we needed to figure out the high-level concepts and messaging protocol first.
In general, this is still a system with very local learning in the sense that each learning module learns on its own and can do a lot on its own. There is no global backpropagation (or backpropagation in general) as in ANNs. The framework we are building is an alternative to current AI systems. We are starting from scratch, with the principles that we have extracted from neuroscience, instead of taking existing AI and incorporating some biological aspects into them.
I hope this answer makes sense? I think it will become a lot clearer as we release the next meeting recordings and the code.
- Viviane
7 Likes
Having yet to read the report on how it is actually being implemented, I’m struck by the parallels I can draw between the architecture outlined in the video and the Robot Operating System (ROS).
A ROS application is composed of several nodes (independent OS processes) communicating through topics (endpoints in an asynchronous publish / subscribe messaging middleware). Messages are defined in a multi-platform format, and by convention include a header field with a time stamp and a reference frame ID. Nodes that provide an interface (“bridge”) to sensors and actuators also publish TF (“TransForm”) messages specifying reference frame transforms between the sensor / actuator frame and a base link frame that is the robot’s default reference frame. Because those components can be attached to articulated joints, transforms can be updated over time, therefore a system of buffers allows nodes to “rewind” and find where e.g. a sensor was relative to the robot body at a previous point in time.
The architecture Jeff describes in the video looks a lot like the above: cortical columns (individually or in small clusters) operate independently as nodes, exchanging messages defined in a common format through a bulletin board-like bus. Reference frame transforms between body parts are collected by a memory component; assuming messages are tagged with a timestamp and a reference frame, this allows nodes to correlate messages from different sources even if communication is asynchronous.
If this is roughly how the Thousand Brains Project system works (again, I still need to read the report Viviane shared), then the conceptual transition from ROS to it would be very natural, which bodes well for attracting new developers. Looking forward to pulling the first code repository commits!
3 Likes
Thanks for sharing and watched it fully. For me the key take away is:
“Apply neocortical principles to create true AI” — Not biologically constrained.
which opens up design ideas of breaking up the complexity into smaller problems to solve and implement. Very exciting times ahead
Having worked with Intel RealSense cameras which uses Visual SLAM before, it was very refreshing. Looking forward to more information in this great community.
2 Likes
Hi @xperroni and @mirroredkube
Thanks for your detailed thoughts and your excitement! Both ROS and SLAM are very interesting approaches with several similarities to our system. Since one of our core principles is that sensorimotor learning and modeling is key to intelligence, the existing approaches to sensorimotor applications and robotics are very interesting to us. It would be great to have closer interactions with the open-source ROS community in the future and to discover more intersections in our approaches!
By the way, we just published a follow-up video where Jeff goes into a bit more detail on the AI bus idea here: https://www.youtube.com/watch?v=VGY2JOHA0vw
- Viviane
3 Likes