There are a lot of things to learn in the world. If I learn to recognize something, I can’t directly share the model I have. If I train my instance of Monty (with the environment and sensor hardware I have) to recognize something, I hope I am able to share it with the other Montys (no need for them to retrain on a similar object (perhaps they might refine my model)).
Forgive me if my thought is off base
2 Likes
This is exactly how we think about it too. We imagine the possibility of industrial applications that have learning modules pre-trained to recognize objects for that domain. As long as the learning modules communicate through the Cortical Messaging Protocol (CMP) they should be interoperable.
1 Like
If I may… That’s the great beauty of the Monty system with the CCP… As far as I understand it, you can for example have a model with the latest GPT system that can give to the Monty network a huge boost. It would be as if in your brain you get implated directly all that knowledge…At least I see it like that ;).
Would these Montys be sharing qualia?
Outsider speaking in here…
To my understanding, no. Not if, by qualia, you’re referring to subjective, perceptual experience. I believe the way they intend it (taking their mention of “domain specific knowledge”), it would be like if you had, for instance, worked in manufacturing for Company A, but then down the road you took that job experience but applied it to Company B - also a manufacturing company.
In a way, exporting memory across monty’s would be a lot like you carrying that work experience across those different companies.
So while the domain specific knowledge of those monty’s may be shared, their subjective experience likely would not be. It’s sort of like how your own subjective experience of working at each of those two companies isn’t identical. They’re different.
This is probably an incredibly convoluted way of thinking about it, but it’s what came to mind 
Also, I’m not sure how compatible a GPT would be with a Monty network. They almost seem more akin to competing systems to me. I’m sure they’d be able to communicate over something like NLP, but direct interface by way of CMP seems unlikely. At least in the near term.
It’ll be interesting to see how robotic systems unfold over the coming years. Everyone seems to be wanting to integrate transformer models into their hardware, but I actually think something like Monty might make for a superior embodied system. Time will tell.
1 Like
Good points! I like your way of putting it @RDaneelOlivaw to directly implant knowledge into Monty, although I agree with @HumbleTraveller that GPT may not be the best fit.
One of the key points of Monty is that it models the sensorimotor data using reference frames, which ANNs like GPT don’t do. This is not to say that one couldn’t figure out a way to adapt an LLM to be able to take a stream of sensorimotor information and output that dataformat again but it will likely not have the nice properties you get with reference frames such as very efficient and robust learning, continual learning without catastrophic forgetting, extrapolating to new tasks, etc (generally ANNs assume i.i.d. data which is not a great fit for sensorimotor applications). Here is a bit more on why we don’t make use of deep learning in Monty: FAQ - Monty
However, your idea is still good. Since Monty is an artificial system, not a real brain, we can easily transfer and “implant” knowledge directly. For instance, you could train an LM once with specific domain knowledge and reuse that LM many times in different setups, give it to other people, etc. You could also directly implant knowledge, like if you have existing 3D models of objects that are relevant for your problem. In general, nothing speaks against also having specific “expert LMs” that work without reference frames or have other non-brain-like capabilities, such as an exact calculator. Many possibilities here.
2 Likes
I see, I’m glad I got the idea right ;). Yeah, it might be interesting to fine-tune a GPT to accept input and output that is more compatible with CCP and Monty in general. This hypothetical system would be huge in selling the modularity of Monty ;). Anyway, step by step ;).
1 Like
I agree with all the thoughts above and would like to see CMP evolve into a Distributed CMP (my background is distributed systems so I am biased 
).
The DCMP would allow different “Monty’s” to interact - very much like how humans learn from others but perhaps and much faster as the communication would be machine to machine, It would also enable remove sensing where a Monty can access remote sensors - think of a Monty as a network monitor with thousands of distributed sensors and continually learning and controlling the state (and security ?) of the network in realtime.
As others mentioned ( @RDaneelOlivaw and @HumbleTraveller ) interfacing with LLMs would be another part of DCMP. LLMs are static repositories of knowledge that have are being stored in a machine context - maybe not quite capable of being directly interfaced by Monty but maybe next iteration. I see LLMs as equivalent to human libraries, if I want to find something out I go the library and read about it. Today Monty could use a vision sensor to read a book that use a language model to parse the text but if it could access the knowledge directly it would be extremely efficient. A Monty could be “kickstarted” as domain expert by providing access to static LLM knowledge and then dynamically learning more in realtime from sensors - in someways this is how human experts are trained - we go to schools to study and then go get practical experience by doing.
So for all this to happen I think the CMP needs to evolve a knowledge exchange protocol that enables Montys to access remote knowledge and sensors.
2 Likes
Why don’t you post a DCMP topic (I would think sharing Monty learnings would be a possibility of a well constructed DCMP). We’d also need a way to have learning modules on one system speaking to sensor & motor modules on another)
yes, great ideas here!
Our current definition of the CMP focuses on the content rather on the specific format or implementation of messages. The key part of a CMP message is that it contains features at a pose in a common reference frame. And the pose is really the key point here. Every message contains location and orientation information inferred from the movement of sensors.
We don’t want to prescribe anything about whether sensors need to be connected to one agent or to many agents or even to many bodies or no physical bodies at all. You can definitely imagine a bunch of cameras distributed in a hose talking to each other through the CMP or a swarm of robots doing the same thing. Also learning to talk to LMs that contain specific domain knowledge should be possible. These are all future applications we would like to enable.
I’m not sure if eventually, we will need to interface with LLMs or if Monty could just acquire domain knowledge on its own more efficiently (and then, since it is not a biological system, we can simply make copies of such learning modules). Afterall, language mostly evolved for us to be able to communicate knowledge to each other but may not be the most efficient way to do so if machines can simply have other ways to communicate with each other (like using the CMP or copying knowledge). There are tons of thoughts and ideas I have around this but we haven’t implemented anything concrete so I won’t get into too much speculation here. You are definitely bringing up lots of interesting topics!
2 Likes
It would be most helpful if someone intimate with Monty’s inner workings did express how info from a remote sensor, and commands to an actuator might be expressed (as a 1st approximation), in say JSON. I’ve been rummaging thru the code trying to imagine this, to no avail. This would greatly facilitate development of Monty support for a big old real world of sensors and actuators (even on platforms where Habitat Sim doesn’t run).
Kindly, someone, please come hither!
1 Like
The implementation of CMP in Monty can be found in the State class: tbp.monty/src/tbp/monty/frameworks/models/states.py at main · thousandbrainsproject/tbp.monty · GitHub
2 Likes
Although Monty is obviously inspired by neuroscience, this topic veers into (or at least near to) genetics, evolution, etc. Indeed, there’s no reason why sharing of Monty’s Learnings couldn’t support forms of evolution or even Lamarckian inheritance:
Lamarckism, also known as Lamarckian inheritance or neo-Lamarckism, is the notion that an organism can pass on to its offspring physical characteristics that the parent organism acquired through use or disuse during its lifetime. It is also called the inheritance of acquired characteristics or more recently soft inheritance.
Basically, what is needed is a way to encode, record, and transmit sets of acquired characteristics.
Supervised Learning
Assume that we have used supervised learning to train a Monty instance (e.g., to recognize something found in its sensory environment). Because we know the training object (e.g., a cup with a Numenta logo), we can annotate the resulting graph entry with some textual tags:
- cup with a Numenta logo
- cup, logo, Numenta, Numenta logo
Note that some of these tags are simple (e.g., cup, logo), but others are complex (cup with a Numenta logo). With a bit of parsing and such, the latter can be transformed into directed graphs of nodes and edges:
cup -[has feature]-> Numenta_logo
Numenta_logo -[has type]-> Numenta
These graphs can then be stored in databases such as ArangoDB, Neo4j, etc. At a minimum, this would provide a searchable index to the learnings. It might also allow us (via graph matching) to assess how well some learnings might play together, discover useful combinations, etc.
Unsupervised Learning
However, things get even more interesting if we introduce unsupervised learning to the mix.
Let’s assume that we’ve “seeded” a Monty instance with some tagged (etc) learnings. Now, we let it explore the environment, building up a larger object graph. It seems like we should be able to combine our tags and graph info, building up more complex object descriptions.
For example, let’s assume that we’ve put a Numenta baseball cap, cup, and T-shirt into the mix. Couldn’t we use the derived graph to “find” the notion of a Numenta logo?
(ducks)
Has any thought been given to embedding Monty into a VR/AR headset? Or use a headset tethered to a lab computer? The goal would be to enable a person to actually watch Monty working during sensorimotor learning (ride along on Monty’s shoulder). Some of this already exists in projects like Microsoft’s Malmo, where you can watch the avatar agent wander around in a third person view.
It might provide insights into representations, constructs, inferences, and processes like voting. Especially if diverse groups of observers were used (coding geeks all think alike 
)
Modern headsets have significant computing power, along with SDKs and development communities.
Hey @scidata ! I personally haven’t thought about using a VR/AR headset, but I’m definitely thinking more about useful visualizations to add to Monty. One idea was to visualize, let’s say, the top 10 hypotheses of objects (could be same objects at different rotations) that Monty is predicting as it sees a patch, and updating after it moves. Another would be visualizing the object model that is being built as Monty moves.
I’ll take a look at Microsoft’s Malmo, but do you have any other ideas on what kind of visualizations you think would be helpful?