There are a lot of things to learn in the world. If I learn to recognize something, I can’t directly share the model I have. If I train my instance of Monty (with the environment and sensor hardware I have) to recognize something, I hope I am able to share it with the other Montys (no need for them to retrain on a similar object (perhaps they might refine my model)).
Forgive me if my thought is off base
2 Likes
This is exactly how we think about it too. We imagine the possibility of industrial applications that have learning modules pre-trained to recognize objects for that domain. As long as the learning modules communicate through the Cortical Messaging Protocol (CMP) they should be interoperable.
1 Like
If I may… That’s the great beauty of the Monty system with the CCP… As far as I understand it, you can for example have a model with the latest GPT system that can give to the Monty network a huge boost. It would be as if in your brain you get implated directly all that knowledge…At least I see it like that ;).
Would these Montys be sharing qualia?
Outsider speaking in here…
To my understanding, no. Not if, by qualia, you’re referring to subjective, perceptual experience. I believe the way they intend it (taking their mention of “domain specific knowledge”), it would be like if you had, for instance, worked in manufacturing for Company A, but then down the road you took that job experience but applied it to Company B - also a manufacturing company.
In a way, exporting memory across monty’s would be a lot like you carrying that work experience across those different companies.
So while the domain specific knowledge of those monty’s may be shared, their subjective experience likely would not be. It’s sort of like how your own subjective experience of working at each of those two companies isn’t identical. They’re different.
This is probably an incredibly convoluted way of thinking about it, but it’s what came to mind 
Also, I’m not sure how compatible a GPT would be with a Monty network. They almost seem more akin to competing systems to me. I’m sure they’d be able to communicate over something like NLP, but direct interface by way of CMP seems unlikely. At least in the near term.
It’ll be interesting to see how robotic systems unfold over the coming years. Everyone seems to be wanting to integrate transformer models into their hardware, but I actually think something like Monty might make for a superior embodied system. Time will tell.
1 Like
Good points! I like your way of putting it @RDaneelOlivaw to directly implant knowledge into Monty, although I agree with @HumbleTraveller that GPT may not be the best fit.
One of the key points of Monty is that it models the sensorimotor data using reference frames, which ANNs like GPT don’t do. This is not to say that one couldn’t figure out a way to adapt an LLM to be able to take a stream of sensorimotor information and output that dataformat again but it will likely not have the nice properties you get with reference frames such as very efficient and robust learning, continual learning without catastrophic forgetting, extrapolating to new tasks, etc (generally ANNs assume i.i.d. data which is not a great fit for sensorimotor applications). Here is a bit more on why we don’t make use of deep learning in Monty: FAQ - Monty
However, your idea is still good. Since Monty is an artificial system, not a real brain, we can easily transfer and “implant” knowledge directly. For instance, you could train an LM once with specific domain knowledge and reuse that LM many times in different setups, give it to other people, etc. You could also directly implant knowledge, like if you have existing 3D models of objects that are relevant for your problem. In general, nothing speaks against also having specific “expert LMs” that work without reference frames or have other non-brain-like capabilities, such as an exact calculator. Many possibilities here.
2 Likes
I see, I’m glad I got the idea right ;). Yeah, it might be interesting to fine-tune a GPT to accept input and output that is more compatible with CCP and Monty in general. This hypothetical system would be huge in selling the modularity of Monty ;). Anyway, step by step ;).
1 Like
I agree with all the thoughts above and would like to see CMP evolve into a Distributed CMP (my background is distributed systems so I am biased 
).
The DCMP would allow different “Monty’s” to interact - very much like how humans learn from others but perhaps and much faster as the communication would be machine to machine, It would also enable remove sensing where a Monty can access remote sensors - think of a Monty as a network monitor with thousands of distributed sensors and continually learning and controlling the state (and security ?) of the network in realtime.
As others mentioned ( @RDaneelOlivaw and @HumbleTraveller ) interfacing with LLMs would be another part of DCMP. LLMs are static repositories of knowledge that have are being stored in a machine context - maybe not quite capable of being directly interfaced by Monty but maybe next iteration. I see LLMs as equivalent to human libraries, if I want to find something out I go the library and read about it. Today Monty could use a vision sensor to read a book that use a language model to parse the text but if it could access the knowledge directly it would be extremely efficient. A Monty could be “kickstarted” as domain expert by providing access to static LLM knowledge and then dynamically learning more in realtime from sensors - in someways this is how human experts are trained - we go to schools to study and then go get practical experience by doing.
So for all this to happen I think the CMP needs to evolve a knowledge exchange protocol that enables Montys to access remote knowledge and sensors.
2 Likes
Why don’t you post a DCMP topic (I would think sharing Monty learnings would be a possibility of a well constructed DCMP). We’d also need a way to have learning modules on one system speaking to sensor & motor modules on another)
yes, great ideas here!
Our current definition of the CMP focuses on the content rather on the specific format or implementation of messages. The key part of a CMP message is that it contains features at a pose in a common reference frame. And the pose is really the key point here. Every message contains location and orientation information inferred from the movement of sensors.
We don’t want to prescribe anything about whether sensors need to be connected to one agent or to many agents or even to many bodies or no physical bodies at all. You can definitely imagine a bunch of cameras distributed in a hose talking to each other through the CMP or a swarm of robots doing the same thing. Also learning to talk to LMs that contain specific domain knowledge should be possible. These are all future applications we would like to enable.
I’m not sure if eventually, we will need to interface with LLMs or if Monty could just acquire domain knowledge on its own more efficiently (and then, since it is not a biological system, we can simply make copies of such learning modules). Afterall, language mostly evolved for us to be able to communicate knowledge to each other but may not be the most efficient way to do so if machines can simply have other ways to communicate with each other (like using the CMP or copying knowledge). There are tons of thoughts and ideas I have around this but we haven’t implemented anything concrete so I won’t get into too much speculation here. You are definitely bringing up lots of interesting topics!
2 Likes
It would be most helpful if someone intimate with Monty’s inner workings did express how info from a remote sensor, and commands to an actuator might be expressed (as a 1st approximation), in say JSON. I’ve been rummaging thru the code trying to imagine this, to no avail. This would greatly facilitate development of Monty support for a big old real world of sensors and actuators (even on platforms where Habitat Sim doesn’t run).
Kindly, someone, please come hither!
1 Like
The implementation of CMP in Monty can be found in the State class: tbp.monty/src/tbp/monty/frameworks/models/states.py at main · thousandbrainsproject/tbp.monty · GitHub
2 Likes