I’m interested in using Monty as a pattern-recognition sub-agent, somehow providing a summary of conclusions made to another system that is focused on executive function, action generation, goal planning , etc.
Where can I learn more about what sorts of interpretable data Monty produces?
Search for CMP or “cortical messaging protocol” here in research and theory
Stumbled on one: 2023/06 - The Cortical Messaging Protocol
Hi @Judah_Meek,
There are two types of information that you may be after here:
1. Monty’s final converged answer
Currently, the output of Monty that you probably care about are most_likely_object and most_likely_rotation. Both of these are output through the logging system. This could be where you could integrate an extension point as a proof of concept for connecting to a third party system. You can also look at the Wandb (src/tbp/monty/frameworks/loggers/wandb_handlers.py) or csv logger (src/tbp/monty/frameworks/loggers/monty_handlers.py). The logged .csv files contain a result column for seeing what Monty has converged on - this column could also contain a list of objects, or ‘no match’.
2. Hypothesis at every step
The CMP is the main output of the LM communicating it’s current hypothesis, if it is above a confidence threshold. If you want to know Monty’s hypothesis at every step, you can look at lm.get_current_mlh() which is a dictionary that includes the hypothesized object ID, orientation, location, and amount of evidence for this hypothesis.
@DLed has recently put together a live view extension that could be an interesting starting point as well. What if (?) Monty had a live view.
Will
I’m grateful for this explanation.
Where does Monty get the terms that it uses to fill most_likely_object?
What does Monty do if none of the existing terms satisfy its set confidence levels?
I suspect there’s a possibility that humans use the same general process that Monty uses to interpret real-world data to abstract clusters of connotations & concepts into neologisms, so I’m really interested in what Monty does if there is no set term to describe what it perceives.
Hi @Judah_Meek, if you’re referring to the names of objects, those are currently provided by the dataset. For example, YCB includes labels like mug, spoon, etc.
most_likely_object is determined by the most-likely hypothesis, which is extracted from the LM.
You can see where these appear in the experiment configuration in this tutorial, particularly the YAML shown here: Pretraining a Model
For contrast, you may also want to look at the unsupervised continual learning tutorial, where Monty learns continuously without labeled object data: Unsupervised Continual Learning
If none of the objects meet a confidence threshold, Monty will output no-match
There’s also a brief discussion of generalization and categorization in the following video where Viviane walks through the behavior. In cases where Monty can’t find an exact match, it tends to settle on structurally similar objects much like a human would: https://youtu.be/yJBhZkkZ-XM?list=PLXpTU6oIscrn5ImFcRJQ3ameHbMWNd9Df&t=3302
I appreciate the effort that you put into your reply.
To clarify why I asked my question: I’ve got a theory that the main obstacle to AGI is the ability to create abstractions & that humans primarily (not exclusively) create abstractions through language.
More specifically, we compress clusters of connotations, sensory experiences, and concepts into a new term or phrase (neologism). We further abstract that neologism by sharing it with others & only neologisms that are abstract enough to be relevant to others spread (& their range is determined by their abstraction/relevance)
I was hoping that monty might be able to give me a clue how to implement this abstraction process.
Based on the information that you’ve provided me, it looks like this abstraction process is just outside what monty does: monty finds existing terms for what it perceives, but abstraction through neologism would require choosing a neologism (probably best to start with a phrase instead of single word, since that would enable the use of existing terms through combination) for what it perceives.
This conversation alone has helped me process through my conceptual exploration of the problem domain, but if you have any thoughts on how monty’s implementation could be changed to make it create neologisms, I’d be really grateful.
Hi @Judah_Meek, yes, Monty doesn’t currently handle abstract concepts. That’s on our roadmap, and the link below outlines the rough sequence in which we plan to add capabilities:
Hi @Judah_Meek , just to add to the information that Will has shared, I would highlight again the existing version of unsupervised learning in Monty that Will linked. During unsupervised learning, if Monty perceives an object that it believes is similar to something it already knows, it will combine this information, forming a model that integrates these representations. If it feels the object it is learning about is something very different, it will learn an entirely new model.
Right now, unsupervised learning is very basic in Monty, as we have been prioritizing compositionality and object behaviors. However, there is already a new type of LM (constrained object models), which should make the above unsupervised learning work even better as a way of “clustering” similar objects in the world. You might therefore be interested in checking the Future Work item here, which basically consist of testing this already implemented method out.
As a final comment, our general view is that representations and concepts emerge early in life prior to any form of language. Language later becomes a way of referencing these representations, but they are fundamentally learned in a sensorimotor way. For more on this, you might find this thread interesting.
Although I strongly suspect that language helps humans to create and use abstractions, other animals appear to have some capabilities in this direction. Here’s a Google AI search I ran:
Which animals (other than humans) seem to be able to form and use abstractions?
FWIW, about 75 years ago, S.I. Hayakawa wrote a fascinating and very accessible book on general semantics called Language in Thought and Action.
I did qualify my claim with “primarily, not exclusively”. I’m aware that I can create abstractions without language or internal dialogue. Language just makes it an easier process (by serving as a form of memoization, for one thing, I suspect).
I appreciate the book reference. I’ll definitely check that out!