Hello, and thoughts about leveraging LLMs in Monty

[srgg6701]

Hi @avner.peled, @nleadholm (as well as @vclay) and everyone involved in this great conversation! I also considered the problem of language quite intensely and have come up with an idea, that intersects with what has been discussed here and in other topics.
In my view, regarding language, we should first separate this problem into two areas: 1) language as a reference structure to the real world and 2) language as a command interface. For now, I will leave the second area for future discussion/development and will focus on the first one.

@nleadholm

If this enables you to explore an application you are interested in, like issuing a voice command to have a Monty robot retrieve an object that is named, then that is great! (bearing in mind that manipulating the environment is also something that is still in our research pipeline)

Regarding the first problem, I believe we need to somehow tie the Monty’s inner representations with spoken and written words. I agree here with @vclay and @ak90 in Abstract Concept in Monty loudspeakerGeneral referencing the real human experience when we internalise language through interaction with the real world. This statement I consider as a key:

@ak90

we will have to have a supervisor and how do we pass multimodal inputs to monty like both verbal and visual

To understand what the words mean we should connect them to phenomena known to us. So, we need something points certain sounds and words to those phenomena. In real life it is usually other people, and since this development is most intense in the childhood, those are parents.
Now, let’s imagine (I guess – not for the first time ) that Monty is a child. So, we need to attach to it a kind of mentor. The child will show that person what they has in their memory and a mentor will name that thing as it sounds or appears in the written form.
Of course it is impossible to do that literally to assign a real person to the instance of Monty. But we can achieve this goal by exploiting (sorry for slight cynicism ) LLM. Monty can show its representations to it:

Now, Monty can reliably assign words that an LLM presents to its memorized objects and the same principle will apply to sounds. Also, I think that this process can be scaled if we use many instances of Monty and then merge their new experiences in a single model which we may consider analogous to the Foundation Model as used in transformers. That’s the idea in a nutshell. What do you think?