(Note: I am not a TBP team member, just a third-party contributor)
Is this for experimental research, or for an end-user application?
So far, the project has mostly focused on visual recognition, whereas the main hurdle for language is associative learning, which is quite a different challenge altogether.
Imagine attempting to decipher Ancient Egyptian without any Rosetta Stone; that’s what “text-only learning” is. That idea’s already a shortcut itself, tackled semi-successfully by deep learning (and ever so unsuccessfully by symbolic AI).
Embodied learning is precisely about carving a sensorimotor “Rosetta Stone” to address the shortcomings of text-only learning and big data approaches, because language is ultimately grounded in sensory experience, as you acknowledged. It is very likely an embodied AI would have to undergo a training process similar to elementary school before reaching the capabilities you’re aiming for.
At present, Monty has no ability to ingest text data, its algorithms are all geometric and colorimetric. So, if your needs are in the shorter-term for an end-user application, an LLM would provide results quicker. For experimental research however, the road is wide open. This thread has a few details: Abstract Concept in Monty - #4 by vclay
Maybe the TBP research team will provide better guidance, I’m curious myself as to what they have to say.