Good points! I like your way of putting it @RDaneelOlivaw to directly implant knowledge into Monty, although I agree with @HumbleTraveller that GPT may not be the best fit.
One of the key points of Monty is that it models the sensorimotor data using reference frames, which ANNs like GPT don’t do. This is not to say that one couldn’t figure out a way to adapt an LLM to be able to take a stream of sensorimotor information and output that dataformat again but it will likely not have the nice properties you get with reference frames such as very efficient and robust learning, continual learning without catastrophic forgetting, extrapolating to new tasks, etc (generally ANNs assume i.i.d. data which is not a great fit for sensorimotor applications). Here is a bit more on why we don’t make use of deep learning in Monty: FAQ - Monty
However, your idea is still good. Since Monty is an artificial system, not a real brain, we can easily transfer and “implant” knowledge directly. For instance, you could train an LM once with specific domain knowledge and reuse that LM many times in different setups, give it to other people, etc. You could also directly implant knowledge, like if you have existing 3D models of objects that are relevant for your problem. In general, nothing speaks against also having specific “expert LMs” that work without reference frames or have other non-brain-like capabilities, such as an exact calculator. Many possibilities here.