Great puns! Looks like some fun projects awaiting Monty!
Sorry it is still a bit involved to get Monty to run in new applications. It will hopefully be a bit more intuitive to understand once we publish more materials and examples with the (team internal) robot hackathon.
In the meantime, just a few thoughts/hints:
- The intended interface between Monty and any hardware would the
EnvironmentDataLoaderandEnvironmentDatasetclasses. Monty is designed with the idea that any of the general components (defined as abstract classes in the code) can be customized, and so you can easily plug and play with the different components (as described here in this video and in the docs) - We have several custom environments implemented already, which are not Habitat. You can see a list of them at the bottom of the documentation page on Environment and agent.
- The vast majority of our experiments currently happen in simulation, but we did test Monty on real-world data during a previous hackathon. There, we streamed data from an iPad camera to the laptop. Then, Monty could virtually move a small sensor patch over the full image. See the demo video here: Project Showcase Code is also available and there is a set of MontyMeetsWorld benchmark experiments that we run frequently.
Generally, for a robotics project, I would currently recommend a similar approach as we took for this little showcase project:
- Stream the data from your sensors to your computer, where Monty runs.
- Feed the incoming sensor data to Monty. It will first go into sensor modules, which convert the raw data into the CMP, then to learning modules, which model the data, and finally, the motor system can translate CMP-compliant goal states into motor commands specific to your robot’s actuators.
- Stream the motor commands back to your robot.
Depending on the application, you may have to implement a custom sensor module and/or motor system in addition to the custom EnvironmentDataLoader. We hope to publish more concrete instructions and guidelines on what applications are possible and which ones are currently not. I hope this helps a little already. Hang in there, we are doing what we can to lower the barriers!
-Viviane