I think this is a brave undertaking and I wish you the best of luck on your project. Especially working out in the open.
I’m probably missing something significant - but for looking around the docs Monty looks more like a object recognition framework, and that space is just assumed in the pose definitions.
Is high precision 3D Euclidean space assumed? Where do the sensor locations (angles, PoV) and tracking come from? Are you assuming the underlying HW platform provides these?
Hi DanML, Scott here – researcher at TBP. Thanks for the question!
In its current form, Monty is indeed aimed at object (and pose) recognition. We’re hard at work developing theory and implementations for other capabilities (e.g., object behavior, compositional objects and scenes), but Monty’s current capabilities lie within object and pose recognition and intelligently moving sensors in the world to support this.
3D space and the existence of objects are Monty’s inductive biases, and sensor locations and orientations are a necessary component of the system. We think of sensor positions and movements the same way we think about parts of the body; they are given by the system similar to how body parts and their movements are given by our nervous systems (i.e. internal models, efference copies of motor signals, etc.). Right now this is handled by software in coordination with a simulated environment with high 3D precision (see note below), and so hardware signals and tracking are not necessary for running Monty as-is. However, we will be rolling out guides for using Monty for robotics shortly, so stay tuned to see how Monty can be integrated with hardware and what a “live” system looks like under these conditions.
A note on precision: Naturally, the precision of 3D data places constraints on how accurately models can be learned and used for recognition. While our simulated environment does return high-precision data, we do run most of our experiments with location and pose noise, which Monty handles quite nicely. We are also embarking on a robotics hackathon next month, so we’ll be working with real-world noise soon, the results of which we’ll make public.
Hope that helps, and thanks again for the question!