Questions regarding environment setup

Monty Code
1
TL;DR

I do not recommend trying to test Monty on a Windows system. While it may be doable, I do not believe it will be stable. And in my opinion the time investment is not worth it. The issue isn’t with Monty itself, but with Habitat-sim.

For the Monty team,

I would strongly consider putting up a warning regarding Habitat-sim and Windows on your getting started guide. I don’t see how this doesn’t leave anything but a bad taste in peoples mouths as they try Monty for the first time.

Hey there guys,

I have some pretty embarrassing questions that I’m hoping you can help me with.

I just wasted a stupid amount of time trying to get Monty deployed onto a windows system (I happen to have an Nvidia 4070 Super on this computer, hence why I wanted to use it). Needless to say, I had problems…

To start, apparently Conda does not support habitat (or wget) natively on Windows. So instead, I tried using a WLS install of Ubuntu (I’ve done something similar for Kali linux in the past, which worked fine). However, it turns out that WLS uses an older version of the Mesa OpenGL drivers which I, for the life of me, could not get to update past 3.x. This caused issues when running benchmarks (which apparently required 4.x). There were other issues present here (for instance, why I am I using OpenGL drivers instead of Nvidias), but those issues were beside the point…

Anyways, carried onward by sheer stubborn dint, I decided that I’d just virtualize Linux using VirtualBox. Sure, this also means I’d lose any potential for GPU acceleration, but at this point I didn’t care. I just want to see the bloody thing begin to work. However, I couldn’t even get past the pytest stage in this scenario. I kept running into weird segmentation faults and memory issues. I’m assuming these were all downstream resource contention issues given the virtualized nature of the OS. I probably could have tried and debug the errors, but at this point I was admittedly annoyed.

I’m on the verge of digging out an old server of mine. Its a dual Xeon rig with 16 cores total (can’t quite recall the clock speed) and about 48 gigs of DDR3. I could throw an old Nvidia GTX 1080 in it if need be. However, before I waste my time doing that, I have to ask: what’s the minimum recommended hardware / suggested environment for testing Monty on? Would I just be better off getting an M3 MacBook for this?

Also, has anyone here managed to get this working on Windows? If so, I’d love to know what I’m doing wrong. Otherwise, if not, I don’t think it would be such a bad idea to include a warning on Monty’s ‘getting started’ page on Github regarding Conda’s compatibility issues on Windows (as it pertains to habitat). I understand that this is likely a me issue (I’ve never used Conda before), however, in the off chance its not, I’d hate to see other people waste eight hours of their day with nothing to show for it.

P.S. last question, does Monty actually support CUDA? We can obviously run it parallel on the CPU, but does it currently support GPU acceleration?

Edit: I revisited this in the evening. Got some of my issues sorted out, though WLS2 wonkiness is still plaguing me. At this point my main issue has to do pytest workers crashing. So likely a habitat-sim issue. I worked on debugging it a little bit. It looks like Habitat-Sim is trying to use EGL for offscreen rendering, but can’t find a compatible EGL device. Its bit weird though because I can render other things to the display. For instance, the glxgears animation. This is an issue for both NVIDIA OpenGL as well as Mesa D3D12. Theres also something weird going on with the driver library install location. But I’ll deal with that another time.

I’m probably just going to nuke this Linux distro and start clean again in the morning. I’m recording my steps along the way, so if I’m able to consistently get a good, working install on Windows I’ll put together a comprehensive guide on getting this stupid thing running via WSL. Ideally with full GPU acceleration. I’m seeing some promising information on people getting CUDA working with WSL through docker. So I might look more into that tomorrow. But anyways, I’m hopping off for now.

Edit 2: Still no luck with WSL2. I was occasionally able to get it to the point where it wasn’t crashing workers, but it ultimately just wasn’t very stable. For instance, when I’d run the ‘python benchmarks/run.py -e randrot_10distinctobj_surf_agent,’ the D3D12 wrapper would cause weird display issues. I would work through those then I’d notice only one object (generally the “mug”) would load, and more often then not the benchmark would run indefinitely, never finishing. Ultimately I think the issue is a compatibility one between D3D12 and Habitat-sim. Nvidia OpenGL works perfectly fine through WSL2, however in order to render, OpenGL needs to pass through Microsoft’s D3D12, which Habitat-sim does not work with. The dev team behind Habitat-sim have gone so far as to say they don’t support Windows environments.

I ended up just throwing a spare disk drive into my computer and installing Ubuntu on that. Everything installed successfully to that within maybe five minutes. Three minutes after that I successfully finished that first 10distinctobj benchmark. All in all, about 10 -15 minutes to get up and running this way.

I would really, really consider putting up a warning regarding Habitat-sim and Windows on your getting started guide. I don’t see how this doesn’t leave anything but a bad taste in peoples mouths as they try Monty for the first time.

5 Likes
2

Sadly, I have no answers to offer you, but I commend your energy and persistence. I also encourage you (and any other explorers) to keep posting with open questions, results, etc.

1 Like
3

@HumbleTraveller I will make sure the docs are updated to reflect this. We (I mean 100% Tristan) is working hard to remove this dependency so we can allow Windows development.

2 Likes
4

Hi @HumbleTraveller,

Thank you for trying, and I apologize for it not working.

As you found, HabitatSim is not supported on Windows. As a result, we’re pursuing two main paths forward. One is to decouple Monty from a specific simulator implementation, like HabitatSim. The other is to integrate with a simulator that does work on Windows.

On the path of decoupling Monty, it should now be possible to run Monty benchmarks/experiments that do not require HabitatSim. This is only marginally useful, as the tests and development still rely heavily on a simulator. The next decoupling tasks on this path are to 1) separate unit tests from integration tests, 2) remove HabitatSim dependency from unit tests, and 3) decouple Monty integration tests from a specific simulator, like HabitatSim. We believe Monty will be fully decoupled from HabitatSim once this work is completed. However, with HabitatSim being the only simulator available, Monty will still rely on it. This brings us to the second path, integrating a different simulator.

The second path involves integrating a second simulator. Once a second simulator is integrated, Monty should become more accessible, assuming the second simulator works on all platforms. One benefit of the already completed initial decoupling work is that it grouped the HabitatSim simulator code into simulators.habitat. This may serve as a guide as to what code needs to be written to integrate a second simulator. As to what the second simulator could be, I shared my thoughts on it so far in The Next Monty 3D Environment Simulator post.

This diagram [1] illustrates what I described above, highlighting work to be done as “injections.” Injection #11 has been completed so far. The general way to read this is that things above require the things below.

2025-01-03 - TBP Code Roadmap - Simulators
2025-01-03 - TBP Code Roadmap - Simulators2004×1486 360 KB

Monty’s current target is the CPU (we use Numpy v1 for computations). Monty on the GPU is future work. HabitatSim does expect a GPU, which we emulate via xvfb in some cases.

  1. Technically, this is a Future Reality Tree. ↩︎

2 Likes
5

Hey there @tslominski!

Definitely no hard feelings towards you or the team. Literally the day after I’d finished editing this post, I’d noticed you guys mentioned the Windows compatibility issue on, literally, your most popular YouTube video (one from back in Dec). Then I noticed your post and all the other post related to this. Also, for whatever it’s worth, xvfb-run was how I got around some of the issues on Windows!

But anyways, thank you for your indepth follow up. I only wish my programming skills were more up to snuff, so I could help out some here!

3 Likes
6

Great to see the persistence @HumbleTraveller and thanks @tslominski for sharing our detailed plan on how to tackle this issue.

Just to highlight here in case someone else comes across this post and likes to contribute:
If you have a simulator that runs on Windows (or another environment in general) that you would like to integrate with Monty, we very much welcome your contributions! This is a great way you could help support this project.

Monty has been written with the idea that it should be easily customizable and people can plug and play with the different modular components. One of these is the environment that Monty interacts with. To add support for a new environment, one needs to write a custom EnvironmentDataset and/or EnvironmentDataLoader. Then, any Monty configuration can be put into this environment and tested there.

We have a guide on customizing Monty here: Customizing Monty
You could also take a look at the environments we already support and how their custom classes are structured here: tbp.monty/src/tbp/monty/frameworks/environments at main · thousandbrainsproject/tbp.monty · GitHub
At the bottom of this page you can find a description of each of the environments we currently support (including non-habitat environments that run on Windows).

If you’d like to take a crack at adding support for another environment and have more questions around this, just let me know.
As a little fun bonus, if someone is interested in helping with this task, we will also add a little player icon for you on our project roadmap (for “Implement an alternative simulator to Habitat that runs on Windows”) :slight_smile:

2 Likes