I am just learning. I started with the tutorials (to make sure I set up my environment right) and I had no problem running any of the first 4 tutorials. On the fifth Multiple Learning Module Tutorial, I got as far as the step:
“python run.py -e dist_agent_5lm_2obj_train” and got the following error. Have you seen this before?
---------training---------
Traceback (most recent call last):
File "run.py", line 44, in <module>
main(all_configs=CONFIGS, experiments=cmd_args.experiments)
File "/home/jsmith/tbp.monty/src/tbp/monty/frameworks/run.py", line 114, in main
run(exp_config)
File "/home/jsmith/tbp.monty/src/tbp/monty/frameworks/run.py", line 54, in run
exp.train()
File "/home/jsmith/tbp.monty/src/tbp/monty/frameworks/experiments/pretraining_experiments.py", line 124, in train
self.run_epoch()
File "/home/jsmith/tbp.monty/src/tbp/monty/frameworks/experiments/monty_experiment.py", line 495, in run_epoch
self.run_episode()
File "/home/jsmith/tbp.monty/src/tbp/monty/frameworks/experiments/pretraining_experiments.py", line 77, in run_episode
self.model.step(observation)
File "/home/jsmith/tbp.monty/src/tbp/monty/frameworks/models/monty_base.py", line 144, in step
self._exploratory_step(observation)
File "/home/jsmith/tbp.monty/src/tbp/monty/frameworks/models/abstract_monty_classes.py", line 37, in _exploratory_step
self.aggregate_sensory_inputs(observation)
File "/home/jsmith/tbp.monty/src/tbp/monty/frameworks/models/monty_base.py", line 152, in aggregate_sensory_inputs
sensor_module.update_state(self.get_agent_state())
File "/home/jsmith/tbp.monty/src/tbp/monty/frameworks/models/sensor_modules.py", line 506, in update_state
sensor_position = state["sensors"][self.sensor_module_id + ".rgba"]["position"]
KeyError: 'patch_0.rgba'
I can try and take a closer look at this tonight, but at a glance, it looks like maybe something failed to initialize. That last line in your trace, KeyError: ‘patch_0.rgba,’ makes me think that the sensor_module_id call bugged out.
Edit: Hmm… I wasn’t able to replicate, unfortunately. But based on your error, something obviously got messed up during the ‘setup and train’ leg of the tutorial. Are you able to provide your my_experiments.py file? It should be found under /home/jsmith/tbp.monty/benchmarks/configs/.
2 Likes
Hi @Jeff_Smith, I also attempted to replicate the problem, but I couldn’t get the same error you’re seeing.
Here’s what I have for my_experiments.py:
file contents
# Copyright 2025 Thousand Brains Project
#
# Copyright may exist in Contributors' modifications
# and/or contributions to the work.
#
# Use of this source code is governed by the MIT
# license that can be found in the LICENSE file or at
# https://opensource.org/licenses/MIT.
import os
from dataclasses import asdict
from benchmarks.configs.names import MyExperiments
from tbp.monty.frameworks.config_utils.config_args import (
FiveLMMontyConfig,
MontyArgs,
MotorSystemConfigNaiveScanSpiral,
PretrainLoggingConfig,
get_cube_face_and_corner_views_rotations,
)
from tbp.monty.frameworks.config_utils.make_dataset_configs import (
EnvironmentDataloaderPerObjectArgs,
ExperimentArgs,
PredefinedObjectInitializer,
get_env_dataloader_per_object_by_idx,
)
from tbp.monty.frameworks.config_utils.policy_setup_utils import (
make_naive_scan_policy_config,
)
from tbp.monty.frameworks.environments import embodied_data as ED
from tbp.monty.frameworks.experiments import (
MontySupervisedObjectPretrainingExperiment,
)
from tbp.monty.simulators.habitat.configs import (
FiveLMMountHabitatDatasetArgs,
)
# Specify directory where an output directory will be created.
project_dir = os.path.expanduser("~/tbp/results/monty/projects")
# Specify a name for the model.
model_name = "dist_agent_5lm_2obj"
# Specify the objects to train on and 14 unique object poses.
object_names = ["mug", "banana"]
train_rotations = get_cube_face_and_corner_views_rotations()
# The config dictionary for the pretraining experiment.
dist_agent_5lm_2obj_train = dict(
# Specify monty experiment class and its args.
# The MontySupervisedObjectPretrainingExperiment class will provide the model
# with object and pose labels for supervised pretraining.
experiment_class=MontySupervisedObjectPretrainingExperiment,
experiment_args=ExperimentArgs(
do_eval=False,
n_train_epochs=len(train_rotations),
),
# Specify logging config.
logging_config=PretrainLoggingConfig(
output_dir=project_dir,
run_name=model_name,
),
# Specify the Monty model. The FiveLLMMontyConfig contains all of the sensor module
# configs, learning module configs, and connectivity matrices we need.
monty_config=FiveLMMontyConfig(
monty_args=MontyArgs(num_exploratory_steps=500),
motor_system_config=MotorSystemConfigNaiveScanSpiral(
motor_system_args=make_naive_scan_policy_config(step_size=5)
),
),
# Set up the environment and agent.
dataset_class=ED.EnvironmentDataset,
dataset_args=FiveLMMountHabitatDatasetArgs(),
# Set up the training dataloader.
train_dataloader_class=ED.InformedEnvironmentDataLoader,
train_dataloader_args=EnvironmentDataloaderPerObjectArgs(
object_names=object_names,
object_init_sampler=PredefinedObjectInitializer(rotations=train_rotations),
),
# Set up the evaluation dataloader. Unused, but required.
eval_dataloader_class=ED.InformedEnvironmentDataLoader, # just placeholder
eval_dataloader_args=get_env_dataloader_per_object_by_idx(start=0, stop=1),
)
experiments = MyExperiments(
# For each experiment name in MyExperiments, add its corresponding
# configuration here.
# e.g.: my_experiment=my_experiment_config
dist_agent_5lm_2obj_train=dist_agent_5lm_2obj_train,
)
CONFIGS = asdict(experiments)
After adding the config name to names.py, when I run it, I get:
(tbp.monty) benchmarks % python run.py -e dist_agent_5lm_2obj_train
MONTY_LOGS not set. Using default directory: ~/tbp/results/monty/
MONTY_MODELS not set. Using default directory: ~/tbp/results/monty/pretrained_models/
MONTY_DATA not set. Using default directory: /Users/me/tbp/data/
WANDB_DIR not set. Using default directory: ~/tbp/results/monty/
---------training---------
building graph for patch_0
building graph for patch_1
building graph for patch_2
building graph for patch_3
building graph for patch_4
building graph for patch_0
building graph for patch_1
building graph for patch_2
building graph for patch_3
building graph for patch_4
...
2 Likes
I appreciate the feedback. In my case, I get the following. Is there supposed to be a habitat_utils.py in /home/jsmith/tbp.monty/src/tbp/monty/simulators/habitat (or someplace else that I can put in my PYTHONPATH?
Hi @Jeff_Smith,
No, there is no habitat_utils.py in tbp/monty/simulators/habitat.
Do you have the latest code for tbp.monty? I recommend pulling the latest code as habitat_utils.py was removed in pull request #149 back on Jan 28.
Hi @Jeff_Smith,
Sorry, I am getting a bit lost about where we are in troubleshooting.
Could you please check out the latest version of Monty and copy and paste the command you ran and the error you are seeing?
Thanks for your help. I got the following after I had performed an update. It had looked like the import of habitat_sim.utils had a problem. I think it had a problem because it couldn’t find this. I can’t either.
python run.py -e dist_agent_5lm_2obj_trainFile “/home/jsmith/tbp.monty/benchmarks/run.py”, line 42, in
CONFIGS = load_configs(experiments)
^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/jsmith/tbp.monty/benchmarks/configs/load.py”, line 37, in load_configs
configs.update(select_config(experiment))
^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/jsmith/tbp.monty/benchmarks/configs/load.py”, line 103, in select_config
from benchmarks.configs.my_experiments import CONFIGS as MY_EXPERIMENTS
File “/home/jsmith/tbp.monty/benchmarks/configs/my_experiments.py”, line 75, in
from tbp.monty.simulators.habitat.configs import (
File “/home/jsmith/tbp.monty/src/tbp/monty/simulators/habitat/init.py”, line 11, in
from .actions import *
File “/home/jsmith/tbp.monty/src/tbp/monty/simulators/habitat/actions.py”, line 14, in
import habitat_sim.utils as hab_utils
ModuleNotFoundError: No module named ‘habitat_sim’
This tells me that the environment setup failed, and the HabitatSim module did not install correctly.
When you run conda list | grep habitat what output do you get?
I got to the bottom of the habitat issue. The good news is that I simply forgot to perform a “conda activate tbp.monty”. The bad news is I get a different error, namely the following. I get a KeyError: ‘patch_0.rgba’.
(tbp.monty) [jsmith@localhost benchmarks]$ python run.py -e dist_agent_5lm_2obj_train
---------training---------
Traceback (most recent call last):
File “run.py”, line 44, in
main(all_configs=CONFIGS, experiments=cmd_args.experiments)
File “/home/jsmith/tbp.monty/src/tbp/monty/frameworks/run.py”, line 115, in main
run(exp_config)
File “/home/jsmith/tbp.monty/src/tbp/monty/frameworks/run.py”, line 55, in run
exp.train()
File “/home/jsmith/tbp.monty/src/tbp/monty/frameworks/experiments/pretraining_experiments.py”, line 124, in train
self.run_epoch()
File “/home/jsmith/tbp.monty/src/tbp/monty/frameworks/experiments/monty_experiment.py”, line 494, in run_epoch
self.run_episode()
File “/home/jsmith/tbp.monty/src/tbp/monty/frameworks/experiments/pretraining_experiments.py”, line 77, in run_episode
self.model.step(observation)
File “/home/jsmith/tbp.monty/src/tbp/monty/frameworks/models/monty_base.py”, line 144, in step
self._exploratory_step(observation)
File “/home/jsmith/tbp.monty/src/tbp/monty/frameworks/models/abstract_monty_classes.py”, line 37, in _exploratory_step
self.aggregate_sensory_inputs(observation)
File “/home/jsmith/tbp.monty/src/tbp/monty/frameworks/models/monty_base.py”, line 152, in aggregate_sensory_inputs
sensor_module.update_state(self.get_agent_state())
File “/home/jsmith/tbp.monty/src/tbp/monty/frameworks/models/sensor_modules.py”, line 506, in update_state
sensor_position = state[“sensors”][self.sensor_module_id + “.rgba”][“position”]
KeyError: ‘patch_0.rgba’
Ok, that’s good news, we are back to the original error at the top of the post.
Since neither @HumbleTraveller nor I were able to reproduce the problem using our code, would you please copy and paste here the contents of your my_experiments.py file so that we can verify that we’re running the same code?
Sure. It is attached. I was trying to use this for all 5 tutorials. It worked for first 4 of 5.
(Attachment my_experiments.py is missing)
Sure. It is attached. I was trying to use this for all 5 tutorials. It worked for first 4 of 5. When I sent this with the .py extension the mailer returned my email. Here it is without .py. Feel free to add it back when you download it.
(Attachment my_experiments is missing)
Hi @Jeff_Smith, it doesn’t look like emailing a response to Discourse allows for including attachments. You may need to paste it into the forum thread: Multiple Learning Module Tutorial question - #13 by Jeff_Smith.
You are right. Here it is:
Copyright 2025 Thousand Brains Project
Hi, @Jeff_Smith, unfortunately your messages are not coming through to the forum.
Any clues in this code for what might be causing the problem
Thanks. I am not sure what to do with the sensor_module_id call bugged out error. I didn’t modify the source code outside of what is in the 5th tutorial.
Hey there Jeff,
No worries. We can look at this together. If you navigate into /home/jsmith/tbp.monty/benchmarks/configs/ there should be a file called my_experiments.py.
@tslominski and I will need to see the contents of this file in order to determine whats going wrong here. We won’t be able to help without it. Are you able to send us that file?
There was. I sent the contents of this file in a previous email. Would it be easier to share a screen.
