Error trying to create an Omniglot experiment

Monty Code
1

Hi!
I’m trying to create an experiment to use Omniglot dataset. I tried just to copy from the first experiment from tutorial, and change what appeared to be necessary. I got:


two_d_experiment = dict(
    experiment_class=MontyExperiment,
    logging_config=LoggingConfig(),
    experiment_args=ExperimentArgs(
        do_eval=False,
        max_train_steps=1,
        n_train_epochs=1,
    ),
    monty_config=SingleCameraMontyConfig(),
    # Data{set, loader} config
    dataset_class=OmniglotEnvironment,
    dataset_args=OmniglotDatasetArgs(),
    train_dataloader_class=OmniglotDataLoader,
    train_dataloader_args=OmniglotDataloaderArgs(),
    eval_dataloader_class=OmniglotDataLoader,
    eval_dataloader_args=OmniglotDataloaderArgs(),
)

(I’m sure that it is not so simple :slight_smile: anyway it was worth to try…
But I’m getting following errors:

run.py -e two_d_experiment 
MONTY_LOGS not set. Using default directory: ~/tbp/results/monty/
MONTY_MODELS not set. Using default directory: ~/tbp/results/monty/pretrained_models/
MONTY_DATA not set. Using default directory: /home/ematos/tbp/data/
WANDB_DIR not set. Using default directory: ~/tbp/results/monty/
Traceback (most recent call last):
  File "/home/ematos/tbp/src/tbp/monty/frameworks/run.py", line 48, in run
    exp.setup_experiment(config)
  File "/home/ematos/tbp/src/tbp/monty/frameworks/experiments/monty_experiment.py", line 71, in setup_experiment
    self.load_dataset_and_dataloaders(config)
  File "/home/ematos/tbp/src/tbp/monty/frameworks/experiments/monty_experiment.py", line 180, in load_dataset_and_dataloaders
    self.dataset = self.load_dataset(dataset_class, dataset_args)
  File "/home/ematos/tbp/src/tbp/monty/frameworks/experiments/monty_experiment.py", line 220, in load_dataset
    raise TypeError("dataset class must be EnvironmentDataset (for now)")
TypeError: dataset class must be EnvironmentDataset (for now)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ematos/tbp/benchmarks/run.py", line 44, in <module>
    main(all_configs=CONFIGS, experiments=cmd_args.experiments)
  File "/home/ematos/tbp/src/tbp/monty/frameworks/run.py", line 115, in main
    run(exp_config)
  File "/home/ematos/tbp/src/tbp/monty/frameworks/run.py", line 59, in run
    exp.evaluate()
  File "/home/ematos/tbp/src/tbp/monty/frameworks/experiments/monty_experiment.py", line 628, in __exit__
    self.close()
  File "/home/ematos/tbp/src/tbp/monty/frameworks/experiments/monty_experiment.py", line 596, in close
    if isinstance(self.dataset, EnvironmentDataset):
AttributeError: 'MontyExperiment' object has no attribute 'dataset'

Process finished with exit code 1

Because the message
TypeError: dataset class must be EnvironmentDataset (for now)

I understood that class OmniglotEnvironment is not the correct to be used…
Any help is appreciated :slight_smile:
Thanks!

2

Hi Ely,

sorry about the trouble with this. I hope the code still works since we haven’t run experiments with the Omniglot dataset in a while (we figured that we need to be able to model compositional objects before we can get good accuracy on this dataset).
For an example config that we used in the past, you could have a look at this one: monty_lab/experiments/configs/graph_experiments.py at f8fe7290eef7b198f01f8fd22c64a5585f495ee1 · thousandbrainsproject/monty_lab · GitHub
There we just use dataset_class=ED.EnvironmentDataset

The OmniglotEnvironment is specified in env_init_func: Callable = field(default=OmniglotEnvironment) in the OmniglotDatasetArgs class.

Hope this helps!
-Viviane

3

Thanks, @vclay :slight_smile:
I was able to run the experiment using that configuration and after make two small changes in the current code:

class HabitatDistantPatchSM(DetailedLoggingSM, NoiseMixin):
...
    def __init__(
        motor_only_step=False,
    ):
...
        self.motor_only_step = motor_only_step

and

class OmniglotEnvironment(EmbodiedEnvironment):
...
    def step(self, _action, amount = 1):

but I’m not sure if it works at the end, as I got no matching:

image
image1024×101 35.8 KB

(I don’t know if this is the expected result…).

About modeling compositional objects, is this a working in progress or we already have something to test with Omniglot dataset?

Thanks!

4

Hi Ely,

if you are using exactly the same config, then you should be able to recognize these digits I think. Are you loading pretrained models like here: monty_lab/experiments/configs/graph_experiments.py at f8fe7290eef7b198f01f8fd22c64a5585f495ee1 · thousandbrainsproject/monty_lab · GitHub
To generate these models you will have to run a pertaining experiment like this first: monty_lab/experiments/configs/more_pretraining_experiments.py at f8fe7290eef7b198f01f8fd22c64a5585f495ee1 · thousandbrainsproject/monty_lab · GitHub
Otherwise the learning module has nothing in its memory and can therefor also not recognize anything. Since all or your episodes have no_match with result = new_object0, it looks like you are in evaluation mode (no new information being added to memory) and you have nothing in memory.

Compositional objects are the next big milestone on our roadmap (see row 50). We already made a good amount of progress on this (figuring out how it conceptually works in the brain and how we want to implement it + adding Monty support for stacking LMs hierarchically) but there are still a few remaining items left for it to work well (listed in that spreadsheet).

Best wishes,
Viviane

5

Hi @ElyMatos
I just wanted to mention that I added two new tutorials to our documentation (New Tutorials on Using Monty in Custom Applications) One of them talks through the Omniglot example in detail and also provides code to follow along.
As I was writing it I actually noticed that the OmniglotEnvironment code was broken and added a fix. Maybe give it a try again now! Let me know if you run into any issues.

  • Viviane
2 Likes