📢 Updates to writing benchmark and experiment configurations

Monty Code
1

To remove accidental coupling between Monty and HabitatSim, we had to rework the configuration of benchmarks and experiments.

Previously, you would create a new file in benchmarks/configs, set up the whole configuration in that file, and export a CONFIGS dictionary. Then, you would import that CONFIGS dictionary in benchmarks/configs/__init__.py.

This was problematic as we loaded all the configurations (and therefore all of their dependencies) when attempting to run any benchmark or experiment.

The recent update changes that by separating experiment names from experiment configurations. This way, we now selectively load a configuration depending on what experiment or benchmark you select to run.

The new setup asks that you declare your experiment names in the MyExperiments class in the benchmarks/configs/names.py file and your configurations separately in benchmarks/configs/my_experiments.py. You can also create your own names class and configurations file if you ensure it gets loaded in benchmarks/configs/load.py.

# benchmarks/configs/my_experiments.py

from dataclasses import asdict

from benchmarks.configs.names import MyExperiments

# Add your experiment configurations here
# e.g.: my_experiment_config = dict(...)


experiments = MyExperiments(
    # For each experiment name in MyExperiments, add its corresponding
    # configuration here.
    # e.g.: my_experiment=my_experiment_config
)
CONFIGS = asdict(experiments)

# benchmarks/configs/names.py

# ...

@dataclass
class MyExperiments:
    # Add your experiment names here
    pass


NAMES.extend(field.name for field in fields(MyExperiments))

For more details about the changes, see the pull request discussion. The tutorials have been updated. They are a good example of the new setup.

3 Likes