Even though we improved how we create configurations with Hydra, what the final configuration will look like is still not very clear.
One mitigation for this is that we print out the configuration at the beginning of each experiment.
However, I want to highlight another method we use. We take snapshots of fully resolved configurations and store them in tests/conf/snapshots. So, if you’re curious about what a full configuration for any of our experiments looks like, that is the directory to peruse.
These snapshots allow us to ensure that every configuration change we make is deliberate. We have a snapshot test in tests/conf/conf_test.py which resolves every configuration shipped with Monty and if that no longer matches the snapshots, the test fails. The failure only means that some configuration component changed. If this change is deliberate, we have a src/tbp/monty/conf/update_snapshots.py utility that will update every snapshot based on current code. This, in turn, will show up as a code difference in a pull request, where we can verify that the effect of our configuration changes on all of our experiments is as intended.