Were observations ever reported with `.rgba` and `.depth` sensor ID suffixes?

tslominski · February 23, 2026, 11:18pm

Hey @AgentRev, I’m moving our conversation here because I find it confusing as part of the pull request for the motor system experiment mode.

Originally from @AgentRev :

Please correct me if I’m wrong, but I don’t think observations were ever reported with .rgba and .depth sensor ID suffixes, were they? Those already seem dealt with here:

github.com/thousandbrainsproject/tbp.monty

src/tbp/monty/simulators/habitat/agents.py

5d4003de6


      
          def process_observations(self, agent_obs) -> AgentObservations:
              """Callback that processes raw Habitat agent observations.
          
              Converts raw observations into Monty-compatible ones.
          
              Args:
                  agent_obs: Raw habitat-sim observations from the agent
          
              Returns:
                  The processed observations grouped by sensor_id
              """
              # Habitat raw sensor observations are flat, where the observation key is
              # composed of the `sensor_id.sensor_type`. The default agent starts by
              # grouping habitat raw observations by sensor_id and sensor_type.
              obs_by_sensor: AgentObservations = defaultdict(dict)
              for sensor_key, data in agent_obs.items():
                  sensor_id, sensor_type = sensor_key.split(".")
                  obs_by_sensor[SensorID(sensor_id)][sensor_type] = data
          
              # Call each sensor to post-process the observation data

This file has been truncated. show original

A typical Observations dict is structured approximately like this:

{
    [AgentID] '0': [AgentObservations] {
        [SensorID] '0': [SensorObservation] {
            'rgba': [ndarray {64, 64, 4}],
            'depth': [ndarray {64, 64}],
            'semantic': [ndarray {64, 64}]
        }
    }
}

And we do need to keep those modality keys since they store different infos, unlike AgentState.sensors from PR 791, where the .rgba and .depth sensor IDs stored identical sensor states.

Or, am I looking at the wrong place?

It turns out you are correct @AgentRev, the observation flow seems to have already dealt with the issue.

However, I think there is one improvement remaining:

sensor_id, sensor_type = sensor_key.split(".")
obs_by_sensor[SensorID(sensor_id)][sensor_type] = data

This violates the constraint that identifiers are opaque. The agent already has a habitat_to_monty_sensor_id_map. What seems to be missing is either a field in the map for what modality the habitat sensor is serving, or alternatively, a new map like habitat_sensor_to_monty_modality_map. Either way, either sensor_type or modality needs to be stored when spec is being created in HabitatAgent.get_spec and then looked up in HabitatAgent.process_observations to correctly convert it to the appropriate modality.

In short, it should look something like:

        for sensor_key, data in agent_obs.items():
            sensor_id = self.habitat_to_monty_sensor_id_map[sensor_key]
            modality = self.something_something...[sensor_key] # look up previously stored modality here
            obs_by_sensor[SensorID(sensor_id)][modality] = data

AgentRev · February 24, 2026, 8:14am

I did suggest some time ago about filing work items as GitHub issues (and why not injections too while at it), i.e. treating it as a public Jira of sorts.

Anyways, yeah that was my follow-up question, I noticed Scott did that in his PR. Will do.