Why is Monty showing poor rotation robustness in Omniglot tests despite exhaustive hypothesis search?

taegeon5846-lab · January 20, 2026, 9:46pm

Hello, I am an undergraduate researcher testing the Monty framework’s robustness to 2D in-plane rotations in the Omniglot environment. I understood that Monty’s core strength is its ability to recognize objects regardless of their pose through hypothesis testing. However, my results show a near-total drop in accuracy (labeled ‘confused’ - misclassified) for any rotated samples.

[Problem Description]

I am testing the Monty framework’s robustness to 2D in-plane rotations in the Omniglot environment. Despite configuring the model for an exhaustive pose search and providing explicit rotations, the model identifies 0° samples correctly but labels all rotated samples as “confused”.

[Key Configuration Changes]

To enable rotation robustness, I made the following specific changes to my omniglot_inference.yaml:

1. Enabling Exhaustive Pose Search

I removed the “upright only” constraint to allow Monty to test all possible orientations.

YAML

YAML

# FROM (Original)hypotheses_updater_args:
  initial_possible_poses: [[0, 0, 0]]# TO (Modified)hypotheses_updater_args:
  initial_possible_poses: null  # Should enable exhaustive search

2. Increasing Tolerance and Match Distance

I relaxed the matching constraints to account for spatial shifts and orientation changes caused by rotation.

YAML

YAML

# FROM (Original)max_match_distance: 5tolerances:
  patch:
    pose_vectors: ${np.array:[45, 45, 45]}# TO (Modified)max_match_distance: 20tolerances:
  patch:
    pose_vectors: ${np.array:[180, 180, 180]} # Maximum tolerance

3. Defining Custom Rotations for Evaluation

I modified the environment interface to test specific 45-degree increments.

YAML

YAML

# ADDED in Modified Configeval_env_interface_args:
  custom_rotations:
    - [0, 0, 0]   # Episode 1
    - [0, 0, 45]  # Episode 2
    - [0, 0, 90]  # Episode 3
    ... (up to 360 degrees)

4. Increasing Motor Steps

I gave the model more time to explore the rotated features.

YAML

YAML

# ADDED in Modified Configmotor_system_config:
  motor_system_args:
    num_steps: 200
    min_steps: 5

[The Result]

Even with these changes:

0° Episodes: Successfully recognized in ~4 steps.
Rotated Episodes (45°, 90°, etc.): Status is consistently confused.
The Issue: The most_likely_rotation in the output does not converge to the values I set in custom_rotations, suggesting the hypothesis testing isn’t finding the match in the 2D plane.

[Questions for Developers]

Coordinate System Sync: Does the EvidenceGraphLM automatically synchronize the sensor’s proprioceptive path with the physical rotation applied via custom_rotations? If the object is rotated but the sensor’s internal “world frame” is not, does this invalidate the graph matching?
Hypothesis Search Space: Is initial_possible_poses: null sufficient to trigger a 360-degree search in the 2D plane for Omniglot, or do I need to explicitly define a rotation grid in the HypothesisUpdater?
Feature Sensitivity: Given that I set pose_vectors tolerance to 180, why would the model still fail to find a match? Is there another feature (e.g., curvature) that is breaking the match due to its lack of rotation invariance?

Thank you for your time and assistance in reviewing this issue. I would greatly appreciate any insights or guidance you could provide to help resolve this behavior.

vclay · January 23, 2026, 1:25pm

Hi @taegeon5846-lab

I’m sorry to hear that you are having trouble getting the expected results. Could you share the full config that you are using? Are you starting with this tutorial? Using Monty in a Custom Application It’s a bit hard to diagnose the issue without that but here a couple of thoughts already:

We normally set initial_possible_poses: "informed" (or initial_possible_poses: uniform if we just want to test a discrete set). I don’t know what happens if you set this value to null in Hydra, but it might cause some unexpected behavior.
You shouldn’t need to adjust the tolerances to deal with rotation and location changes. Those are only important if the morphology of the object can vary or you get noisy observations.
I don’t think there is a custom_rotations parameter in the OmniglotEnvironmentInterface. Did you make any modifications to that class that implements showing rotated versions of the characters?
motor_system_args doesn’t have a num_steps or min_steps parameter. Those are set in the highest level of the config (for the experiment). For example have a look at how max_total_steps is set here: tbp.monty/conf/experiment/base_77obj_surf_agent.yaml at ebb88e8e9519803d146a7410c51555738f14159c · thousandbrainsproject/tbp.monty · GitHub However, I am not sure this change is necessary as an Omniglot character only has a certain number of points on it before you see the same places again and I think the default number of steps should already be above that.
If you also want to set the minimum number of steps (how many steps need to be taken before monty is allowed to converge/terminate an episode), you can do it like this: tbp.monty/conf/experiment/base_config_10distinctobj_surf_agent.yaml at ebb88e8e9519803d146a7410c51555738f14159c · thousandbrainsproject/tbp.monty · GitHub Monty’s configs can be complex and confusing. It might be useful to have a look at this video https://www.youtube.com/watch?v=B-Kd7Z3pXWk where @tslominski goes over an entire config written out in one file (instead of inheriting from other configs) to get an overview of what parts would be configured where. Also, this tutorial gives a bit more of a breakdown of the configs Running Your First Experiment

Let me know if that helps update the config to actually do what you want or if you have more questions.

Best wishes,

Viviane

taegeon5846-lab · January 24, 2026, 4:59pm

Hi Viviane,

Thank you for your guidance.

Following your advice, I corrected the max_total_steps placement and set initial_possible_poses to "uniform". I also reverted the tolerances to their default values.

[Current Results & Observations] The model is now exploring the full hypothesis space, but the recognition performance varies significantly depending on the angle.

Based on my test runs, here is the breakdown:

Consistent Success: 0°, 90°, 180°, 360°
Consistent Failure (“Confused”): 45°, 225°, 315°
Strange Cases:
- 270°: This consistently failed, which is surprising because 90° and 180° worked perfectly. Since 270° is an orthogonal rotation, it shouldn’t have severe aliasing artifacts.
- 135°: This was inconsistent (succeeded in one episode, failed in another).

[Hypothesis] For the oblique angles (45°, 225°, etc.), I suspect the failure is due to image artifacts. Since I used scipy.ndimage.rotate with mode='nearest' to preserve binary values, the resulting images have jagged edges (aliasing), which might be distorting the curvature features.

However, I am unsure why 270° is failing, as the image quality should be identical to 90°.

[My Code Implementation] To clarify my setup, here are the modifications I made across the three files:

1. embodied_data.py (Interface Level) I modified OmniglotEnvironmentInterface to iterate through the custom_rotations list and pass the angle.

Python

# Modified 'change_object_by_idx' method
def change_object_by_idx(self, idx):
    # ... (standard loading code) ...

    # 1. Get the rotation vector from iterator
    rotation_vec = [0, 0, 0]
    if self.rotation_iterator:
        try:
            rotation_vec = next(self.rotation_iterator)
        except StopIteration:
            pass
    
    # 2. Pass rotation angle to the environment
    if hasattr(self.env, "set_rotation"):
        self.env.set_rotation(rotation_vec[2])

    # 3. Store ground truth for evaluation
    self.primary_target = {
        "object": self.object_names[idx],
        "euler_rotation": np.array(rotation_vec),
        # ...
    }

2. two_d_data.py (Environment Level) I modified OmniglotEnvironment to apply rotation to the image and stroke coordinates.

Python

# Modified 'load_new_character_data' method
def load_new_character_data(self):
    # ... (load image and path) ...

    if current_angle != 0:
        # 1. Rotate the image (using nearest to keep it binary)
        current_image = scipy.ndimage.rotate(
            current_image, 
            current_angle, 
            reshape=False, 
            mode='nearest'
        )

        # 2. Rotate the stroke coordinates (locations)
        theta = np.radians(current_angle)
        c, s = np.cos(theta), np.sin(theta)
        
        # ... (coordinate transformation logic) ...
        
        locations[:, 0] = new_y + cy
        locations[:, 1] = new_x + cx

    return current_image, locations

3. omniglot_inference.yaml (Experiment Config) I defined the rotation list to test various angles.

YAML

defaults:
  - /experiment/config/defaults@config
  # ... (other defaults) ...
  - /experiment/config/environment/omniglot@config.env_interface_config
  - /experiment/config/environment_interface/omniglot@config.eval_env_interface_args

_target_: tbp.monty.frameworks.experiments.object_recognition_experiments.MontyObjectRecognitionExperiment

config:
  # FIXED: Step count moved inside config
  max_total_steps: 200
  
  monty_config:
    learning_module_configs:
      learning_module_0:
        learning_module_args:
          max_match_distance: 5
          tolerances:
            patch:
              principal_curvatures_log: ${np.ones:2}
              pose_vectors: ${np.array:[45, 45, 45]}
          
          # FIXED: Set to uniform to enable full search
          hypotheses_updater_args:
            initial_possible_poses: "uniform"

  eval_env_interface_args:
    custom_rotations:
      - [0, 0, 0]    # Success
      - [0, 0, 45]   # Fails
      - [0, 0, 90]   # Success
      - [0, 0, 135]  # Inconsistent
      - [0, 0, 270]  # Fails (Unexpected)
      # ...

I have attached the full Python files, the CSV result, and sample images showing the 45° distortion.

Could you provide any insights on why 270° might be failing while 90° works, and how to better handle the aliasing at oblique angles?

Since the system allows only PDF/Image attachments, I have converted the Python source files to PDF format and attached them.

Best regards, Taegeon

45 deg

90 deg

135 deg

csv file

omniglot_inference.pdf (33.7 KB)
two_d_data.pdf (41.2 KB)
embodied_data.pdf (4.7 KB)

Rich_Morin · January 24, 2026, 11:50pm

@taegeon5846-lab I mostly want to say welcome aboard and commend you on the detailed and organized bug report. I’ll be interested to see how things work out…

FWIW, as an interested observer, I’d like to see more images to get a visual idea of the distortions. For example, a few representative patches in all orientations. Also, have you cross-checked the rotations, in case they are producing artifacts?

vclay · January 29, 2026, 1:09pm

Hi @taegeon5846-lab

thanks for the updates and extra info. Am I understanding it right that the artifacts you are seeing are after you rotate the images in the environment interface? Are the images you show the stimuli that Monty is given? It seems this is an issue with the environment, unrelated to Monty.

Have you double-checked that when you rotate the locations, their new values correctly index the rotated image? If you look at get_image_patch, the locations are used to extract the patch from the image that Monty sees at each step. Did you double-check that the locations are integer values that correctly map to the image indices when you rotate?

A quick debugging step could be to turn on show_sensor_output in the experiment config and see Monty’s inputs and hypotheses. Maybe something obviously wrong will pop out there. Or add some plotting code to your environment interface function to visualize the extracted patches and their locations on the image.

Minor side note: The “uniform” option is the most inefficient one. If you already know that your rotations are only along one axis, you could provide a list of rotations to test along that axis (like [[0,0,0],[0,0,45],…). In most our experiments we set this to “informed” which means Monty will infer the possible orientations based on the first observations. That way, any rotation could be inferred (not just a discrete list) and you only test ~2 rotations. But this if more of a note on efficiency, it shouldn’t be the cause for your problems.

Best wishes,

Viviane

taegeon5846-lab · February 11, 2026, 6:53pm

Hi Viviane,

Thank you for the guidance. Following your advice, I verified the visual-proprioceptive alignment. As shown in the attached debug_alignment.png, the transformed motor coordinates (red dots) now perfectly overlay the rotated visual strokes (black pixels). This confirms that the environment interface code is correct.

I then ran the full rotation experiments (0° to 360°, step 15°).

1. Observations from the Results (See attached eval_stats.csv ) The results show a strong sensitivity to rotation artifacts, varying significantly by angle and character complexity.

General Trend: The model performs relatively better at major angles (0°, 45°, 135°, 180°…) where the rasterization aliasing is consistent or minimal.
The Issue:
1. Performance drops sharply at fine increments (15°, 30°, 75°…) where the strokes become jagged due to aliasing.
2. Even at orthogonal angles like 90° or 270°, some complex characters (e.g., in the Korean dataset) fail to match, suggesting that even slight pixel rendering changes disrupt the feature extraction.

[My Hypothesis] Since Omniglot strokes are extremely thin, any rotation (even 90°) creates slight variations in stroke thickness and connectivity due to rasterization. It seems Monty’s feature extraction (curvature or pose vectors) is highly sensitive to these pixel artifacts, leading to matching failures even when the alignment is perfect.

[Question] Given that the alignment is fixed, what would be the best approach to handle these rotation artifacts? I suspect some parameter tuning might be necessary to make the model more robust to these visual variations. Could you please review my code and results and provide your recommendations?

I have attached my modified two_d_data.py and the full results for your review.

Best regards, Taegeon Park

two_d_data.py

def load_new_character_data(self):img_char_dir = (self.data_path/ “images_background”/ self.current_alphabet/ f"character{self.character_id:02d}")stroke_char_dir = (self.data_path/ “strokes_background”/ self.current_alphabet/ f"character{self.character_id:02d}")first_img_char_child = next(img_char_dir.iterdir()).namechar_img_names = first_img_char_child.split(““)[0]char_stem = f”{char_img_names}{self.character_version:02d}”img_file = img_char_dir / f"{char_stem}.png"   

    # 1. Load image and motor path
    current_image = load_img(img_file)
    move_path = load_motor(stroke_char_dir / f"{char_stem}.txt")
    logger.info(f"Finished loading new image from {img_file}")
    
    # 2. Convert motor path to pixel coordinates
    locations = self.motor_to_locations(move_path)

    # -------------------------------------------------------------
    # [MODIFIED] Custom Rotation Logic with Padding & Alignment Fix
    # -------------------------------------------------------------
    import scipy.ndimage
    import numpy as np
    import matplotlib.pyplot as plt
    import os, time

    # Check current rotation angle
    current_angle = 0
    if hasattr(self, 'current_rotation_angle'):
         current_angle = self.current_rotation_angle

    if current_angle != 0:
        # (1) Apply Padding to prevent clipping during rotation
        pad_size = 40
        current_image = np.pad(current_image, pad_size, mode='constant', constant_values=0)
        locations[:, 0] += pad_size
        locations[:, 1] += pad_size

        # (2) Rotate Image using Bilinear Interpolation (order=1)
        # Using order=1 reduces aliasing artifacts compared to nearest neighbor (order=0)
        current_image = scipy.ndimage.rotate(
            current_image, 
            current_angle, 
            reshape=False, 
            order=1,        
            mode='constant', 
            cval=0
        )

        # (3) Convert back to Boolean
        # Bilinear interpolation produces float values. We convert back to boolean 
        # to maintain compatibility with the sensor module (specifically for the ~patch operation).
        current_image = (current_image > 0.5)

        # (4) Rotate Coordinates (2D Rotation Matrix)
        # Manually rotate the motor coordinates to align with the rotated image.
        h, w = current_image.shape
        cy, cx = (h - 1) / 2.0, (w - 1) / 2.0
        
        # Note: We use negative angle here to match the image rotation direction
        theta = np.radians(-current_angle)
        c, s = np.cos(theta), np.sin(theta)
        
        # locations[:, 0] is X (width), locations[:, 1] is Y (height)
        old_x = locations[:, 0] - cx
        old_y = locations[:, 1] - cy
        
        new_x = old_x * c - old_y * s
        new_y = old_x * s + old_y * c
        
        locations[:, 0] = new_x + cx
        locations[:, 1] = new_y + cy

        # (5) [DEBUG] Save Alignment Verification Image
        # This saves an image showing the overlay of red dots (motor) on black pixels (visual).
        save_dir = os.path.expanduser("~/tbp/debug_images")
        os.makedirs(save_dir, exist_ok=True)
        timestamp = int(time.time() * 1000)
        debug_filename = f"debug_char{self.character_id}_{int(current_angle)}deg_{timestamp}.png"
        full_path = os.path.join(save_dir, debug_filename)
        
        plt.figure(figsize=(5, 5))
        plt.title(f"Alignment Check: {current_angle} deg")
        # Convert boolean image to float for visualization
        plt.imshow(current_image.astype(float), cmap='gray', origin='upper')
        plt.scatter(locations[:, 0], locations[:, 1], s=5, c='red', marker='x', label='Motor Coords')
        plt.legend()
        plt.savefig(full_path)
        plt.close()
        print(f"\n[DEBUG] Alignment check image saved at: {full_path}\n")

    # -------------------------------------------------------------
    # Filter out coordinates that are outside the image patch area
    max_h, max_w = current_image.shape
    locs_in_range = np.where(
        (locations[:, 1] > self.patch_size)
        & (locations[:, 0] > self.patch_size)
        & (locations[:, 1] < max_h - self.patch_size)
        & (locations[:, 0] < max_w - self.patch_size)
    )
    
    if isinstance(locs_in_range, tuple):
         locations = locations[locs_in_range[0]]
    else:
         locations = locations[locs_in_range]
         
    # Safety check to prevent ZeroDivisionError if all points are filtered out
    if len(locations) < 2:
        print(f"Warning: Too few locations remaining after rotation. Padding might be insufficient.")
        locations = np.vstack([locations, locations]) if len(locations) > 0 else np.array([[max_h/2, max_w/2], [max_h/2, max_w/2]])

    self.max_steps = len(locations) - 1
    return current_image, locations

Description of debug_alignment.png:

Black Pixels (Background/Stroke): This represents the visual input (current_image) after applying scipy.ndimage.rotate with bilinear interpolation and boolean conversion.
Red Crosses (Overlay): These represent the motor coordinates (locations) after applying the 2D rotation matrix.
Conclusion: As shown in the image, the red motor path perfectly traces the black visual strokes. This confirms that the coordinate alignment between the agent’s internal model and the visual observation is correct.

eval_stats_korean_45deg.csv

	primary_performance	stepwise_performance	num_steps	rotation_error	result	most_likely_object	primary_target_object	stepwise_target_object	highest_evidence	time	symmetry_evidence	monty_steps	monty_matching_steps	individual_ts_performance	individual_ts_reached_at_step	primary_target_position	primary_target_rotation_euler	most_likely_rotation	num_possible_matches	detected_location	detected_rotation	detected_scale	location_rel_body	detected_path	individual_ts_rotation_error	mean_objects_per_graph	mean_graphs_per_object	TFNP	possible_match_sources	most_likely_location	episode_avg_prediction_error	goal_states_attempted	goal_state_achieved	primary_target_rotation_quat	primary_target_scale	lm_id
LM_0	correct	confused	4	0	Korean_1	Korean_1	Korean_1	no_label	5.499672	25.00563	0	4	4	correct	4	[0 0 0]	[0 0 0]	[0. 0. 0.]	1	[4.66637330e+01 2.22222302e-02 2.40594375e+01]	[0. 0. 0.]	1	[4.66637312e+01 2.22222300e-02 2.40594381e+01]	[4.66637330e+01 2.22222302e-02 2.40594375e+01]	0	1	1	target_in_possible_matches_(TP)	Korean_1	[4.66637330e+01 2.22222302e-02 2.40594375e+01]	0.166703	0	0	[0. 0. 0. 1.]	1	LM_0
LM_0	correct	confused	14	45	Korean_2	Korean_2	Korean_2	no_label	8.037945	27.08372	5	14	14	correct	14	[0 0 0]	[ 0 0 45]	[ 0. 45. 0.]	1	[7.86637965e+01 2.22452911e-02 2.70596577e+01]	[ 0. 45. 0.]	1	[9.16693385e+01 2.22222222e-02 5.69987600e+01]	[7.86637965e+01 2.22452911e-02 2.70596577e+01]	45	1	1	target_in_possible_matches_(TP)	Korean_2	[7.86637965e+01 2.22452911e-02 2.70596577e+01]	0.665205	0	0	[0. 0. 0. 1.]	1	LM_0
LM_0	confused	confused	11		Korean_17	Korean_17	Korean_3	no_label	7.613115	24.27021	1	11	11	confused	11	[0 0 0]	[ 0 0 90]	[180. 0. 180.]	1	[2.16637866e+01 2.22222225e-02 4.60599639e+01]	[180. 0. 180.]	1	[1.10881660e+02 2.22222222e-02 9.45584910e+01]	[2.16637866e+01 2.22222225e-02 4.60599639e+01]	1	1	target_not_matched_(FN)	Korean_17	[2.16637866e+01 2.22222225e-02 4.60599639e+01]	0.684056	0	0	[0. 0. 0. 1.]	1	LM_0
LM_0	correct	confused	18	135	Korean_4	Korean_4	Korean_4	no_label	11.16486	28.60793	5	18	18	correct	18	[0 0 0]	[ 0 0 135]	[180. 45. 180.]	1	[6.96637650e+01 2.22546530e-02 3.80597306e+01]	[180. 45. 180.]	1	[7.09631179e+01 2.22222222e-02 9.11386702e+01]	[6.96637650e+01 2.22546530e-02 3.80597306e+01]	135	1	1	target_in_possible_matches_(TP)	Korean_4	[6.96637650e+01 2.22546530e-02 3.80597306e+01]	0.592205	0	0	[0. 0. 0. 1.]	1	LM_0
LM_0	correct	confused	17	180	Korean_5	Korean_5	Korean_5	no_label	15.70878	29.66879	5	17	17	correct	17	[0 0 0]	[ 0 0 180]	[180. 0. 180.]	1	[5.16637344e+01 2.22253501e-02 4.90594673e+01]	[180. 0. 180.]	1	[9.53807132e+01 2.22222222e-02 9.63405620e+01]	[5.16637344e+01 2.22253501e-02 4.90594673e+01]	180	1	1	target_in_possible_matches_(TP)	Korean_5	[5.16637344e+01 2.22253501e-02 4.90594673e+01]	0.564025	0	0	[0. 0. 0. 1.]	1	LM_0
LM_0	correct	confused	20	135	Korean_6	Korean_6	Korean_6	no_label	9.66398	30.51775	5	20	20	correct	20	[0 0 0]	[ 0 0 225]	[180. -45. 180.]	1	[6.86637611e+01 2.22515331e-02 6.20597001e+01]	[180. 315. 180.]	1	[7.46974365e+01 2.22222222e-02 9.55814348e+01]	[6.86637611e+01 2.22515331e-02 6.20597001e+01]	135	1	1	target_in_possible_matches_(TP)	Korean_6	[6.86637611e+01 2.22515331e-02 6.20597001e+01]	0.744474	0	0	[0. 0. 0. 1.]	1	LM_0
LM_0	confused_mlh	confused_mlh	100		[‘Korean_11’, ‘Korean_13’, ‘Korean_18’]	Korean_11	Korean_7	no_label	64.97986	63.13752	3	100	100	time_out		[0 0 0]	[ 0 0 270]	[0. 0. 0.]	3	[None, None, None]	[None, None, None]	[8.81627842e+01 2.22222222e-02 8.38415090e+01]	1	1	target_not_matched_(FN)	Korean_11-Korean_13-Korean_18	[3.76637318e+01 2.22222225e-02 4.60594381e+01]	0.507509	0	0	[0. 0. 0. 1.]	1	LM_0
LM_0	correct	confused	25	45	Korean_8	Korean_8	Korean_8	no_label	11.86194	35.69491	5	25	25	correct	25	[0 0 0]	[ 0 0 315]	[ 0. -45. 0.]	1	[4.06637611e+01 2.22515331e-02 3.90597000e+01]	[ 0. 315. 0.]	1	[9.25752299e+01 2.22222300e-02 7.41694468e+01]	[4.06637611e+01 2.22515331e-02 3.90597000e+01]	45	1	1	target_in_possible_matches_(TP)	Korean_8	[4.06637611e+01 2.22515331e-02 3.90597000e+01]	0.606457	0	0	[0. 0. 0. 1.]	1	LM_0
LM_0	correct	confused	10	0	Korean_9	Korean_9	Korean_9	no_label	14.49922	31.40053	5	10	10	correct	10	[0 0 0]	[ 0 0 360]	[0. 0. 0.]	1	[5.26637604e+01 2.22515408e-02 6.30597032e+01]	[0. 0. 0.]	1	[9.26637605e+01 2.22515406e-02 1.03059702e+02]	[5.26637604e+01 2.22515408e-02 6.30597032e+01]	0	1	1	target_in_possible_matches_(TP)	Korean_9	[5.26637604e+01 2.22515408e-02 6.30597032e+01]	0.281845	0	0	[0. 0. 0. 1.]	1	LM_0

eval_stats_korean_15deg.csv

	primary_performance	stepwise_performance	num_steps	rotation_error	result	most_likely_object	primary_target_object	stepwise_target_object	highest_evidence	time	symmetry_evidence	monty_steps	monty_matching_steps	individual_ts_performance	individual_ts_reached_at_step	primary_target_position	primary_target_rotation_euler	most_likely_rotation	num_possible_matches	detected_location	detected_rotation	detected_scale	location_rel_body	detected_path	individual_ts_rotation_error	mean_objects_per_graph	mean_graphs_per_object	TFNP	possible_match_sources	most_likely_location	episode_avg_prediction_error	goal_states_attempted	goal_state_achieved	primary_target_rotation_quat	primary_target_scale	lm_id
LM_0	correct	confused	4	0	Korean_1	Korean_1	Korean_1	no_label	5.499672	24.82332	0	4	4	correct	4	[0 0 0]	[0 0 0]	[0. 0. 0.]	1	[4.66637330e+01 2.22222302e-02 2.40594375e+01]	[0. 0. 0.]	1	[4.66637312e+01 2.22222300e-02 2.40594381e+01]	[4.66637330e+01 2.22222302e-02 2.40594375e+01]	0	1	1	target_in_possible_matches_(TP)	Korean_1	[4.66637330e+01 2.22222302e-02 2.40594375e+01]	0.166703	0	0	[0. 0. 0. 1.]	1	LM_0
LM_0	confused	confused	19		Korean_16	Korean_16	Korean_2	no_label	3.90972	29.28185	5	19	19	confused	19	[0 0 0]	[ 0 0 15]	[-7.0167093e-15 -4.5000000e+01 -1.8000000e+02]	1	[6.04771832e+01 2.22253666e-02 4.90760770e+01]	[ 0. 315. 180.]	1	[9.36915364e+01 2.22222222e-02 8.43602549e+01]	[6.04771832e+01 2.22253666e-02 4.90760770e+01]	1	1	target_not_matched_(FN)	Korean_16	[6.04771832e+01 2.22253666e-02 4.90760770e+01]	0.804447	0	0	[0. 0. 0. 1.]	1	LM_0
LM_0	confused	confused	9		Korean_16	Korean_16	Korean_3	no_label	5.664952	24.81543	0	9	9	confused	9	[0 0 0]	[ 0 0 30]	[-7.0167093e-15 -4.5000000e+01 -1.8000000e+02]	1	[7.45288108e+01 2.22222305e-02 5.91836407e+01]	[ 0. 315. 180.]	1	[9.14094281e+01 2.22222300e-02 9.58555915e+01]	[7.45288108e+01 2.22222305e-02 5.91836407e+01]	1	1	target_not_matched_(FN)	Korean_16	[7.45288108e+01 2.22222305e-02 5.91836407e+01]	0.810155	0	0	[0. 0. 0. 1.]	1	LM_0
LM_0	correct	confused	9	45	Korean_4	Korean_4	Korean_4	no_label	5.035258	25.40772	1	9	9	correct	9	[0 0 0]	[ 0 0 45]	[ 0. 45. 0.]	1	[7.46637650e+01 2.22546607e-02 2.30597305e+01]	[ 0. 45. 0.]	1	[8.60124905e+01 2.22284719e-02 5.69988163e+01]	[7.46637650e+01 2.22546607e-02 2.30597305e+01]	45	1	1	target_in_possible_matches_(TP)	Korean_4	[7.46637650e+01 2.22546607e-02 2.30597305e+01]	0.877367	0	0	[0. 0. 0. 1.]	1	LM_0
LM_0	correct	confused	8	180	Korean_5	Korean_5	Korean_5	no_label	5.772624	24.12442	0	8	8	correct	8	[0 0 0]	[ 0 0 60]	[ 1.80000000e+02 1.27222187e-14 -7.01670930e-15]	1	[3.62811026e+01 2.22839801e-02 4.98087056e+01]	[180. 0. 0.]	1	[6.41424081e+01 2.22222222e-02 7.42081305e+01]	[3.62811026e+01 2.22839801e-02 4.98087056e+01]	180	1	1	target_in_possible_matches_(TP)	Korean_5	[3.62811026e+01 2.22839801e-02 4.98087056e+01]	0.631375	0	0	[0. 0. 0. 1.]	1	LM_0
LM_0	confused	confused	6		Korean_16	Korean_16	Korean_6	no_label	5.215668	28.39568	0	6	6	confused	6	[0 0 0]	[ 0 0 75]	[ 0. 0. -180.]	1	[5.66707983e+01 2.22484193e-02 3.11343839e+01]	[ 0. 0. 180.]	1	[5.93311247e+01 2.22222222e-02 9.00232411e+01]	[5.66707983e+01 2.22484193e-02 3.11343839e+01]	1	1	target_not_matched_(FN)	Korean_16	[5.66707983e+01 2.22484193e-02 3.11343839e+01]	0.787526	0	0	[0. 0. 0. 1.]	1	LM_0
LM_0	confused	confused	17		Korean_18	Korean_18	Korean_7	no_label	14.24606	28.80998	1	17	17	confused	17	[0 0 0]	[ 0 0 90]	[0. 0. 0.]	1	[3.46637611e+01 2.22515408e-02 4.60597000e+01]	[0. 0. 0.]	1	[7.18816895e+01 2.22515406e-02 1.16558755e+02]	[3.46637611e+01 2.22515408e-02 4.60597000e+01]	1	1	target_not_matched_(FN)	Korean_18	[3.46637611e+01 2.22515408e-02 4.60597000e+01]	0.566108	0	0	[0. 0. 0. 1.]	1	LM_0
LM_0	confused	confused	14		Korean_11	Korean_11	Korean_8	no_label	6.354663	27.07049	5	14	14	confused	14	[0 0 0]	[ 0 0 105]	[ 1.80000000e+02 1.27222187e-14 -7.01670930e-15]	1	[7.92213092e+01 2.22546695e-02 5.83992311e+01]	[180. 0. 0.]	1	[1.01379673e+02 2.22222222e-02 8.90285396e+01]	[7.92213092e+01 2.22546695e-02 5.83992311e+01]	1	1	target_not_matched_(FN)	Korean_11	[7.92213092e+01 2.22546695e-02 5.83992311e+01]	0.704665	0	0	[0. 0. 0. 1.]	1	LM_0
LM_0	confused	confused	10		Korean_10	Korean_10	Korean_9	no_label	4.30316	24.83264	0	10	10	confused	10	[0 0 0]	[ 0 0 120]	[-1.80000000e+02 -4.50000000e+01 -9.92312545e-15]	1	[4.44024541e+01 3.28171809e-02 6.00897350e+01]	[180. 315. 0.]	1	[1.01106017e+02 2.22222300e-02 8.62147180e+01]	[4.44024541e+01 3.28171809e-02 6.00897350e+01]	1	1	target_not_matched_(FN)	Korean_10	[4.44024541e+01 3.28171809e-02 6.00897350e+01]	0.857825	0	0	[0. 0. 0. 1.]	1	LM_0
LM_0	correct	confused	23	135	Korean_10	Korean_10	Korean_10	no_label	14.12845	30.8657	5	23	23	correct	23	[0 0 0]	[ 0 0 135]	[180. 45. 180.]	1	[7.86637611e+01 2.22515408e-02 3.10597019e+01]	[180. 45. 180.]	1	[5.96494094e+01 2.22222300e-02 8.97244567e+01]	[7.86637611e+01 2.22515408e-02 3.10597019e+01]	135	1	1	target_in_possible_matches_(TP)	Korean_10	[7.86637611e+01 2.22515408e-02 3.10597019e+01]	0.589395	0	0	[0. 0. 0. 1.]	1	LM_0
LM_0	confused	confused	16		Korean_12	Korean_12	Korean_11	no_label	6.772426	26.73733	5	16	16	confused	16	[0 0 0]	[ 0 0 150]	[180. 45. 180.]	1	[9.33911005e+01 2.22222225e-02 8.05631333e+01]	[180. 45. 180.]	1	[9.92002257e+01 2.22222222e-02 6.64843153e+01]	[9.33911005e+01 2.22222225e-02 8.05631333e+01]	1	1	target_not_matched_(FN)	Korean_12	[9.33911005e+01 2.22222225e-02 8.05631333e+01]	0.713081	0	0	[0. 0. 0. 1.]	1	LM_0
LM_0	correct	confused	22	180	Korean_12	Korean_12	Korean_12	no_label	11.24528	29.35809	5	22	22	correct	22	[0 0 0]	[ 0 0 165]	[180. 0. 180.]	1	[7.98822881e+01 2.22515331e-02 7.03573664e+01]	[180. 0. 180.]	1	[6.15611921e+01 2.22222222e-02 7.59013026e+01]	[7.98822881e+01 2.22515331e-02 7.03573664e+01]	180	1	1	target_in_possible_matches_(TP)	Korean_12	[7.98822881e+01 2.22515331e-02 7.03573664e+01]	0.696512	0	0	[0. 0. 0. 1.]	1	LM_0
LM_0	correct	confused	18	180	Korean_13	Korean_13	Korean_13	no_label	17.45999	28.22116	5	18	18	correct	18	[0 0 0]	[ 0 0 180]	[180. 0. 180.]	1	[5.66637306e+01 2.22222305e-02 3.10594367e+01]	[180. 0. 180.]	1	[9.13807132e+01 2.22222300e-02 1.15340562e+02]	[5.66637306e+01 2.22222305e-02 3.10594367e+01]	180	1	1	target_in_possible_matches_(TP)	Korean_13	[5.66637306e+01 2.22222305e-02 3.10594367e+01]	0.536428	0	0	[0. 0. 0. 1.]	1	LM_0
LM_0	confused	confused	4		Korean_10	Korean_10	Korean_14	no_label	3.46565	29.0661	0	4	4	confused	4	[0 0 0]	[ 0 0 195]	[0. 0. 0.]	1	[3.85909860e+01 2.22284719e-02 6.31026688e+01]	[0. 0. 0.]	1	[7.27953569e+01 2.22222222e-02 1.23379248e+02]	[3.85909860e+01 2.22284719e-02 6.31026688e+01]	1	1	target_not_matched_(FN)	Korean_10	[3.85909860e+01 2.22284719e-02 6.31026688e+01]	0.608064	0	0	[0. 0. 0. 1.]	1	LM_0
LM_0	confused	confused	4		Korean_17	Korean_17	Korean_15	no_label	4.212246	25.01292	0	4	4	confused	4	[0 0 0]	[ 0 0 210]	[0. 0. 0.]	1	[6.26111984e+01 2.22515408e-02 4.30597000e+01]	[0. 0. 0.]	1	[1.07001042e+02 2.22222300e-02 1.11427095e+02]	[6.26111984e+01 2.22515408e-02 4.30597000e+01]	1	1	target_not_matched_(FN)	Korean_17	[6.26111984e+01 2.22515408e-02 4.30597000e+01]	0.585244	0	0	[0. 0. 0. 1.]	1	LM_0
LM_0	correct	confused	17	135	Korean_16	Korean_16	Korean_16	no_label	7.686012	30.08374	5	17	17	correct	17	[0 0 0]	[ 0 0 225]	[180. -45. 180.]	1	[6.96637305e+01 2.22222073e-02 2.40594369e+01]	[180. 315. 180.]	1	[1.00860387e+02 2.22222222e-02 1.24572813e+02]	[6.96637305e+01 2.22222073e-02 2.40594369e+01]	135	1	1	target_in_possible_matches_(TP)	Korean_16	[6.96637305e+01 2.22222073e-02 2.40594369e+01]	0.700176	0	0	[0. 0. 0. 1.]	1	LM_0
LM_0	no_match	no_match	12		new_object25	Korean_11	Korean_17	no_label	-0.65305	31.56329	2	12	12	no_match	11	[0 0 0]	[ 0 0 240]	[0. 0. 0.]	0	[None, None, None]	[None, None, None]	[1.14696265e+02 2.22222222e-02 1.37208530e+02]	1	1	target_not_matched_(FN)	[8.09387222e+01 2.22222137e-02 9.31190377e+01]	0.931163	0	0	[0. 0. 0. 1.]	1	LM_0
LM_0	confused	confused	8		Korean_20	Korean_20	Korean_18	no_label	2.955348	30.81592	0	8	8	confused	8	[0 0 0]	[ 0 0 255]	[ 1.80000000e+02 1.27222187e-14 -7.01670930e-15]	1	[7.20540359e+01 2.22316075e-02 5.26411848e+01]	[180. 0. 0.]	1	[1.15262112e+02 2.22222222e-02 7.94210014e+01]	[7.20540359e+01 2.22316075e-02 5.26411848e+01]	1	1	target_not_matched_(FN)	Korean_20	[7.20540359e+01 2.22316075e-02 5.26411848e+01]	1	0	0	[0. 0. 0. 1.]	1	LM_0
LM_0	confused_mlh	confused_mlh	100		[‘Korean_1’, ‘Korean_3’, ‘Korean_10’, ‘Korean_11’, ‘Korean_15’, ‘Korean_17’, ‘Korean_21’]	Korean_11	Korean_19	no_label	119.9387	65.53784	0	100	100	time_out		[0 0 0]	[ 0 0 270]	[0. 0. 0.]	7	[None, None, None]	[None, None, None]	[1.00162784e+02 2.22222222e-02 7.88415090e+01]	1	1	target_not_matched_(FN)	Korean_1-Korean_10-Korean_11-Korean_15-Korean_17-Korean_21-Korean_3	[7.76637356e+01 2.22253424e-02 3.70594667e+01]	0.272198	0	0	[0. 0. 0. 1.]	1	LM_0
LM_0	confused	confused	13		Korean_3	Korean_3	Korean_20	no_label	6.95925	30.79676	3	13	13	confused	13	[0 0 0]	[ 0 0 285]	[0. 0. 0.]	1	[7.35974415e+01 2.22284798e-02 5.63007251e+01]	[0. 0. 0.]	1	[1.12576289e+02 2.22253509e-02 1.14273903e+02]	[7.35974415e+01 2.22284798e-02 5.63007251e+01]	1	1	target_not_matched_(FN)	Korean_3	[7.35974415e+01 2.22284798e-02 5.63007251e+01]	0.640823	0	0	[0. 0. 0. 1.]	1	LM_0
LM_0	confused	confused	13		Korean_12	Korean_12	Korean_21	no_label	11.67968	30.74096	5	13	13	confused	13	[0 0 0]	[ 0 0 300]	[0. 0. 0.]	1	[8.69317065e+01 2.22515331e-02 4.57199540e+01]	[0. 0. 0.]	1	[1.21579444e+02 2.22222222e-02 8.51134853e+01]	[8.69317065e+01 2.22515331e-02 4.57199540e+01]	1	1	target_not_matched_(FN)	Korean_12	[8.69317065e+01 2.22515331e-02 4.57199540e+01]	0.555923	0	0	[0. 0. 0. 1.]	1	LM_0
LM_0	confused_mlh	confused_mlh	100		[‘Korean_10’, ‘Korean_11’, ‘Korean_16’]	Korean_16	Korean_22	no_label	46.93914	70.3846	0	100	100	time_out		[0 0 0]	[ 0 0 315]	[ 0. -45. 0.]	3	[None, None, None]	[None, None, None]	[1.18738187e+02 2.22284719e-02 1.08817735e+02]	1	1	target_not_matched_(FN)	Korean_10-Korean_11-Korean_16	[7.26637611e+01 2.22515408e-02 4.50597000e+01]	0.551794	0	0	[0. 0. 0. 1.]	1	LM_0
LM_0	confused	confused	14		Korean_16	Korean_16	Korean_23	no_label	2.956677	35.83074	3	14	14	confused	14	[0 0 0]	[ 0 0 330]	[ 0. -45. 0.]	1	[9.14878182e+01 2.22484116e-02 3.09147915e+01]	[ 0. 315. 0.]	1	[1.30289082e+02 2.22222300e-02 1.07202872e+02]	[9.14878182e+01 2.22484116e-02 3.09147915e+01]	1	1	target_not_matched_(FN)	Korean_16	[9.14878182e+01 2.22484116e-02 3.09147915e+01]	0.75852	0	0	[0. 0. 0. 1.]	1	LM_0
LM_0	confused	confused	12		Korean_8	Korean_8	Korean_24	no_label	5.52111	33.1092	1	12	12	confused	12	[0 0 0]	[ 0 0 345]	[ 1.80000000e+02 1.27222187e-14 -7.01670930e-15]	1	[7.47585344e+01 2.22808514e-02 1.32267655e+01]	[180. 0. 0.]	1	[1.03026035e+02 2.22222222e-02 1.11567361e+02]	[7.47585344e+01 2.22808514e-02 1.32267655e+01]	1	1	target_not_matched_(FN)	Korean_8	[7.47585344e+01 2.22808514e-02 1.32267655e+01]	0.674608	0	0	[0. 0. 0. 1.]	1	LM_0
LM_0	correct	confused	14	0	Korean_25	Korean_25	Korean_25	no_label	20.4992	35.18951	5	14	14	correct	14	[0 0 0]	[ 0 0 360]	[0. 0. 0.]	1	[3.56637604e+01 2.22515408e-02 6.60597013e+01]	[0. 0. 0.]	1	[7.56637605e+01 2.22515406e-02 1.06059702e+02]	[3.56637604e+01 2.22515408e-02 6.60597013e+01]	0	1	1	target_in_possible_matches_(TP)	Korean_25	[3.56637604e+01 2.22515408e-02 6.60597013e+01]	0.166687	0	0	[0. 0. 0. 1.]	1	LM_0

vclay · February 12, 2026, 8:52am

Hi @taegeon5846-lab

thanks for the updates. One quick question: Are you still using initial_possible_poses=”uniform”? Because it that case, there won’t be pose hypotheses initialized for 15 degree increments but only for 45 degree increments (see the code here: tbp.monty/src/tbp/monty/frameworks/utils/graph_matching_utils.py at 197d0f0ca18c53c4ec788220cd7bd7f6f9480c04 · thousandbrainsproject/tbp.monty · GitHub). That would be an alternative explanation for why those rotations that aren’t multiples of 45 work less well. If that is the case, you could either adjust n_degrees_sampled to be larger in that function, set the possible poses manually in the array, as I mentioned in my previous post, or use the “informed” option (which is what we usually do to allow detecting any rotation).