My name is Collin, I am a Machine Learning engineer who has spent most of my career working on neural rendering applied research projects for GPU companies. I have long been interested in neuroscience and biologically inspired learning algorithms, and have been researching opportunities to explore this space in my free time. Discovering the Thousand Brains Project has been exciting and motivating!
For the last two years I have been implementing and optimizing small neural networks on a mobile GPU, which has led me to spend a great deal of time considering the inefficiencies of modern deep learning. This has led me to read more and more about neuroscience and neuromorphic computing, and cemented my strong belief in the need for biologically inspired algorithms in order to improve the efficiency of AI.
In particular, I have been exploring my own ideas related to conditional computation and hierarchy. After reading the Thousand Brains Theory book, I am now convinced of the need to reground the development of general problem solving from biological principles. I find the core principle of reference frames to enable robust representations compelling, particularly as we consider extending these principles to more abstract reasoning tasks beyond sensorimotor processing.
My background is a mix of AI development in neural rendering/computer vision, GPU architecture, and GPU kernel programming/optimizations. While I’m still exploring the codebase to understand where I can best contribute, I’m particularly interested in how my experience with hardware-software co-design might be relevant to the project’s efficiency goals as it scales. I’m excited to start with smaller contributions while learning more about the architecture and project roadmap.
I would love to chat more with the team and discuss how I can best help. Looking forward to working together on this amazing open source project!
Welcome Aboard! When I lived in the Bay Area, I was able to attend some of Jeff’s talks in person. Be sure to take that opportunity, since you’re nearby.
In other news, I’ve been promoting the idea of using Elixir’s Nx (Numerical Extension) library as a way to code Monty’s mathematical algorithms for GPUs. You might find it interesting to examine Nx and evaluate its suitability for this purpose.
Welcome to the forum! Please feel free to reach out any time. We do have a overview of how the community can help us as a starting point: Community Involvement 🌏
Welcome to the forums @collin , that’s awesome to hear about your interest in the TBP, as well as your background. Definitely think there are some great ways you could contribute - let us know any questions you have once you’ve had a chance to look at the link @brainwaves shared.
@nleadholm and @brainwaves Thank you for the welcome! I have spent the last week exploring the codebase, documentation, and tutorials. There are a lot of interesting directions to explore on your roadmap that I would be interested in, particularly the item for GPU acceleration as that aligns well with my background. Is there someone on the team I could chat with about the current thinking on GPU acceleration for the project? @Rich_Morin already provided one lead with the Nx library.
I am also interested in exploring the simulation environments in more detail, as I have experience with game engines. Perhaps working on one of the items such as creating a multi object scene dataset could be a good starting point.
Any directions on what might be the most useful ways for me to contribute would be appreciated!
Hi @collin ,
welcome to the TBP community! I’m happy to read that you have looked through a lot of our resources and are interested in contributing to the project It sounds like you have a great background to be able to help us with some of our current issues!
I would say the biggest pain point we have right now that matches what you are interested in is the 3D simulator. Our current simulator (HabitatSim) is coupled to an old Python version and is forcing us to use GPU instances even though all of Monty’s operations are good to parallelize across CPUs. It also doesn’t run on Windows, which makes it hard for Windows users to run some of our benchmark experiments, unit tests, and tutorials (even though Monty code itself does not depend on Windows) @tslominski wrote out a nice post about this here The Next Monty 3D Environment Simulator
If we could have an alternative simulator integrated in the Monty code base (like Mujoco) it would solve several of our current issues.
GPU acceleration for Monty is interesting, however, we haven’t prioritized it as much so far, as Monty can be run in a parallelized way, eliminating the need for GPUs. Being able to run it on CPUs only has some nice practical advantages. However, we are transitioning to running some of our experiments on GPU instances now (because of the Habitat simulator issues on CPUs) so if you have ideas for making the most of those GPUs by also having Monty make better use of them those are certainly welcome contributions as well. However, we would like to avoid a strict dependency on GPUs.
I’m happy to give more details on either one of those topics although @tslominski is probably the best person to comment more.
Best wishes and looking forward to hearing more from you,
Viviane
As @vclay already mentioned, our current GPU-adjacent work involves integrating another simulator. However, that is more about code refactoring to allow another simulator class to be used, and is not GPU-specific. We (@jshoemaker specifically) intend to start that simulator work sometime in June. At that point, we will undoubtedly experience the challenges of figuring out GPU-related dependencies on various systems.
As I’m trying to think of an overlap between Monty’s needs and what might interest you, what do you think about seeing how Monty itself (not a simulator) can be updated to use different backends for its computations: MontyBackend.CPU, MontyBackend.GPU, MontyBackend.CUDA, MontyBackend.METAL, etc. Maybe this would be a per-Learning-Module configuration? I haven’t explored enough to know what makes the most sense yet. Supporting multiple backends in this manner helps us avoid GPU dependency while being able to take advantage of it if available and desirable.
Initially, it would be interesting to see what performance gains are achieved (if any) using a different backend for current computations.
@vclay@tslominski Thanks for the responses! Based on Viviane’s message I spent some time this morning looking over MuJoCo and HabitatSim. It would be fun to dive into MuJoCo if that is still of value, but I am also fine to hold off if you want to stick to your current schedule.
GPU acceleration does interest me though I would be happy to explore any roadmap items that are useful. That said, I think supporting multiple different backends to take advantage of available hardware as you described is a valuable direction to pursue. I will start looking into GPU support and see if there are any operations that see a performance boost.
(I’m also in Bay Area)