Cortical column I/O vs. Monty's message handling

The CMP notions and questions thread is diving into all sorts of implementation issues. I’m enjoying this and finding it very useful, but I’d also like to know more about TBP’s goals:

Alice: Would you tell me, please, which way I ought to go from here?
The Cheshire Cat: That depends a good deal on where you want to get to.
– Lewis Carroll, Alice in Wonderland

So, can we split off a discussion on goals and general approaches? As a starting point, I’d like to dig into a recent remark by @vclay:

In the brain the different types of connectivity (voting vs. feed forward vs. feedback) originate and terminate in distinct layers of the cortical columns. Therefore the neuroanatomy directly conveys what type of message is being received and how it will be interpreted.

Here are some naive initial questions; ELI5…

Q: Can a given cortical layer send and/or receive multiple types of messages?

Q: Can multiple types of messages be sent along the same neural connection?

Q: Is each of Monty’s modules (e.g., LM, SM) planned to emulate an entire cortical column or (just) a subset of its layers?

If these questions are covered in some of TBP’s written documentation, feel free to provide relevant links…

-r

I spent some time asking ChatGPT about the cortical layers. I then hacked together a spreadsheet: Neocortex Components and Functions, by Layer. I dunno how accurate this is, let alone how useful it might be to anyone. So, YMMV…

-r

1 Like

Hi @Rich_Morin , great questions.

Q: Can a given cortical layer send and/or receive multiple types of messages?

Generally speaking, yes, although not necessarily all layers will. For an example of this occurring, layer 3 sends its primary input to cortical columns further up in the hierarchy (received by a layer 4). However, layer 3 also sends lateral connections for voting to the layer 3s in other cortical columns. We believe that the lateral connections are more likely to be communicating a set / union of possible objects, while the feed-forward connection would be the most-likely object currently perceived by the column. As such, while the messages are all “Cortical Messaging Protocol” compliant, there are differences in their numerosity and how they are processed.

Other examples include how L5 projects motor outputs to sub-cortical areas, and L6 projects top-down feedback to lower level cortical columns, but both of these layers also have lateral connections that likely support a form of voting-type messaging.

Q: Can multiple types of messages be sent along the same neural connection?

In the brain’s cortical columns, any given layer is a complex mix of different cell types, and so while in the examples above a “single layer” may be responsible for different signals, this might correspond to different cell sub-populations within a layer. In particular, different sub-populations tend to have stereotyped connectivity to other parts of the cortex, and so any given message could be the responsibility of a particular cell type.

That said, there are several instances where the output axon of a cell splits and has multiple stereotyped targets in quite distinct areas. In these cases, how the information is processed is likely to be different based on how it is received.

An example of this is shown in the below diagram. L6 top-down projections (purple) primarily end up in L1, affecting many of the cells in a target column via the “apical dendrites” that cells in that column extend up to L1 (grey branches). However, on its way to these targets, L6 also sends very local synapses to cells in the L6 layer of the receiving column (purple “button” like protrusion).

We believe that while the information received in L1 communicates more broad information like object ID (because of the wide-ranging targets), the more localized information in L6 can activate a specific location in that object’s reference frame.

Q: Is each of Monty’s modules (e.g., LM, SM) planned to emulate an entire cortical column or (just) a subset of its layers?

The LMs are designed to have an approximately 1:1 correspondence to an entire cortical column, although there are still features we believe that cortical columns capture, which LMs have not yet implemented.

For example, L5 is very important for motor control, and we believe it may also play a role in how object behaviors are modeled (e.g. the ability of a stapler to open and close, how another person looks when running, etc.). Currently we capture some of this with the “Goal-Stage Generator” class, but most functionality of the biological L5 has not yet been implemented in our learning modules.

In the end, we expect all layers to be captured by our learning module architecture, although the way that their computational properties are structured in the code will not always have a 1:1 correspondence to e.g. an anatomical layer.

Hope that helps, let me know if I can clarify anything further and otherwise Merry Christmas!

5 Likes

My spreadsheet only has columns for layers L1 - L6, but Niels’ diagrams break L5 and L6 into subparts a and b. Is this the general view of the column used by TBP? If so, perhaps my spreadsheet should use this, as well. Speaking of which, any feedback on the spreadsheet would be greatly appreciated!

-r

This may be incorrect (it was gleamed from a pretty old video: https://www.youtube.com/watch?v=7tYxK8DZKYU). So if anyone from Numenta see’s this and notices something wrong, please don’t hesitate to correct me. My intention isn’t to misrepresent you.

That said, to my understanding, Layer 6a and 6b are the layers that perform the columns actual spatial-coordinate processing. L6a measures orientation via modified HD cells, L6b measures location via grid-like cells. They then communicate their findings via the depolarization of neuronal mini-columns found in Layers 4 and 5b.

(PS: The way it was described in the video made it sound as though layers 2 and 3 essentially worked together to perform the same functions as layer 5a. So in my descriptions I describe them in combination, simply as “layers ⅔”…)

Layers ⅔ and 5a serve as the output layers for the column. These layers are capable of temporal pooling; that is, remembering input stimuli over a given time-length. These layers, working with with layers 4 and 5b, comprise much of the columns predictive processing function.

We can view each column as essentially being comprised of two sensory motor subsystems. As you could likely tell from earlier, one of those subsystem is processing location. The other is processing spatial orientation. Layer 5b is serving as the input for that second subsystem. If we were to blow the diagram apart into these corresponding subsystems, it might look like this:

Also, it might help if you’re able to wrap your head around the processing of information as it progresses through the columnar structure.

To my understanding, the data flow would look like this:
New input enters the column through Layer 4. Layer 4 is part of the spatial orientation subsystem (as highlighted by the blue coloring). Its responsibility is twofold: to track incoming object orientation, and to make predictions about that object’s future orientation and scale. It does this via the following:

(1) New data enters layer 4. At time of input, the subsystem does not yet know the orientation of the token represented by that input. However, based on previously learned experience, Layer 4 may attempt to make a prediction as to what that orientation is, polarizing a number of its intra-columnar neuronal populations in the process…

**Fun fact: Layers 4 and 5b are each capable of sequencing patterns. They are the layers responsible for the column’s ability to build models and remember the world. As such, each layer contains approximately 5000 neurons and during any given sparse activation (a reaction to a given input) 100 of these neurons will become active. This method of sparse activation gives each layer 3X10^211 possible neuronal sequencing patterns within its search space–each of these patterns representing a unique token. Additionally, the average pattern overlap is ~2%, making them tolerant to noise and fault.

…As it does this, Layer 4 will bifurcate its throughput off into two directions:

  • It will send a projection up to layer ⅔.

    Layer ⅔ is a temporal pooling layer and, as such, is stable over time. This projection is essentially Layer 4 saying “I predict the input to be the following…” Layer ⅔ will then open its search space for all available models matching the Layer 4 prediction. As additional input is projected into L⅔ from L4 that search space will narrow until only a single representation remains. They have a pretty good write-up on it here: Frontiers | A Theory of How Columns in the Neocortex Enable Learning the Structure of the World

  • Simultaneously, Layer 4 will send a second stream of information down to layer 6a. Layer 6a will then process the input through the use of modified Head Direction (HD) Cells. Once L6a processes the true orientation of the input signal it will depolarize a range of neuronal columns found within L4. (I’m a little unsure as to how layer 6 is processing the information tbh. My intuition says to look into how the hippocampal complex does it, and possibly the cerebellum. But I’m not 100% sure. Need to look more into this myself…)

    These ‘neuronal columns’ are often called ‘minicolumns’ and are a critical component in a column’s ability to perform predictive processing. We’ll find these minicolumns all throughout the broader columnar structure, but they’re especially prevalent in layers 4 and 5b.

(2) With the minicolumns in L4 depolarized, L4 will then relay the correct orientation back up to L⅔. If L4’s initial prediction proved correct everything will continue as normal.

(3) L⅔ will now take its information and send it off to Layer 5b. In this way, the output of one subsystem becomes the input for the second.

(4)This entire process will now repeat for the second subsystem. However, instead of measuring orientation it will now measure spatial location via grid cells. When the second subsystem has finished, Layer 5a will release a motor-behavioral output, affecting bodily functions, movements, and/or other cortical processes. (What it affects depends on the positioning of the column itself.)

Hopefully this is somewhat useful to you. And again, if anything is blatantly wrong with this, please point it out to me. I too would like my notes to be correct :slight_smile:

3 Likes

Dylan- As a complete neuroscience newb, I can’t address the accuracy of this write-up. However, if the Numenta folks say it’s basically OK, let’s turn it into a Gist (or whatever). It can then be tweaked, vetted, and generally turned into an accessible and reliable jump start document for prospective developers.

1 Like

Sounds good to me. Also, Merry Christmas! :christmas_tree:

2 Likes

Merry Christmas and happy New Year!

@Rich_Morin

Thanks for sharing, the Functions area on the spreadsheet looks broadly correct, the one thing that I’m not sure about is what’s meant by “initiates long-range feedforward connections” in L1. The Components section I would be a bit more hesitant to comment on, aspects like L1 mostly being apical dendrites is definitely true, but there are strange omissions, e.g. L4 should contain pyramidal cells (in addition to spiny stellate cells).

If you’re interested in learning more about cortical anatomy, a good starting point might be this paper from 2007: https://www.cell.com/current-biology/fulltext/S0960-9822(07)01148-7

I think a reasonable summary of the division of L5 and L6 into sub-layers is that it’s been anatomically observed, and we’ve discussed some possibilities about their computational significance, although it is currently quite speculative. Thus I wouldn’t worry too much about including it for now.

For example, the video @HumbleTraveller you linked to is a discussion of possible computational roles (location vs. orientation specific), although we’re not certain about these, and in particular Jeff’s discussion of two separate sensorimotor systems is not how I believe we would characterize it today.

In terms of your summary more generally @HumbleTraveller, I’d say it captures the current state nicely, other than as said the two stream point (not your fault of course, it’s what Jeff discusses in that video!). More generally, how L5 integrates information from the rest of the column, and what it does with it, is probably the least understood part from our perspective. The main thing we are certain about is its importance for motor action, given its projections to sub-cortical structures. One other point is that we generally now discuss that L6a → location, L6b → orientation.

For what it’s worth, we’ve been working on-and-off on a more neuroscience focused paper which, when it is eventually posted, should hopefully provide some clarity on these kinds of questions.

4 Likes

Thanks for the additions and corrections! I’ll try to find some time to polish it up a bit, then post a new version. (I suspect that @HumbleTraveller will do so, as well (:-)).

On a related note, I’d love to see an annotated, DOT-style diagram (i.e., bubbles and arrows) and/or pseudo-code description for your planned MVP. Ideally, this would show the major modules and expected connectivity, including the cortical I/O “ports” and types, typical messages, etc. Using this as a basis, we could start to talk about a functional, Actor-style recasting…

-r

2 Likes

Sounds good. Re. diagrams, if you haven’t already seen it, you might be interested in our white-paper that we just published, building off of some of the earlier documentation we’ve shared. In particular, I think figures 5 and 19 would be of interest. If there’s notation that you feel would help make the diagrams easier to understand however, please definitely feed that back.


2 Likes