Using MCP as the cognitive layer in a robotics stack

Research and Theory
, ,
5

The diagrams above obviously leave a lot of questions unanswered. For instance, how should the messages be addressed, transmitted, etc? Since a need for addressing is inescapable (and transmission modes may vary), let’s look at some existing and vaguely possible addressing APIs.

Elixir - Nodes, PIDs, etc.

I expect (or at least hope) that Elixir will play a role in at least some Monty implementations, but its addressing model is more robust than semantically convenient. So, we’ll probably want to “paint over it”, rather than using it directly.

Still, it can be useful to keep this model in mind, if only because its design reflects the typical architectures and constraints of systems and networks:

  • Millions of Elixir processes can run on a single BEAM instance.
  • Hundreds of OS processes can run on a single processor.
  • An arbitrary number of processors can be networked together.

As you might expect, sending messages within a single BEAM instance is very fast. Messages to other BEAM instances takes substantially longer and messages to BEAM instances on other processors can take arbitrarily longer. So, as in the cortex, we probably want to keep highly related activity at close hand.

In any event, Elixir messages are addressed using a “PID” (Process ID), which is displayed (something) like <0.123.0>. The PID is a largely opaque integer triple which identifies the intended recipient:

  • the (lightweight) Elixir process
  • the “node” (i.e., BEAM instance, OS process)
  • the processor (e.g., DNS name, IP address)

Given a valid PID, addressing and transmission are handled automagically and robustly. There are also ways to make things a bit less clunky. For example, BEAM processes can be “registered” under specified names, either locally or within a cluster.

However, as noted above, this isn’t a semantically convenient way for Monty’s modules to refer to each other. So, let’s start from the other end, asking ourselves about Monty’s message addressing needs.

Leveraging Locality and Infrastructure

AIUI, the neurons in cortical columns mostly direct (really, broadcast) messages to nearby targets. However, messages may also be sent to distant targets and/or in response to a received message from a more distant neuron. So, the target neuron might be in:

  • a specific level in the current (or a nearby) column or minicolumn
  • a “higher level” column or a “lower level” service in the vicinity
  • a distant target, in response to a coincident event and/or message
  • …

Clearly, it would be handy to have a convenient and flexible way to address messages. To simplify things, let’s consider a topology that roughly matches brain structures and some naming conventions that we can leverage with pattern matching.

As a starting point, we can use the rectangular grid of LMs that I proposed over in Pre-wiring Patches of Pixels?:

For simplicity, let’s consider only a single grayscale camera, generating square (1024x1024 pixel) images. We’d like each LM to concentrate on a relatively small (32x32 pixel) patch. This gives us 1024 LMs, which we can arrange as a 32x32 array.

Conceptually, each LM receives data from the patch “below” it. The LM then exchanges information with other (nearby) LMs, in a manner that diminishes over distance. This allows model (etc) sharing, while keeping the overall connectivity manageable.

This “array of LMs” is a very simplistic model of a cortical region. It also assumes the presence of assorted infrastructure (e.g., distant targets, “higher level” columns, “lower level” services).

Let’s set up a few (Sci-Fi!) addressing patterns to see how this might work, using Bash-style brace expansion:

all_around:     "{N,NE,E,SE,S,SW,W,NW}"
northerly:      "{N,NE,NW}"
southerly:      "{S,SE,SW}"

# Send a vote to 24 neighboring LMs,
# up to three (constant) hops away.

ex1_all_around:
  directions:   "{all_around}/{1..3}"
  message_type: "vote"
  target_type:  "LM"

# Send a vote to nine neighboring LMs,
# up to three (constant, northerly) hops away.

ex2_northerly:
  directions:    "{northerly}/{1..3}"
  message_type:  "vote"
  target_type:   "LM"

# Send a vote to nine neighboring LMs,
# up to three (increasing, southerly) hops away.

ex3_southerly:
  directions:    "{southerly}/{1,3,6}"
  message_type:  "vote"
  target_type:   "LM"

# Send a vote to the neighboring LM "above".

ex4_LM_1_above:
  directions:    "above/1"
  message_type:  "vote"
  target_type:   "LM"

# Send a vote to the neighboring SM "below".

ex5_SM_1_below:
  directions:    "below/1"
  message_type:  "vote"
  target_type:   "SM"

Note: Careful Reader may observe that a pattern could easily “fall off the edge” of the region. I’m assuming that this can be handled in a “reasonable” manner…

1 Like