FWIW, the RasPi 5 has two high speed camera ports, as well as USB links that can handle lower speed devices. All told, it should be possible to collect quite a bit of video data, either passively accepting the M5’s motions or even directing them in true sensorimotor fashion.
So, I’d like to hear what folks think about the prospects and best approach(es) to collecting and integrating this data in Monty.
Hmmmm… The difference in sampling rates is an interesting challenge. I’m working through creating nested operating windows, bound to HW clock cycles. Basically, using a global clock to provide a temporal frame of reference. It’s a bio-inspired approach to spatio-temporal integration, but it’s obviously not implemented yet, so it can’t help here. Even still, you might be able to steal and idea or two from it.
Barring that, we might be able to tinker with evidence accumulation rates?
I’d need some time to sit and think through this one, honestly. Would be curious to know what thoughts others have too.
That looks very cool and not all that expensive. I’m not sure how it would fit into a Monty-based system, but then I assume that the fancy processing is all optional. I also worry about the “tight integration with Raspberry Pi’s camera software stack” which might make it hard to use with Nerves.
What are the project’s current notions about clocks, simultaneity detection, synchronization, etc? Clearly, the brain has periodic “waves” that pass through; should Monty have something like that?
Speaking of hardware clocks and such, the Network Time Protocol - Wikipedia manages to keep disparate and widely distributed clocks in sync:
I wonder if Monty will be forever constrained to viewing the world thru an RGB straw, ignoring a full image at a glance, and not able to make use of smart sensors, which may be able to recognize the features, of even multi-object scenes?
Essentially, we would have nested operating cycles (or windows), inside of which you could have certain model-based policy states operate. These windows would then be synchronized across LM’s by a common temporal reference frame. In the brain, its the medullary nuclei which provides this. For Monty, I assume utilizing the clock cycles of the CPU would provide the same kind of functionality.
The smallest oscillatory wave is known as a gamma wave. It oscillates at a frequency of 30-100 Hz (or about about 30-100 operations per second). In context to your project, I was imagining you could process the highspeed camera throughput with every op of a given gamma window, whereas it’d be maybe every third gamma window for the slow camera throughput. You could then synchronize disparate LM data at every theta window refresh, for example. This way the Monty system could remain stable/synchronized despite the sampling differences of its sensors.
P.S. Yes, I know that 30-100 ops/second is slow for a modern processer (most operate in the range of ~4 billion ops/sec). The example is only meant to demonstrate the nesting behavior.
Lastly, as for simultaneity detection, I’m still working through that. The brain has mechanisms for coincidence detection, though I suspect this is something already being worked on by the TBP team.
@rolly,
I don’t think this will be the case for forever, no. As long as the sensor is able to transmit into CMP format, I see no reason why you couldn’t have any number of arbitrary sensor type.
I’m currently working out all of the different sensor input vectors found on the human body and will be posting something on that soon. But to use our own physiology as an example, we have around 24 different sensor types (depending on how finely you want to class them).
Also, its good to remember, that our entire observable world is made up of around 100,000 of those straws (of which only a small subset are RBG capable). And yet we manage to get along just fine.
Thanks much for the rundown on oscillatory waves, coupling, etc. Since no good deed goes unpunished, there are some follow-up comments, questions, and war stories interspersed below…
After assorted part-time jobs and a purgatory year doing COBOL, I moved on to Acroamatics (i.e., esoteric lectures :-), a tiny company that did a lot of work with PCM Telemetry.
We mostly dealt with commutated data streams, generated by combinations of nested “rotors”. The motivation for commutation was parsimony and simple, conservative design: housekeeping values (e.g., batteries, power system) don’t require as much attention as mission-related data, but they stiil need to be dealt with.
By the time our code saw the data, it had been frame synchronized by hardware. So, we saw a series of fixed length records with designated, fixed-width fields. Most of the fields would be used for mission-related data.
However, a couple would be reserved for housekeeping data. These would be time-sliced by some factor. Some of the resulting locations might be cycled (via “sub-commutators”) into subsidiary categories, including power systems. Somewhere down the tree, there would be a time slice for battery temperature.
Your approach seems quite reminiscent of all this to me…
In a proof of concept system, this could be done by a single “time-keeper” process. However, for a production version, I’d want to see at least one “time-keeper” actor in and for each BEAM instance. These could communicate and coordinate with each other via CRDTs, NTP, etc. And a pony.
I assume that the time scale for waves will be influenced both by the physical world and by the vagaries of computer systems. That said, it seems quite likely that we’ll want to support various sampling rates, etc.
So, the time-keeper actors might cycle through a data structure of notification instructions, resulting in periodic update messages (e.g., {:ping, time_stamp) to subscribing clients. Sounds a bit like pub-sub to me…
Of course, all of this could make Monty “aware” of time in ways that humans are generally not…
Thats pretty cool. Sounds like you’ve been around a bit!
Have you ever worked in the HF radio spectrum? When I was with the Army, we used to train up on ionospheric refraction of radio signals using that. If I remember right, the best we were able to accomplish was transmitting imagery from Michigan to a base in Alaska. It was only a single image. But still, was kinda cool considering it was simple point-to-point comms.
Your approach seems quite reminiscent of all this to me…
I mean, both the brain and PCM are trying to solve a lot of the same problems. They both use some form of time-division multiplexing, data compression and error correction. Its not too surprising that we’d see them converge on some of the same stratagies.
I assume that the time scale for waves will be influenced both by the physical world and by the vagaries of computer systems.
Oh, for sure. Some of the ways the brain is influenced include things like E/I ratio balance. As for external stimuli, physically moving around in the enviroment is known to force theta modulation in the hippocampus (via the medial septum). But then you also have these things called rythem generators. They basically force the surrounding network to sync with them. The hippocampus is one such generator, another important one is the medulla. The medulla is actually a type of generator known as a CPG. Basically, structures that can maintain their rythem independently. Coincidentally, the medulla is actually the only CPG found in the brain. Most other CPG’s are found in your organs. They’re the reason things like your heart can beat, even in the absence of brain activity. Interestingly enough though, you lungs don’t have this (it relies on the medulla). This likely has to do with our ability to voluntarily hold our breath.
Alright, I’m done geeking out about nervous systems and networks now lol.
I’m curious about your time-keeper actor statement. Are you able to expand on your thought there?
There’s enough biojargon flying around here that I had to look this up to find out whether it was a typo or a previously unknown term. Seems to be the former, but please let me know if I’m still missing something. FWIW, “rhythm” is a good hangman word, because it only has a single, part-time vowel.
I’m happy to give this a try, though I’m not any sort of expert. Also, I won’t try to find the simplest, most optimized, or even most relevant approach. So, YMMV…
Many software tools have been developed to schedule periodic and/or aperiodic activities. In Unix, for example, we have the at(1) and cron(1) commands. Elixir has libraries that provide similar facilities, including Oban, Quantum, and Periodic. FWIW, I found a nice article on the latter: Periodic jobs in Elixir with Periodic.
In short, getting time-based notifications is pretty much a matter of making the right library calls (i.e., “a simple matter of software”). The notion of “cycling through a data structure of notification instructions” adds some complexity, but it’s not really arcane.
Let’s say that you want notification messages to be sent at various intervals. You could ask Periodic to regularly call a function that generates each desired message, but this might get complex and/or provide too little flexibility.
Alternatively, you could generate a :ping message at the fastest desired rate, then generate less frequent messages via a subsidiary “rotor” (i.e., commutation) actor. Here’s a proof of concept sketch…
Set up a list where each item contains a commutation factor, message types, outgoing addresses, etc. When the rotor receives a :ping message, walk through the list. Divide each time_stamp by the relevant commutation factor. If the remainder is “small enough”, generate a notification message. Rinse, repeat…
Of course, if the list gets long enough, a linear traversal will become inefficient. So, you could introduce a list of lists, etc. This should let the rotor reduce the number of divisions and such to roughly log(N) of the number of items.
On a vaguely related note, back in the day I developed data analysis software for the Solar Maximum Mission satellite. The challenge was to detect solar “events”, while avoiding drop-outs and collecting background levels before and after each event. My solution used a (rather cool, IMHO) two-pass data analysis approach.
The Good News is that each orbit of the satellite took it behind the Earth for a while, then out where it could see the Sun. So, the incoming data was nicely divided up by orbits. Based on this, I scanned each set of orbit data for items of interest (e.g., drop-outs, flares). I recorded these, along with their time stamps. In the case of solar events (e.g., flares), I fudged the time stamps to provide times that were a bit before the actual event.
I then sorted the entries by their time stamps and used the result to generate a “microcode table” containing time stamps and desired actions (e.g., ignore this stretch of data, collect background or event data). Finally, I walked through the orbit again, using the table to tell me what I needed to do at each time.
Re. rythem generators
Oops. Yeah, that was a typo. I meant to say rhythmic generators. Also known as Central Pattern Generators (or CPGs). They’re essentially just brain mechanisms which help establish network connectivity/synchronicity.
Also, those orbital dynamics you mentioned, with regards to the satellite, actually makes for a good example of what I was talking about earlier. Many types of CPGs in the brain require external stimuli to reliably time the execution of their functions. And just like you used the orbit cycle data to help calculate those events, the brain uses similar ‘sleep-wake cycles’ for its various functions too.
All that said, your satellite experience is pretty awesome. Do you still work in the field now, or are you retired?
Also, I was thinking of the binding of functions to CPU clock cycles versus relying on a time-keeping function (eg. A cron job ), and the advantages/disadvantages of each. I’d be curious to know your thoughts…
CPU Clock Cycle:
Pro: higher precision. Relying on clock cycles just allows for more precise timing. Thinking at the mico or nanosecond scale.
Pro: no OS dependency. Could theoretically run on bare-metal. This could be better for things like microcontrollers, embedded systems, that sort of thing.
Con: tied to CPU performance. Though we might be able to say the same thing for things operating above the OS.
Con: potentially higher CPU usage. This would depend on how its implemented. But if it was done via something like polling, it could be more inefficient than using a time-keeper.
Time-keeper Actor:
Pro: easy to use/maintain.
Pro: potentially more scalable. Though this might be a moot point if functions/policies are implemented well enough in a clock cycle-type approach. So IDK.
Con: lower precision/higher latency. From what I know, most time-based schedulers operate at the scale of seconds and minutes.
Con: limited reactivity. Unlike a clock cycle approach, which can be event-driven, most schedulers that I’m aware of operate at predefined intervals.
When you guys were implementing programs into your various aerospace platforms, which approach did you find worked best? I ask because I suspect one of the more popular platforms we’ll find Monty operating in, once its more mature, is going to be that of (semi-)autonomous space systems.
Starting around 1975, I worked as a contract scientific programmer and technical writer at NRL, SLAC, and assorted smaller outfits. I also ran Prime Time Freeware, a one-horse publisher of mixed-media (book / CD-ROM) FOSS collections. I now refer to myself as “semi-retired” (mostly as a face-saving measure :-} ).
Defining the source of all chronological truth is a bit above my pay grade, but here are some comments…
Most computer systems have both a hardware clock and some software which makes “the current time” available to system and application software. Since most of my background is tied to *ix systems, I’ll use them as my starting point. Unix time counts the number of seconds since the “epoch”:
The Unix time 0 is exactly midnight UTC on 1 January 1970, with Unix time incrementing by 1 for every non-leap second after this.
Some *ix variants subdivide the time further, going down to (say) milliseconds. In general, the precision is quite good; however, I don’t know of any general-purpose variants that support micro-, let alone nanosecond granularity.
Still, I don’t think that any of this is likely to be a problem for Monty. My assumption is that Monty has only a few reasons to concern itself with time, e.g.:
the “sleep-wake cycles” which you mentioned
assessing the simultaneity and/or speed of events
…
NTP allows distributed computer systems to maintain a “pretty good” common idea of time, in the face of common network issues (e.g., bandwidth limits, data loss and/or error, latency). It’s also a popular (nay, ubiquitous), well supported, and well understood standard. So, what’s not to like?
For situations where you need to know the exact ordering of events, something like a CRDT seems to be the Golden Path. (This could be used in a time-keeper actor; no need to make it part of all Monty modules.)
What I would like to see Monty add is some time-related support. Just after receiving a message, each module should update a couple of fields and include them in any outgoing (generally, CMP) messages. For example:
message_count - an internal counter, incremented for each incoming message
system_time - a copy of the (OS) system time
I suspect that these two fields, coupled with some CRDT magic, could handle most of Monty’s synchronization needs.
I’ve been thinking a bit about camera placement and data interpretation for use cases with multiple cameras. I’ll mostly use my proposed Maslow 4 sled as an example, but note that (a) cheap cameras are readily available and (b) various endeavors might benefit from multiple cameras.
Nothing is expected to be creeping up on the M4 sled, so watching the surrounding area isn’t a strong motivation. That said, here are some possible use cases.
high-speed video monitoring of the router
It would be nice to be able to collect info on the router’s bit and its near vicinity (e.g., activity, results, state). So, mount a pair of high-speed video cameras on opposite sides (e.g., +/- X) of the tip.
wide-area scanning of documents, etc.
The sled can move at 100 ips, so an edge-mounted camera (e.g., cantilevered up and out) could scan a large area pretty quickly. Using multiple cameras would speed things up, provide redundancy, etc.
Cameras could also be mounted at the sled’s edge (e.g., raised up and angled down). Angled cameras would encounter issues such as depth of field and keystoning, but these aren’t necessarily fatal flaws. For instance, they could be used to scan the surrounding surface at lower resolution (e.g., looking for and discounting blank areas).
Time for a discursion on supporting multiple eyes. Human eyesight is almost entirely binocular. Horse eyesight is monocular to the sides and rear, binocular at the front (except for a fairly large region just in front of the face which is totally blind). Spiders and such have LOTS of eyes, working (somehow) in parallel, and they don’t even have a cortex.
So, (how) should Monty integrate data from multiple cameras? Should we consider Bug-eyed Montys?
Inquiring gnomes need to mine…
