What if (?) Monty had a live view

having spent some time with python’s asyncio now, I must say - in Python, yes! Python is not a concurrency-oriented language even without the GIL (the disablement of which makes it even less of a concurrency-oriented language). In the long run, asynchronicity is just a life fact - sensors are physically parallel, thus also asynchronous. ZeroMQ having a separate I/O thread can help with async communication without blocking the interpreter but beyond that I now also don’t see the point of introducing asynchronicity in the single interpreter python version of Monty

3 Likes

has anyone tried running it yet?

I’m somewhat dazzled by the progress the Elixir ecosystem is doing, and here’s a refreshing set of architectural features that might make it interesting for future Monty - without having to “rewrite everything in Elixir/Nx”:

Dashbit blog: Distributed Python dataframes and machine learning with Livebook and Elixir

Notable features that could come in handy:

mechanics:

  • Zero-copy transfers of Apache Arrow data between Elixir and Python, allowing developers to load data in Elixir and transform it in Python or vice-versa

distribution and scheduling:

  • Distributed Python execution via the Erlang distribution, thanks to our distributed garbage collector
  • Support for spawning and running Python/Elixir code across K8s clusters and Fly.io machines

unprecedented terse syntax and LiveBook goodies:

I guess, the real power of these options comes when one starts playing around with that type of architecture.

More importantly, we have implemented a distributed garbage collector. So when remote_eval/4 returns, it simply holds references to objects in the remote node. This means you load a dataframe in one node, then pass the reference to another node, and decide if you want to explicitly copy it (using pickle or cloudpickle ) or perform further remote execution.

The ideas continue persistently to build upon the sensible concurrency- and distribution-first nature of Erlang/OTP/and thus Elixir that allows treating non-concurrency-oriented languages (by design, not out of necessity) as single-threaded actors, simplifying both the code in sequential languages but also concurrency, distribution and fault-tolerance in one sensible abstraction.