Pushing values frequently from Elixir via ZeroMQ and polling the latest value in Python less frequently (as e.g. in an experiment loop). See the demo: output
This might be a bit challenging due to the finicky python dependency management for now (due to habitat_sim, for example)
Elixir.ElixirStreamingProducer started, pushing every 100ms,
incrementing the counter by 1
Elixir.PythonPollingReceiver started, connecting...
Elixir.PythonPollingReceiver connected, polling every 1000ms
python: received message: {"counter":8,"random_number":81}
python: received message: {"counter":14,"random_number":58}
python: received message: {"counter":21,"random_number":55}
python: received message: {"counter":30,"random_number":56}
python: received message: {"counter":34,"random_number":18}
python: received message: {"counter":40,"random_number":69}
Elixir.ElixirStreamingToPythonViaZmq stopping after a timeout
which natively connects a python instance into an Erlang/Elixir cluster
this option might be as risky as habitat_sim, as the library hasn’t been updated in a while
python-pyrlang-2-1 | received :hello_from_elixir #1 from <1734706659.110.0 @ erl@elixir.local>
elixir-1 | received :hello_from_python #1 from #PID<18079.0.2> @ py@pyrlang-2.local
python-pyrlang-2-1 | received :hello_from_elixir #2 from <1734706659.110.0 @ erl@elixir.local>
elixir-1 | received :hello_from_python #2 from #PID<18079.0.2> @ py@pyrlang-2.local
elixir-1 | received :hello_from_python #3 from #PID<18078.0.2> @ py@pyrlang-1.local
python-pyrlang-1-1 | received :hello_from_elixir #4 from <1734706659.110.0 @ erl@elixir.local>
elixir-1 | received :hello_from_python #4 from #PID<18078.0.2> @ py@pyrlang-1.local
So, if you omit HabitatSim installation, the main Monty loop should work. However, some of the tests will fail (expecting HabitatSim), and Monty will need some environment to interact with.
I remember, there was the effort to remove it. Since I made a brute force attempt to install monty as a dependency from git, all the declared dependencies must be there, thus the installation failures. If one would want to make tbp.monty code more reusable, it’ll have to be decoupled, even while staying a monorepo, e.g. something along the lines of the ports and adapters architecture, allowing for separate packages that don’t have the unnecessary test dependencies.
Is publishing on PyPi an option at all?
The only way of declaring monty as the dependency that seemed to work from uv was the git repo. I haven’t yet dug deeper how I could try to avoid the habitat_sim dependency this way.
I don’t consider publishing on PyPi a viable option right now because not all of the dependencies are available there. habitat_sim is one of those, but I also ran into some issues with pyg::pyg, pytorch::pytorch, and pytorch::torchvision being different on PyPi vs Anaconda if I recall correctly. I don’t remember the exact problem anymore, though. Those packages come to mind because this was the Windows environment.yml to install Monty without HabitatSim, and those were still being sourced from Anaconda. I haven’t reattempted installing those from PyPi recently.
in a sense, using Python from Elixir is not necessarily required. The demo lets these communicate via ZeroMQ (json-serialized) or via OTP messaging in the second case. So, it’s not critical, however, it’s a sign of future troubles of uses of tbp.monty code in environments where conda is not an option or leads to incompatibilities
Currently, tbp.monty is intended for research, and our focus is exploration and experiments. We expect to change the code and focus on enabling and reducing the cost of change.
The type of Monty running in other environments starts to resemble some sort of platform, where there is enough stability for others to build on top of it, integrate it across hardware and languages, etc.
I see two main options for proceeding. In Option 1, directly evolve the existing tbp.monty code into the platform. I think this will fail because of conflicting change requirements from research vs. the stability needed for a platform to be widely adopted.
That brings us to Option 2, where tbp.monty continues to improve with a focus on research, and we start developing tbp.tbd as that stable platform.
I think, eventually, we’ll pursue Option 2 unless someone beats us to it .
Well, there is also an Option 3 where tbp.monty does evolve into a platform, and some sort of tbp.research ends up splitting from it for research needs. I think Option 3 could also be viable.
Moving habitat_sim into an optional dependency is something that makes sense in tbp.monty-as-a-platform context. However, tests and most of the benchmarks still depend on habitat_sim, so in tbp.monty-as-research context, it is not an optional dependency yet. This is a great example of exactly the tension I highlighted earlier and why one code base is unlikely to serve both needs.
OTP is a collection of useful middleware, libraries, and tools written in the Erlang programming language. It is an integral part of the open-source distribution of Erlang. The name OTP was originally an acronym for Open Telecom Platform, which was a branding attempt before Ericsson released Erlang/OTP as open source. However neither Erlang nor OTP is specific to telecom applications.
.
and why one code base is unlikely to serve both needs.