I’m creating this topic in an attempt to break my habit of asking @tslominski for extra work in random comment sections on GitHub.
So, what next injection or feature you’d want assistance with? Anything goes.
3 Likes
Hey @AgentRev, thank you for volunteering.
I propose working towards the Desired Effect (333 Telemetry collection is configurable down to the Python module level.).
Leading to it is (Injection #81 Use structured logging to emit telemetry inline).
This is a large project. Basically, the goal is to replace our custom logging framework for all the scientific data with a logging-like telemetry pipeline that emits snapshot events. Essentially, replacing tbp.monty/src/tbp/monty/frameworks/loggers/graph_matching_loggers.py at main · thousandbrainsproject/tbp.monty · GitHub and everything that interacts with it. That is, no longer collecting data at hardcoded step, episode, or epoch boundaries, but instead emit them inline with appropriate event metadata so that any application reading the telemetry snapshot stream can construct whatever it needs for data analysis/visualization purposes.
The Current Reality Tree Undesirable Effect (103 Platform internal representations are difficult to visualize) contains more context on the problem we are running into.
I have only thought through the design of this at a very high level. Essentially, what I’m thinking is something analogous to the logger interface, but for telemetry.
So, where we have logger:
logger = logging.getLogger(__name__)
# ...
logger.error(...)
logger.warn(...)
logger.info(...)
logger.debug(...)
We would have an analogous pipeline (using logging as the implementation detail) that would emit telemetry events:
telemetry = monty.getTelemetry(__name__)
# ...
telemetry.snapshot(...) # some scientific data
telemetry.snapshot(...) # some scientific data
This way, we could reuse all the existing logging infrastructure to select, handle, and filter scientific data emitted by the application.
Aside from the emit side of things, this project also requires rethinking and redoing all of the live and offline visualizations.
This is a large topic. I think we’ll need an RFC to keep everyone on the same page. Before we start going through all the details, I want to see if this is of interest first.
4 Likes
Yeah, sounds great. It’s been a recurring subject of discussion since Brighton, especially in the live view thread. I’ll do some reading and thinking on that, to get into the mindset.
2 Likes
I noticed that @ash has done some work on a telemetry prototype here, perhaps it can be of use: tbp.monty/src/tbp/monty/libs/telemetry at 08b20e113072be40014883dd83d174d34b7b8365 · thousandbrainsproject/tbp.monty · GitHub
Seeing all the other fine work you and the team have been doing over the last weeks, I assume this feature is probably less urgent? If desired, I could also help with shorter-term stuff, like @jshoemaker’s MuJoCo integration.
For the MuJoCo integration, I’ve got a pretty clear roadmap on what needs to happen, and I’m in the middle of the first task, so I’m not sure how easy it would be to work on it in parallel. I’m planning to pull that prototype apart into separate stages, and it’s all very fresh in my head at the moment, including the changes I want to make along the way.
1 Like