On the risks and benefits of open-sourcing

General
1

Continuing the discussion from Greetings, from the Mediterranean coast: (splitting this out of your intro thread since this is a big topic on its own)

Hi @ash ,

I appreciate your concerns, and I think, although Monty’s capabilities are currently far away from this, it is good to think about these implications as early as possible. We talk about the implications of open-sourcing this project internally as well and had several strong arguments for doing this. But it is always good to get more perspectives on this topic, so I thought I’ll share our rationale here. The safety-related points fall into two buckets: transparency and defense against consolidation.

Transparency

  • We have made the project open-source and MIT licensed. A large motivating factor for doing so was to ensure that it’s available to everyone, everywhere, and to benefit as many people as possible.

  • It seems unrealistic to expect a powerful technology, as we anticipate Monty to be, to remain under the control of one well-meaning entity. People who want to do harm will find ways to do so. Instead of trying to fight a hopeless battle to wall off this technology from malicious actors, we decided to go the opposite direction and enable anyone to use and understand this technology. We believe that this will make it easier to uncover and prevent realistic risks that the technology poses.

  • We also open-sourced the project so that people could actually have the source code to find bugs and make it more secure. When we say open-source, we don’t just mean that the model weights can be downloaded, but all the source code and data used for training, so it is transparent and replicable. It’s also worth noting that Monty doesn’t require prohibitive amounts of money/compute (such as open-source LLMs do).

  • We publish our research meetings well ahead of Monty’s progress so that academia and industry can learn about the ideas we are exploring and use them in some additive way. As Monty progresses to become more capable, we hope to get more and more input from safety researchers and experts in specific fields where this technology could be transformative. It also gives a head start to policymakers to anticipate potential impacts on society from such a new technology.

  • Transparency like this allows for partnerships and collaboration with the community, academia, and industry to gain early feedback on the direction of the project, which we value deeply.

  • Compared to deep-learning black-box approaches, Monty, in its current iteration, is very explainable and transparent. It is easy to see how it comes to conclusions and why it produces the output it produces. This is useful to understand the outputs and potential sources of error, but also makes it easier to add safeguards around it.

  • Giving everyone access to the technology instead of a few select entities (or even one having a monopoly on it) makes it much easier for people to develop defenses against malicious use cases. It also doesn’t put the responsibility of a powerful technology into the hands of a few entities (who are also in no way guaranteed not to misuse it just because there are fewer of them).

Defense Against Consolidation

I know this doesn’t address the human application of this technology for good or evil, but I think the below points are essential to stop centralized control of this technology, which could end up trending towards a dystopic future.

  • Numenta has a patent non-assert pledge for all of the groundbreaking work on which Monty is based. We think patents are an important way of protecting open-source projects, as they prevent legal attacks from hurting the open-source community or companies that use Monty in products.

  • We will continue to patent new findings and file them under the non-assert pledge so they can be used to defend the open nature of the project.

As you also pointed out, it seems unrealistic to think this technology wouldn’t be developed eventually. (I also think that there are many unsolved problems in the world that will require us to innovate, and what we are building here is a promising candidate for solving them). Of course, any powerful technology has the potential to be misused and can come with great societal changes. We concluded that the safest way to build this technology is together, transparently, and open-source. Happy to hear more thoughts on this!

  • Viviane
6 Likes
2

Thanks, @vclay, for the clear and unblinking exposition of TBP’s rationale, stance, and practices regarding licensing, open source, patent usage, etc. I’m not at all sure where all of this AI stuff will lead us, but I remain hopeful and (like TBP) I’m betting on the side of letting a thousand flowers bloom.

By way of full disclosure, I’ve been a fan of freely redistributable software for several decades. Although I’ve actually written and released very little of it myself, I’ve been very active on the (re)distribution side, starting with the Sun User Group’s SUGtapes (in the mid-1980’s) and continuing with the Prime Time Freeware releases, a bit later. In the process, I met and had extensive chats with various authors (mostly to clarify licensing terms).

I was also privileged to be present at the seminal meeting in Palo Alto, CA. At that meeting, Eric Raymond credited Christine Peterson with coining the term “open source”. (She is also famous for her talks and writings on nanotechnology, e.g. the 2003 hearing on THE SOCIETAL IMPLICATIONS OF NANOTECHNOLOGY.)

A lot has been said and written about Open Source in general and Free Software in particular. For example, there’s:

Having watched these movements develop over several decades, I’m still not sure where I stand on “which licenses are better”, overall. Mostly, I think it depends on the goals one is trying to achieve. In any event, the combination of an MIT license with a patent pledge fits TBP’s stated goals very well.

3 Likes
4

First of all, thank you so much for such a detailed response! Honestly I wasn’t expecting any reaction to what I wrote, I understand that this isn’t necessarily the right place for that kind of feedback.

I also understand your (and your team’s) reasoning. It is a well-meaning position, and, given circumstances, at least it’s a measured and mindful approach to the issue. You are good and mindful people; which, well, is the main reason we are all here in the first place.

But, the thing is - we’ve been hearing this exact reasoning again and again.

I remember several examples of this throughout the past ten or so years, and they led to either of the three:

  • adoption only by the most resourceful (face recognition algorithms, YOLO algorithms, open source LLMs)

  • empowering crime. giving rise to new kinds of crimes, scams, and general enshittification (voice deepfakes, visual deepfakes, LLMs again)

  • people losing their careers, wasting years of education and money (Stable Diffusion)

The technology we’re discussing here will hit the same exact wall. Unless someone is willing to make robotics accessible to “everyone” along with the software. In a way, the potential for centralization here is even worse, because even the most expensive server can be rented for a couple hundred bucks an hour, whereas things like production lines and logistic chains are completely off limits for the general public; they’re only accessible to the most resourceful. And you see these people every day in the news. It’s not the best selection of human beings to be even remotely trusted with anything of that sort.

Seriously. We’ve been here before. Quite a few times by now. This reasoning doesn’t work, not the way we’d probably like it to. It’s way past time we stop pretending it does. We can have all good intentions in the world, but it’s not enough.

But yes, it can’t not evolve either.

So the only correct approach is to evolve it mindfully too.

PS: I’m not here to argue, or to try to persuade someone. I’m not arguing for changing the course of the project in any way. It’s already out there. The only thing in my control in this case is what and how much I can contribute to it, and I’m content with that.

2 Likes
5

I seem to recall Jeff mentioning a book about managing open source projects, but can’t remember the title.

6

I don’t think it was Jeff, but we like the book “Producing Open Source Software - How to Run a Successful Free Software Project” by Karl Fogel

2 Likes
7

I definitely appreciate your points, @ash and it is important to have this conversation. I don’t want to sound like I think there will be no changes or risks with a novel technology like the one we’re building. Also, open-sourcing surely doesn’t solve all problems. But as you said, the technology can’t not evolve either. I don’t think we can just prevent progress. We have the blueprint for building intelligent machines right in front of us (in our brains), so someone will do it. At the TBP, we try to do this in the safest and most transparent way possible.

You are right, that even though running our system doesn’t require prohibitive amounts of data or compute, many future applications will require robotics hardware, which isn’t accessible to everyone.

Luckily, Monty, with its current capabilities, isn’t anywhere near replacing someone’s job, so we still have some time to figure out how to adapt to this new technology and evolve with it. I am happy you are still interested in contributing, and I’m always open to suggestions on how we can develop this new technology in a way that most benefits society.

6 Likes
8

Like the first generations of computers. Like, bronze tools, even. Needs some time to mature.

The same could be said about other technologies, especially the invention of the Internet itself.

This will happen regardless of TBP’s outcome. People from both pro-AI and anti-AI camps agree on one thing: there will be socio-economic disruptions while supply and demand is shifting. Nobody can accurately predict what will come out of it, but personally, I side with the optimists. I think that having a career will start mattering less for livelihood security as AI gets more serious. It’ll take some time, but we’ll get there.

If embodiment proves to indeed be a hard requirement for achieving general intelligence, I posit that it would not be a perpetual requirement. You could use a robot body to bootstrap and train the model, and once it reaches a certain “IQ”, transfer it to a simpler non-robot device, without loss of cognitive or learning abilities, and make it quite accessible.

I believe Monty has the potential to become the “Linux” of AI. In a world of proprietary frontier models, this kind of project is a necessary counterbalance. The barrier to entry is tremendously high, requiring astronomical effort. TBP and Numenta have been at it for a combined 20+ years, and there’s still lotsa work that remains.

If not them, then who? There’s no other serious contender. I don’t want a future where the chosen few have a monopoly over AI, locked away in corporate cathedrals of computation. My cortical columns unanimously refuse that outcome. It belongs out there, grounded in the real world, in the hands of the common folk. They have a right to explainability, a right to repair, and a right to privacy.

Let’s give 'em.

4 Likes
9

Two different things are mixed up here: brain-inspired algorithms for robotics and general intelligence. The second one, the longer nobody knows what it takes to create it, the better. But the first one simply means, potentially, more intelligent and capable robots - still in the hands of the few. None of these rights mean anything if the common folk can’t get a hold of rare earths or whatever. Which is why the comparison with bronze tools doesn’t hold: ancient people didn’t need planetary-wide supply chains to use bronze. The level of complexity required in this case is unmanageable, and the power imbalance baked into the current system is just too high. We keep talking about how this or that technology will make the world better, but unless it is actually deliberately applied to make it so, preferably by someone who knows what they’re doing, it’s all just an exercise in magical thinking. It won’t happen by itself, left to its own devices it will only exacerbate what we already have.

1 Like
10

Look, all I’m saying is - it is much more complicated than we tend to think. Right now, there’s no need for extra levels of precaution or some specific kind of deliberation. But they will be required in the future. The project is not there yet, but it might get there eventually. Which is why in my original post I objected to its long-term goals, not the short-term ones.

2 Likes
11

There are a number of different ways to run AI code:

  • cloud-based vs. local processing
  • open source vs. proprietary code, models, etc.
  • …

Buried among all the combinations are a few strategies:

  • Download open source models from archives.
  • Train new models on compute farms; use them wherever.
  • Use curated combinations of code and models.
  • Use each technology for what it seems best at doing.
  • Use open source code and models by preference.

In line with this, I’d (eventually) like to see the TBP release “ultility” modules that can advertise and provide specified services (e.g., fast training for recognition and identification of objects, movements, etc).

Then, for example, I might be able to start up a set of assorted AI components to collectively perform desired tasks. Some of these might be “in the cloud”, while others might be running on a laptop or even a cell phone.

1 Like
12

These are 2 sides of the same coin; better robotics enabled by generalization capabilities toward novel scenarios. When I say general intelligence, I’m talking about ARC-AGI’s definition, which Hawkins agrees with:

AGI is a system that can efficiently acquire new skills outside of its training data.

Granted, I was a bit ambiguous between physical capabilities and cognitive capabilities in my previous reply. I was more focusing on the latter than the former, e.g. running a Monty-powered conversational model on a local device.

Yes, I agree it will require some sort of shepherding as it gets more advanced. I don’t know how much of that is part of Hawkins’ vision and objectives, but ideally, there will have to be a workgroup to help steer the technology’s alignment.

This workgroup could perhaps even distribute a range pre-trained models, in a similar fashion to DeepSeek and Qwen. I know there will be people who will pry guardrails off, just like abliterated LLMs, but that’s the unavoidable tradeoff of open-source.

I still think open-source is a better approach to AI than Big Tech and their “trust me bro” attitude. Legislation wouldn’t help with AI alignment either, because lawmakers don’t have that much leverage on what tech companies do behind closed doors.

You can’t prevent bad actors from exploiting your code, but with Hanlon’s razor in mind, how would you “steer” usage of an open-source project? Well, I have some relevant experience to share…

In 2013, I co-founded the A3Wasteland project, a video game mod for Arma 3. At that time, a lot of Arma modders were starting to close their source code by turning their mods into SaaS-like platforms, because they disliked that people made derivative works of their creations. (The game’s file format made it very easy to extract anyone’s source code.)

I thought that was really dumb, so I went full open source with AGPLv3, the strongest copyleft license there is, requiring providers to publish their code, server stuff and all. I even deliberately intertwined client and server code in a way that made it very difficult to SaaS-ify.

At its peak, there were over a hundred communities running my mod on their servers, with thousands of players. I had no direct control over these communities, I just told everyone to fork my code and customize it how they wanted.

As time progressed, communities altered my code more and more, sometimes in ways that became detrimental to the broader playerbase. The situation was getting a bit ridiculous for the players, who were my number 1 priority at all times.

I had to do something, but had no control on how communities used my code… Most open-source projects rally around a main repo, but in my case, the main repo was just the seed, with a hundred forks as its roots.

Therefore, I tried a roundabout way to nudge things in the right direction. I was releasing content updates semi-regularly, which were often quickly adopted and deployed by the communities, as they competed with each other to have the latest and best features. I started using these content updates to introduce balance changes and try to undo some of the damage.

I was in active discussion with prominent communities about why I was introducing those changes and how they would benefit everyone. This approach was relatively well-received, most accepted the changes, and it resolved some of the gameplay issues. A minority didn’t like it and refused, but that was inevitable.

So, it proved to me that there is a viable middle ground for open-source projects to guide a decentralized ecosystem of end users in a certain direction, without giving in to authoritarianism’s siren song.

I foresee the project might evolve into something similar, with tons of variants around the world. Not necessarily of the codebase, but rather of the models. Can this sort of steering approach work at a global scale? I don’t know. All I know is that it’s the only principled one in my book. It won’t stop all bad actors, but it would certainly foster good ones.

It’s a little early, but we’ll figure this out.

3 Likes
13

I mean no disrespect, and I’m truly supportive of the experience you’ve had, but this is deeply unserious; we can’t be comparing something that, by its own admission, can potentially turn the world upside down to a video game mod. Stakes are completely different. Yes, there are indeed ways to guide an open-source project, there’s even a concept of a benevolent dictator for life, but if/when this project gets where it wants to be (which is not even necessarily the capacity for generalization, but simply models efficient enough to replace whatever is employed right now on the market), none of them will work. And yes, we can democratize what’s already out there, because an OpenAI subscription is already more or less accessible anyway; that is a good pursuit, and the least harmful pursuit, too, given the circumstances. Something like this could be distributed widely, under any license. Which, presently, would either tank the entire US economy, because a handful of companies have been passing around the same purse in a circle to save the country from recession - or force cloud providers and consumer electronics to increase prices to make up for the damage done (or both). But that’s beside the point. On the other hand, if we’re talking about widely distributing efficient models for robotics and/or generalization, this will lead to what I’ve already described at length.

Fundamentally, this project is about algorithms and architecture, that, coincidentally, can be applied to various domains. And it is everyone’s responsibility to be deliberate about the choice of these domains. If Monty is to become infrastructure, the question “infrastructure for what exactly” needs to be answered. I don’t think I’ll be able to formulate it better than that.

14

On the other hand, being selective regarding potential domains means giving up the leverage in the ones that won’t be selected; so it’s even worse. Theoretically, what could be done is models could be developed, including in these critical domains, their efficiency proven, results published in some form, but the models themselves - distributed only if (and preferably deleted until) someone somewhere develops a similar solution and starts using it to make a profit. This would discourage companies from investing into their own research. But to all the wrong people this will be a declaration of war, and they will hate you for that

15

Ideas should be judged on their pros and cons, not their origin. I stated that I don’t know if this would work. I simply haven’t encountered a more appealing idea yet. My book is always open to new ones.

I know what the stakes are. I’m not taking this lightly by any means, quite the opposite. The team clearly isn’t either.

That’s quite the assertion. Would you care to provide concrete examples as to why none would “work”?

I’ve never been a fan of the BDFL paradigm. A leader’s ideas should face the same level of scrutiny as a random person. Likewise, a random person’s ideas should not be dismissed on the basis of reputation. Hence why a consensus-based open ecosystem seems to have potential for a model training workgroup.

(Surprisingly, none of the fine folks over at LessWrong seem to have made any meaningful proposal for such an approach. They mostly focus on the output rather than the input.)

In the last YT short, Hawkins said that the Thousand Brains principles might become the dominant form of AI at some point. Come to think of it, his book did seem to hint between the lines at a grander plan beyond Monty… Whatever it is, I want front row seats.

Although, please keep swinging the sword of Damocles. It is by all means warranted.

2 Likes
16

They won’t work, because they won’t matter. Even if the open source community restraints itself, and it probably will restrain itself, via elaborate guardrails, the development outside of it won’t, in the pursuit of whatever competitive advantages others might think of.

In some competition optimizing for X, the opportunity arises to throw some other value under the bus for improved X. Those who take it prosper. Those who don’t take it die out. Eventually, everyone’s relative status is about the same as before, but everyone’s absolute status is worse than before. The process continues until all other values that can be traded off have been – in other words, until human ingenuity cannot possibly figure out a way to make things any worse.

Once there are clear instructions and a clear pathway for making it do what they want it to do, they will. And they won’t bother adhering to any policy or anyone’s opinion on that. And this ain’t even Linux: you can’t tell Linux “build me a swarm of armed drones” and expect it to deliver.

As for swinging - oh that’s nothing, you should see my socials, I’m restraining myself here. You’re decent people, and we only need to build proper and complete mental models of the things we’re discussing. Me included, maybe I am missing something, but I’m not seeing it yet.

1 Like
17

That makes two of us.

Re-read sections VII and VIII of the essay you quoted.

Good actors outnumber bad actors by a large margin. Therefore, the open nature of the training ecosystem would make it naturally grow into the countermeasure the author theorized. Such a community even has the potential to push it all the way toward a “Game B”-class endeavor, if people put their mind to it.

“If we’re really lucky, we could see broader positive cultural changes and a grassroots upsurge of R&D on the AGI for the greater good could be part of this broader cultural change. It seems like a rosy-eyed, optimistic possibility, but it’s not impossible.” — Ben Goertzel, 2022

Although, unlike Goertzel, I don’t think of it as a game of luck, but as a plant seed. Hawkins and the team have been hard at work crafting it for years, for all of us. Once it’s ready, we must make it sprout and grow a garden. We can figure this out.

1 Like
18

Goertzel even actively tries, bless him; I’ve been getting frequent emails from his SingularityNET project with regards to “beneficial AGI”, whatever these folks mean by that. Which again confirms the idea that effort needs to be actively put into it, and even that is not a guarantee of success, but at least it gives a chance.

19

SingularityNET is neuro-symbolic papier mâché. It will melt in the rain. The cortical seed, however, will germinate. :eye:

20

I’m not saying they will achieve anything; I’d attended one of their online conferences like a year ago, and the things some panelists were discussing - serious scientists, supposedly smart and respectable people - made me climb walls. This entire field is just awful. The level of discussion even at its very top is infuriating. But as far as intentions go, Ben seems to be a decent example. Surrounded by all kinds of fools and grifters, but still. Again, to your point that there are many more good actors than bad actors - I can’t say the same about the topic of AI/AGI particularly, unfortunately.

1 Like